wsprdaemon problem
After upgrading my antennas and power supply for my KiwiSDRs, my wsprdaemon set-up has stopped working.
wsprdaemon V2.7a
KiwiSDRs V1.383
Raspberry Pi 4
all running off 2-12V deep cycle marine batteries (trickle charged, connected in parallel).
After booting the Raspberry Pi 4 (from a fresh install of the operating system), everything seems to work properly - then after 1-2 minutes, all of the KiwiSDRs drop their
wsprdaemon assignments and reload (dumping whatever WSPR and Noise results that were recorded before they are reported).
Even simpler wsprdaemon.config scripts I've used previously do not work anymore - swapping to a different Raspberry Pi 4 has the same problem.
The Kiwis seem to be working properly.
600' Sky Loop: http://kk6pr.ddns.net:8075
136' 80-10 EFHW: http://kk6pr.ddns.net:8076
I'm usually able to debug/fix these things, but I haven't a clue what might be causing this.
Any suggestions?
73 / Rick
KK6PR
wsprdaemon V2.7a
KiwiSDRs V1.383
Raspberry Pi 4
all running off 2-12V deep cycle marine batteries (trickle charged, connected in parallel).
After booting the Raspberry Pi 4 (from a fresh install of the operating system), everything seems to work properly - then after 1-2 minutes, all of the KiwiSDRs drop their
wsprdaemon assignments and reload (dumping whatever WSPR and Noise results that were recorded before they are reported).
Even simpler wsprdaemon.config scripts I've used previously do not work anymore - swapping to a different Raspberry Pi 4 has the same problem.
The Kiwis seem to be working properly.
600' Sky Loop: http://kk6pr.ddns.net:8075
136' 80-10 EFHW: http://kk6pr.ddns.net:8076
I'm usually able to debug/fix these things, but I haven't a clue what might be causing this.
Any suggestions?
73 / Rick
KK6PR
Comments
This happens with newly formatted, OS/wsprdaemon installed SanDisk SD cards, with plenty of disk space. Everything works fine until it comes time to report the results and then the connections between the Kiwis and Raspberry Pi drops (every 1-2 minutes). I've tried different Raspberry Pis, SD cards, OS versions - it even happens when only 1 Kiwi is assigned to just 1 WSPR frequencey.
I've been doing this for around 2 years w/o any problems, so don't know why this happening now.
Power cycle any network switches or hubs in the line.
Check the power supplies to same.
Change ports and or cables.
If it happens with varied OS on the PI then that makes me think the packets are getting black holed, or wrongly routed.
If it is every few minutes is could be a PSU or switch on the way out.
Check ARP, make sure you don't have any wrong IP bindings in the router.
Do a long term ping from a few devices on the network to see if it is specific to the Kiwi/PI or network path.
E.G. from PI "ping -c 500 kiwi"
From a third device "ping -c 500 PI" we want to see if the general network is solid during the drops.
When I first encountered this problem I suspected the Kiwis were not getting enough power - they are all running off of two 50lb 12V Marine/RV deep-cycle batteries connected in parallel. One of them was getting rather old, so I replaced both of them with brand new batteries. They test fine and nothing else runs of them, I don't think that is the issue.
Ports and cables have been working for over two years - nothing has changed or been damaged. Connections/communications between all devices are working.
Pinging everything works, no inordinate delays/problems.
The 'every few minutes' aligns almost perfectly with the end of recording and whatever analysis/reporting stage comes after that. The link between the Kiwis and Raspberry Pi drops, so WSPR and Noise results are deleted and not reported.
Over night I had two of the Kiwis on separate antennas monitoring 80/40/30/20 meter WSPR - using just the Kiwi extension, with no wsprdaemon. After six hours, both (KK6PR-L & KK6PR-E) had reported over 1000 (~200 unique) contacts each. The results we posted at WSPRnet and the VK7JJ websites, but not at WA2ZKD Live Spots.
Not sure exactly what happened at 2320Z on Mar 9th - but that's when it all quit working.
Just test it, break the problem down.
Think of it this way - at kit install did you measured the metal type and thickness of the plating on the RJ45 plugs then checked the that against the mating surfaces so that you are sure there will be no possible corrosion after two years under the predicted temperature and humidity? No? well you could just test it now.
"Pinging everything works" did you try the longer run?
You mention the batteries but not the DC-DC after that (is the voltage stable under load?). Did you test any router / switch dc supplies?
Does the wsprdaemon use an IP address or a name that resolves locally?
Is IPv6 enabled on the network? Where does the PI get it's DNS?
What other devices share this network?
(I have seen issues years ago on a Linux install where the routing went screwy when an early Windows7 box was added to the network, as long as IPv6 was off on the Windows box it was fine but the MS tended to want to route all traffic (to it's own little dev/null).
CenturyLink gave me a lot of trouble when I first tried opening the ports for public access to my Kiwis. Trying to set it up via my router's Port Forwarding page didn't work, they said open ports weren't allowed and wanted to sell me a commercial account. After a week of daily calls and threatening to cancel, everything magically started working and I haven't had any problems since.
I'm not doing anything different now that requires those sorts of changes though. I can access all my Kiwis from the Raspberry Pi - in fact, I can run my Kiwi's WSPR extension from a browser window and the results are reported just fine - I guess I'll be doing it that way until a solution is found.
Yes, I did the long ping test - certainly longer than the two-minute interval during which this situation occurs.
There are a lot of things that are working fine, so many of them can be eliminated as suspect.
The two batteries are rated at 600-amps cranking power (each) - without a direct short somewhere, they'll never come close to drawing that. From the battery, the Kiwis are connected to DC-DC converters rated at 3-amps each.
The four Kiwis are connected to the CenturyLink modem via a Netgear GS608 hub (they are the only devices connected to it). It has eight ports, I'm only using four - if they're all sending a lot of data at the same time (every two minutes) perhaps it's not able to keep up during those spikes in activity - I don't know, I'm not really a Network guru. I may try connecting the hub directly to the batteries.
As for the other network issues you mentioned, I'm not that savvy. I've been running three Kiwis for over a year without problem - I don't understand why adding the fourth would break everything to the point that even one doesn't work now (using the previous wsprdaemon configuration that has worked so well).
As much as I would like to combine the results from my two antennas into one grand WSPR receive score - perhaps separating the results by antenna type may be more realistic and interesting, something I can already do with the setup I have.
Did you add the fourth Kiwi as a complete new name and IP in there, or move / rename others?
I doubt you are over loading any working Netgear switch purchased in the last twenty years, the problem looks like a miconfiguration as lowering the load does not fix it.
When you say you know the exact time it started what else happened then? is there any shedule in the WD config, was that some fixed time after last config edit?
Have you recently measured the actual voltage being supplied to the Pi, and have you measured the voltage actually supplied to the Kiwis? Never assume the power supply system is fine - always verify.
It is entirely possible that you've got a sick SD card. Have you attempted this with a different brand/size/type of SD Card?
Thanks 73
Rick / KK6PR
I've disabled wifi on the Pi and using a cable connection - everything is working fine now.
'git pull' will download v2.7b which fixes this problem