Connections from wsprdaemon being reset after a little over 1 hour - SOLVED - overload issue
It looks as if since the last update to v1.489 all connections from a Raspberry Pi running latest wsprdaemon are automatically reset after al little over 60 minutes.
Does anybody else experience this as well? Can it be seen from the log if this is a KiwiSDR / Beagle action or triggered from the outside?
The log looks like the below
Wed Feb 23 10:23:03 12:14:44.342 01234567 /users REQUESTED from 192.168.1.82 Wed Feb 23 10:24:39 12:16:20.686 012345.7 6 L 18106.10 kHz usb z0 "wsprdaemon_v3.0a" 192.168.1.82 (LEAVING after 1:03:10) Wed Feb 23 10:24:41 12:16:22.435 01234567 6 TLIMIT exempt local connection from 192.168.1.82 Wed Feb 23 10:24:41 12:16:22.436 01234567 6 PWD kiwi SND ALLOWED: config pwd set, but is_local and auto-login set Wed Feb 23 10:24:41 12:16:22.544 01234567 6 L 18106.10 kHz usb z0 "wsprdaemon_v3.0a" 192.168.1.82 (ARRIVED) Wed Feb 23 10:25:02 12:16:43.945 .1234567 0 L 24926.10 kHz usb z0 "wsprdaemon_v3.0a" 192.168.1.82 (LEAVING after 1:03:32) Wed Feb 23 10:25:05 12:16:46.433 01234567 0 TLIMIT exempt local connection from 192.168.1.82 Wed Feb 23 10:25:05 12:16:46.433 01234567 0 PWD kiwi SND ALLOWED: config pwd set, but is_local and auto-login set Wed Feb 23 10:25:05 12:16:46.540 01234567 0 L 24926.10 kHz usb z0 "wsprdaemon_v3.0a" 192.168.1.82 (ARRIVED) Wed Feb 23 10:25:06 12:16:47.863 01234567 /users REQUESTED from 192.168.1.82 Wed Feb 23 10:25:14 12:16:55.511 0123.567 4 L 10140.20 kHz usb z0 "wsprdaemon_v3.0a" 192.168.1.82 (LEAVING after 1:03:47) Wed Feb 2
Note. I do indeed use a beta wsprdaemon but that has been very stable before the release of v1.489
Thanks for any suggestions
de Erwin
Comments
I have 2 kiwi with BBAI 1.489 and 2.10k WD running here and am not seeing that.
Running 2.10k now, let's see what happens.
thanks for the reaction James.
Same with the WD 2.10k after an even shorter period
NOT happy 😪
what else is the RPi doing other than WD? I run 28 channels through an Odroid XU4 running 2.10k without an issue but that is right on the edge of too much for it.
Doing nothing else, I have had word from other users for a different Beagle AI? that this heralds the complete failing of the Kiwi/Beagle hardware within a few days ...
I have now switched off all channels but 1 wspr and see what happens. The reset time was around 55 minutes still.
ADC clock66.666519 (33.0k avgs)
Up20:29:57, v1.489, 8 SDR ch, 12 GPS ch
GPSacq yes, track 7, good 3, fixes 33.0k
WF22 fpsAudio48.0k, Qlen 5
BB55,16,25 usi%1.0 GHzFPGA29%
for reference, I have 2 14ch BBAI feeding the 26 channels to the XU4. I left 1 ch on each kiwi availablle for me to connect and monitor
eg:
I do not see the CPU-temp here :
Config: v1.489, 8 SDR channels, 12 GPS channels | Uptime: 20:49:17 | UTC: 18:57 | Local: 19:57 Europe/Paris (CET)
GPS: acquire pause, track 4, good 4, fixes 33.5k, ADC clock 66.666520 (33.5k avgs)
SNR: All 14 dB, HF 10 dB
Beagle: cpu 58% usr | 18% sys | 21% idle,
1.0 GHz,
FPGA eCPU: 11%
Network (all channels): audio 72 kB/s, waterfall 9 kB/s (16 fps), http 0 kB/s, total 81 kB/s (648 kb/s)
Stats: 0 dropped, 5 underruns, 0 sequence, 17 realtime
Realtime response histograms:
Reset
Datapump: 0, 2.7k, 208, 78, 0, 0, 0, 0
SoundInQ: 0, 5.1k, 418, 261, 70, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
RX0
"F4VTQ" (192.168.1.58, unknown location) 3965.45 kHz drm z6 DRM 0:25:55
RX1
RX2
RX3
RX4
RX5
RX6
"wsprdaemon_v2.10k" (192.168.1.82, unknown location) 3570.10 kHz usb z0 0:34:39
How is it networked?
If you do a continuous ping from another device is it stable, or does that drop?
Is it publicly accessible?
Also from an ssh terminal (copied from the AI thread) "Type "ct" to check cpu temp (divide by 1000 for deg C)."
CPU temp only on BBAI/14 ch
Stable, 0.6 ms average from same LAN
Open from internet but not publicly advertised
Do you have SNR measurement configured to run every hour? That's the only thing I can think of that would run on that schedule.
The v1.489 release (and 488 for that matter) had only minor changes. And none of them had anything to do with networking.
Does
crontab -lshow anything configured to run every hour? Have you recently make this Kiwi open to the Internet? If so, it's possible it's been infected by a virus. Although you'd almost certainly have to have forced an open ssh port with a password-less root account for that to happen.SNR was on without issues, it should not influence connections from wsprdemon of course. I switched it off yesterday, no change in the situation.
No password so no ssh access from external, default settings.
Reset to factory sd card software now?
Sorry on the temp I saw mention of AI early in the thread.
On the ping did you run it for over the duration of drops?
By Ping "not influence connections from wsprdemon of course." Do you mean the wsprdeamon is running on the same device?
Is this a genuine Kiwi, on a Beaglebone green or something else? Have you downloaded the blacklist on the network section of the admin? If it is open to the internet on the http server port it is possible you have some external cause so limit access to just the IP's you require (by the router if possible)
On the ping did you run it for over the duration of drops? No because I would need to be sitting next to it to monitor the exact moment of the reset to see what has happened just before that on the ping. I could write ping to a text file ..
By Ping "not influence connections from wsprdemon of course." Do you mean the wsprdeamon is running on the same device? No it is on a RPi; SNR did not influence situation, as it answered that part of your questions. It did not answer your ping question.
Yes it is a genuine Kiwi with a BB in the Seeed alu case. From a reputable German Radio amateur webshop.
All standard installed, including blacklist that nicely loads on restart.
As far as I know the access is only via my ddns setup.
When I do a factory software reinstall all external access should be off and I will see what happens before making it externally available. Just made a backup of the factory SD card and the BB/Kiwi itself as well.
Ping average all the time also when reset happened 0.684 ms with deviation 0.074 ms.
So nothing happening to the network connection.
WSPRdaemon is running from (again) another RPi, the 3rd used for testing, so that is not the source of the issue.
This could be either end, the PI needs to be able to get to the Kiwi reliably so if you have a third machine (E.G. windows) run two terminals at roughly the same time. That should give you a comparative result close enough to see what is happening over an hour and quarter. Could be that the PI is losing network (WiFi? 0.684 ms -suggests this)
ping -n 4500 -w 1000 [kiwi-IP] > pingtestKIWI.txt
ping -c 4500 -w 1000 [PI-IP] > pingtestPI.txt
Obviously you could run it for much less time if you can see a pattern and know a break is coming up.
Reinstall from factory SD + auto-update to 1.489 no difference, reset around 1h 22 m
Not sure why the network would be a problem when it was not a problem the first couple of weeks but will setup the double ping test.
At this point you should ask on the wsprdaemon forum: https://groups.io/g/wsprdaemon/topics
The only other Kiwi forum thread I could find was this one: https://forum.kiwisdr.com/index.php?p=/discussion/comment/9661/p1 which turned out to be a wsprdaemon bug (the disconnects were more frequent -- every few minutes).
My theory is that WD is detecting many overload events being reported by kiwirecorder and WD restarts the kiwirecorder to stop the overload log file from growing so large that it fills the file system.
In WD 3.0 I will use the newly introduced adc_ov line status line to obtain that overload information, so there will be no need to restart kiwirecorder sessions when there are many overloads
But that is still a theory
Sounds plausible
I spotted this early on but didn't see the point in mentioning it, overload is now the point.
SNR: All 14 dB, HF 10 dB
Also if related to overload the duration would reflect the time of day(?).
Indeed solved by Rob, and thanks for all support from the others reacting to my situation.
Overload as a result of a strong LW station and many AM signals. Introducing a 15 dB attenuation in the LZ1AQ antenna feed and setting the antenna amplifier to a max of 2Vpp introducing another 3.6 dB attenuation cleared it all up.
KiwiSDR happily blathering away on 8 channels wspr again.
😀