KiwiSDR unintentionally shutting down (caused by SW update???)
I have two KiwiSDRs, one local, and one in a remote location 200km away. The remote Kiwi has had a bad habit of turning off without any explainable reason. Since I only visit the remote site in weekends, I switched the two Kiwis previous week, so now the suspect Kiwi is in my primary location.
Last Friday I drove up to the secondary site, bringing the "good" Kiwi. I connected it to antenna, ethernet and power, and switched it on. When trying to log in, i got the response "software upgrade in progress...". (v1.424)
When trying to log in to the other Kiwi, at the primary site, it was dead. No ping. (it was working a few hours earlier)
Both Kiwis have similar power supplies (lab power of 3 or more amps (at around 10V) followed by a LM350T 3A linear regulator at 5.5V inside the Kiwi aluminium profile cabinet (not the original one), current consumption ca 0.6-0.8A fluctuating
I have measured voltages on the Kiwi when it is in the error state:
:input at the Kiwi barrel connector 5.5V
:5VExt pins to BB 5.5V
: 5V from BB ~0V
:3.3V from BB ~0V
The shutdown has been occurring regularly, at least since June/July, and for a while, I am getting a strong suspicion that is is coincident with the software update process. It seems _this_ Kiwi runs a SHUTDOWN and not a reboot after update.
I have not yet restarted the "dead" Kiwi, awaiting tips/advice on checking logs and startup messages from the Kiwi,
73 de la2zoa, Knut
Comments
Do you have SSH access to the BBB under the Kiwi? If you do then you can ssh in and "less /var/log/messages" as root and see what's going on.
You can also use the admin console ; console tab to get the same, but its lots harder to read long text files there.
I have 3 KiwiSDR units that have run here for a few years. Start up problems have either come from a power supply starting after a shutdown and the voltage not staying above 5 v at the instant the KiwiSDR tries to start, or the voltage drop in the supply cable caused the supplied voltage to drop below 5 v at the moment the kiwiSDR tried to boot. A larger wire gauge solves the supply drop in that connection, and for me, installing a startup delay timer between the supply and the KiwiSDR solved the voltage drop caused by the delay in the supply coming up to operating voltage at startup. The KiwiSDR needs to have 5 vdc minimum when it starts and thereafter.
Generally, once the supply voltage is at 5 volts and the KiwiSDR is reconnected to it, it will boot just fine. If this is the case, then add the delay timer between the supply and the KiwiSDR. The update process complicates the testing, so be sure that the KiwiSDR is updated before testing power supply.
Ron - KA7U
Thank you, Ron.
I don't believe there is a problem with voltage drop in the power supply. The LM350T (3A) regulator is less then 6 cm from the Pi, and has an additional 220uF capacitor at it's output. There are no problems with initial start-up from powerdown, My other Kiwi with identical PS (without the 220uF) works perfectly. If voltage drop at start-up was the cause, it should also be apparent at any (or many) startups, not only at reboot after sw upgrade.
This is why I suspect that for some reason, the cause is that the Beagle executes a "shutdown" instead of a "reboot"
Unfortunately I don't know where to look to find the Kiwi software update scripts to see if they could be tampered with. I found that port 22 could unintentionally have been left open in my internet router (it is closed now)...
Knut
Regarding port 22. I have heard of devices that some friendly white hat had discovered "open on the web without decent password" and just shut them down to give the owner a clue something was wide open, while at the same taking the device out of harms way.
I really can't see that the software build would shut the device down unless the EMMC is corrupt (?)
I'd watch the logs while running the manual update.
I think this really does need logs as the assumption here is that it is related to the software update whereas the software update may be related to a power or network interruption (assuming you often have active users).
73 Stu
Please check the admin page options:
After updaterestart servervsreboot Beagle(typically "restart")Daily restart?Yes/No (typically "No")Although these wouldn't explain why the Kiwi failed to subsequently respond unless there are power quality problems (e.g. brownout) during the peak high-current draw of a reboot. You really need to study the times in the log files to determine the sequence of events. This is not always easy.
Re your observation of the "good" Kiwi updating to v1.424 as soon as it was started at the secondary site. More information is needed. What version was this Kiwi running previously? Why wasn't it able to update at the primary site? No consistent Internet access? Had it updated without issues previously?
New shutdown 29/30 - 11:
After the previous shutdown, I disabled auto update, the config is now:
In Update Tab: Automatic check update: No, Automatically install: No
In Control Tab: Daily restart: No,
After restart fw is still 1.424, so there have been no update, hence my assumption about updates causing the shutdown is most probably wrong.
I will try to investigate the power supply quality, using a scope, but I should really have had an analog data logger, which I don't have.
Is it possible to set up logging from the Kiwi to a network drive, to possible find what may lead to the shutdown?
73 Knut
If you have another Linux machine on the network open the syslog to remote devices (UDP 514) then edit (on the Kiwi) /etc/rsyslog.conf to send the logs out to the remote server example "@192.168.1.5".
The Beagle is really quite strict about how it will boot so worst case is probably momentary mains failure where the power drops to say 4V but comes straight back up.
You could try one of those tiny DC-DC switchers, they are remarkable and what noise there is is very well controlled. They seem well suited to the particularities of BB, come up fast as well as recover fast but have very little capacitance so if the supply is off, it's off straight away. at least that way you would know if adding the extra cap to the field Kiwi is actually causing more issues that it is curing.
73 Stu
Problem Solved!
last night, I pulled out my multi-meter and oscilloscope to make measurements on the Kiwi.
The Kiwi is mounted in an aluminium cabinet with no external access to the 5V Kiwi barrel connector, since the cabinet also contains a LM-350 adjustable voltage regulator module, with 5.3V output soldered directly to the 5V barrel connector pins on the Kiwi PCB.
Using the digital multi-meter, the power input to the Kiwi looked fine, 5.3V and stable.
Then I connected the the oscilloscope to the 5V input, and noticed that the input voltage to the Kiwi contained very deep voltage drops when vibrating the cabinet and/or touching the power wiring inside the cabinet. Finally I located the problem to a imperfect connection in a PCB screw terminal connector on the voltage regulator board. Fixing the connection and re-checking with the scope to verify solved the problem.
Lesson learned: Never trust a digital multi-meter alone when investigating problems with power supplies!
Case closed...
73 Knut😀
Us oldtimers rely on Simpson 260 VOM at times!
Glad you got it fixed
Well, I thought i had fixed it... at least the intermittent voltage drops are gone. (I am not sure the would have been visible even on a Simpson, since the were very short)
This morning it had powered off again, and after manually restarted by toggling power switch, it had upgraded fw from v1.424 to v1.427.
Is there a way to send all logging (as in the admin Log page) in real time to an external device like a network syslog server, external network disk, a memory stick, or an USB serial port?
73 Knut
I just noiced that Powernumpty has a setup for syslog server above. I have a Synology NAS im my lan, I will try to set that up tomorrow. Thanks Stu!
Knut
I see Synology may have an add on for that.
My /etc/rsyslog.conf edits are similar to those found in https://www.debuntu.org/how-to-remote-syslog-logging-on-debian-and-ubuntu/