KiwiSDRs stuck at "Upgrade - Y/N" prompts

ka7oei

An interesting thing happened at the Northern Utah WebSDR where we have three KiwiSDRs. The power got unceremoniously shut off when, unknown to us, the local power utility started work on a project to upgrade the lines that feed the WebSDR site (we aren't the customer of record so they did not know to contact us) and when power was restored, unlike other servers on site, the KiwiSDRs never came back up - and the Kiwis have been quite reliable on this regard in the past. We know that the power went on and off several more times over the next few days due to additional utility work and weather, but they never, ever came back up on their own. Note that this is a very rural location and that the duration of these power outages were such that they exceeded the duration of any reasonable UPS that one might have.

After being unresponsive for more than a week, which included some really bad weather which made safe site access impossible - one of locals near the site (it's a 130km/80 mile drive each way for me) finally had time to investigate and on a hunch, he connected a monitor (via dongle) and keyboard to the Kiwi after finding that it seemed to boot - sort of - but the Ethernet interface wouldn't appear to go active - and found it wanting a "Y/N" answer to a question about OS upgrade (I got this related to me second hand so I'm likely omitting details).

Taking the safe route, he chose "N", finding an identical situation on all three units, and they proceeded to start up, download the newest KiwiSDR sources, update and come online. Was this likely a result of the "unclean" power-down, or is this something else?

Even with the help of the excellent search feature of this forum - and Google - I couldn't seem to generate a "hit" on a topic similar to this, but I'm sure that it can be explained with a link.

73,
Clint
KA7OEI

P.S. Both the admin and debian user passwords had, long ago, been reset for the default.

jks

So this doesn't make a lot of sense. The logs below support the idea of 3 separate power failures.

What I'm more interested in is this "Y/N prompt" business. It is true that the Kiwi Makefile does a number of "apt-get -y install ..." commands during an update. This normally has a "Y/N"-style prompt before it installs anything. The "-y" flag guarantees that it never asks and always assumes "yes". But the "-y" is missing in one of the apt-get commands in the Makefile (will be fixed in the next release).

BUT this should make no difference as the Kiwi server is run in the background and apt-get will never prompt unless there is a controlling terminal on which it can ask the question. I tried this myself. Intentionally removing some needed packages and removing the "-y" from the Makefile. The apt-get never got stuck. It worked just like it's supposed to.

So I can't imaging how it is your friend saw a prompt from a command running in the background after simply logging in. He must have done something else? It just doesn't make any sense. If the apt-get that is currently missing the -y were actually stuck prompting there would be hundreds of effected Kiwis worldwide. And there don't seem to be. If he has any additional information I would like to know..

http://kiwisdr1.utahsdr.org:8073
Nov 25 07:42:09 kiwisdr kiwid: 1d:04:15:44.309 ........          UPDATE: switching to new version 1.352
update reboot
Nov 25 07:42:21 kiwisdr kiwid: 00:00:00.243   KiwiSDR v1.352 --------------------------------------------------------------------
...
Nov 26 20:29:45 kiwisdr kiwid: 1d:12:47:23.931 0.234567 0         7204.00 kHz lsb z6  "108.215.192.243" Reno, Nevada, USA (ARRIVED)
power cut?
Nov 26 21:50:59 kiwisdr kiwid: 00:00:00.190   KiwiSDR v1.352 --------------------------------------------------------------------
...
Nov 28 08:41:12 kiwisdr kiwid: 1d:10:50:13.725 ..234567 0         1988.00 kHz lsb z8  "73.34.6.3" Pueblo, Colorado, USA (LEAVING after 0:02:14)
power cut?
Nov 28 09:27:20 kiwisdr kiwid: 00:00:00.209   KiwiSDR v1.352 --------------------------------------------------------------------
...
Nov 28 09:28:03 kiwisdr kiwid: 00:00:43.577 ........          TIMEZONE: "MST", "America\/Denver"
power cut? then failure to reboot
...
intervention
Dec 10 23:11:07 kiwisdr kiwid: 00:00:00.935   KiwiSDR v1.352 --------------------------------------------------------------------
Dec 10 23:20:43 kiwisdr kiwid: 00:09:36.723 ........          UPDATE: switching to new version 1.359
update reboot
Dec 10 23:20:55 kiwisdr kiwid: 00:00:00.682   KiwiSDR v1.359 --------------------------------------------------------------------

http://kiwisdr2.utahsdr.org:8073
Nov 25 07:40:07 kiwisdr kiwid: 1d:05:32:50.814 ........          UPDATE: switching to new version 1.352
update reboot
Nov 25 07:40:19 kiwisdr kiwid: 00:00:00.922   KiwiSDR v1.352 --------------------------------------------------------------------
...
Nov 26 19:10:19 kiwisdr kiwid: 1d:11:30:00.754 ..234567 0        14280.00 kHz usb z9  "190.201.99.239" Anaco, Venezuela (LEAVING after 0:13:43)
power cut?
Nov 26 21:51:19 kiwisdr kiwid: 00:00:00.795   KiwiSDR v1.352 --------------------------------------------------------------------
...
Nov 28 07:35:33 kiwisdr kiwid: 1d:09:44:14.799 ..234567        7     0.00 kHz  am z0  "NorthernUtahWebSDR_KiwiWSPR" 10.5.94.49 (ARRIVED)
power cut?
Nov 28 09:27:19 kiwisdr kiwid: 00:00:00.675   KiwiSDR v1.352 --------------------------------------------------------------------
...
Nov 28 09:28:03 kiwisdr kiwid: 00:00:44.176 ........          TIMEZONE: "MST", "America\/Denver"
power cut? then failure to reboot
...
intervention
Dec 10 23:11:11 kiwisdr kiwid: 00:00:00.082   KiwiSDR v1.352 --------------------------------------------------------------------
Dec 10 23:20:30 kiwisdr kiwid: 00:09:19.778 ........          UPDATE: switching to new version 1.359
update reboot
Dec 10 23:20:42 kiwisdr kiwid: 00:00:00.777   KiwiSDR v1.359 --------------------------------------------------------------------

ka7oei

I see what you mean about the power failures - and I'd never heard of this happening, either, which probably explains the lack of search results.

Previously, when all three Kiwis were offline, we were able to attribute it to the pair of linear power supplies (in redundant "wire and" diode configuration) crowbarring - which we have occasionally seen due to our extremely dirty mains power (which sometimes goes as low as 90 and as high as 140 volts for seconds at a time.)

The friend that was up there who says he saw the "Y/N" prompt had been up there a few times before finally getting them online - and he'd only power-cycled them a few times, which correlates with your logs.

One of the possibilities that we'd considered was that the network routing had been messed up - but these are on the same switch as three Northern Utah WebSDR servers, which never had connectivity issues, and the network guy *swears* that the routing wasn't changed, which is borne out by the fact that you seemed to see entries in the logs.

Perhaps the most interesting thing about this is that a number of times, including 5-10 minutes after my friend had power-cycled the the Kiwis, I'd checked the ARP table on "websdr3.utahsdr.org" - a machine which is on the same switch - and I could always see the KiwiSDRs before (and can right now) - but during the time period that they were "stuck", I *never* saw any of them on the ARP table, let alone able to ping them with their internal "10" network address from that machine, which is in the same subnet as the WebSDR servers themselves.

At this point, I don't know what to think if I take both the guy that says he pressed "N" and the other (network) guy at their words.

Let us hope that this does not happen again!

Thanks for your hard work.

73,
Clint
KA7OEI

jks

he saw the "Y/N" prompt

But he saw it WHERE???? Under what conditions? In the Kiwi log / syslog? Somehow on the terminal screen when he connected a monitor/keyboard or connected to the serial port? After he tried to run some command manually?

I spit acid at my laptop screen when you guys don't give me enough information/context. It's absolutely maddening!

ka7oei

Although this may cause more acid spitting he said, during our very brief conversation, that it involved the use of the micro-USB port and gear that he'd used on other BB projects of his own.

I have not talked to him since and that is all I know: I will refrain from speculating on the specifics.

Clint