The KiwiSDR 2 online store is open for orders! Please visit kiwisdr.nz
KiwiSDRs stuck at "Upgrade - Y/N" prompts
An interesting thing happened at the Northern Utah WebSDR where we have three KiwiSDRs. The power got unceremoniously shut off when, unknown to us, the local power utility started work on a project to upgrade the lines that feed the WebSDR site (we aren't the customer of record so they did not know to contact us) and when power was restored, unlike other servers on site, the KiwiSDRs never came back up - and the Kiwis have been quite reliable on this regard in the past. We know that the power went on and off several more times over the next few days due to additional utility work and weather, but they never, ever came back up on their own. Note that this is a very rural location and that the duration of these power outages were such that they exceeded the duration of any reasonable UPS that one might have.
After being unresponsive for more than a week, which included some really bad weather which made safe site access impossible - one of locals near the site (it's a 130km/80 mile drive each way for me) finally had time to investigate and on a hunch, he connected a monitor (via dongle) and keyboard to the Kiwi after finding that it seemed to boot - sort of - but the Ethernet interface wouldn't appear to go active - and found it wanting a "Y/N" answer to a question about OS upgrade (I got this related to me second hand so I'm likely omitting details).
Taking the safe route, he chose "N", finding an identical situation on all three units, and they proceeded to start up, download the newest KiwiSDR sources, update and come online. Was this likely a result of the "unclean" power-down, or is this something else?
Even with the help of the excellent search feature of this forum - and Google - I couldn't seem to generate a "hit" on a topic similar to this, but I'm sure that it can be explained with a link.
73,
Clint
KA7OEI
P.S. Both the admin and debian user passwords had, long ago, been reset for the default.
After being unresponsive for more than a week, which included some really bad weather which made safe site access impossible - one of locals near the site (it's a 130km/80 mile drive each way for me) finally had time to investigate and on a hunch, he connected a monitor (via dongle) and keyboard to the Kiwi after finding that it seemed to boot - sort of - but the Ethernet interface wouldn't appear to go active - and found it wanting a "Y/N" answer to a question about OS upgrade (I got this related to me second hand so I'm likely omitting details).
Taking the safe route, he chose "N", finding an identical situation on all three units, and they proceeded to start up, download the newest KiwiSDR sources, update and come online. Was this likely a result of the "unclean" power-down, or is this something else?
Even with the help of the excellent search feature of this forum - and Google - I couldn't seem to generate a "hit" on a topic similar to this, but I'm sure that it can be explained with a link.
73,
Clint
KA7OEI
P.S. Both the admin and debian user passwords had, long ago, been reset for the default.
Comments
What I'm more interested in is this "Y/N prompt" business. It is true that the Kiwi Makefile does a number of "apt-get -y install ..." commands during an update. This normally has a "Y/N"-style prompt before it installs anything. The "-y" flag guarantees that it never asks and always assumes "yes". But the "-y" is missing in one of the apt-get commands in the Makefile (will be fixed in the next release).
BUT this should make no difference as the Kiwi server is run in the background and apt-get will never prompt unless there is a controlling terminal on which it can ask the question. I tried this myself. Intentionally removing some needed packages and removing the "-y" from the Makefile. The apt-get never got stuck. It worked just like it's supposed to.
So I can't imaging how it is your friend saw a prompt from a command running in the background after simply logging in. He must have done something else? It just doesn't make any sense. If the apt-get that is currently missing the -y were actually stuck prompting there would be hundreds of effected Kiwis worldwide. And there don't seem to be. If he has any additional information I would like to know..
Previously, when all three Kiwis were offline, we were able to attribute it to the pair of linear power supplies (in redundant "wire and" diode configuration) crowbarring - which we have occasionally seen due to our extremely dirty mains power (which sometimes goes as low as 90 and as high as 140 volts for seconds at a time.)
The friend that was up there who says he saw the "Y/N" prompt had been up there a few times before finally getting them online - and he'd only power-cycled them a few times, which correlates with your logs.
One of the possibilities that we'd considered was that the network routing had been messed up - but these are on the same switch as three Northern Utah WebSDR servers, which never had connectivity issues, and the network guy *swears* that the routing wasn't changed, which is borne out by the fact that you seemed to see entries in the logs.
Perhaps the most interesting thing about this is that a number of times, including 5-10 minutes after my friend had power-cycled the the Kiwis, I'd checked the ARP table on "websdr3.utahsdr.org" - a machine which is on the same switch - and I could always see the KiwiSDRs before (and can right now) - but during the time period that they were "stuck", I *never* saw any of them on the ARP table, let alone able to ping them with their internal "10" network address from that machine, which is in the same subnet as the WebSDR servers themselves.
At this point, I don't know what to think if I take both the guy that says he pressed "N" and the other (network) guy at their words.
Let us hope that this does not happen again!
Thanks for your hard work.
73,
Clint
KA7OEI
I spit acid at my laptop screen when you guys don't give me enough information/context. It's absolutely maddening!
I have not talked to him since and that is all I know: I will refrain from speculating on the specifics.
Clint