Cape Overlay Upgrade [spidev0 broken with Debian 8.10 upgrade, manual workaround using v1.175]
Hi, I have had a KiwiSDR running great since initial installation a few months ago. I maintain some Linux servers, check-for and install any updates on them daily, and as part of that routine have been doing the same for BeagleBone via the KiwiSRD admin system console, like this:
sudo apt-get update
sudo apt-get dist-upgrade
Lots of upgrades have been installed that way without incident until the last time I did that. Since then the KiwiSDR wont start. I was distracted by something during that last upgrade and had just noticed that one of the upgrades was to a cape overlay as I was typing 'Y' to continue. I realized immediately that I should have investigated compatibility before installing, but installation had started and it was too late. After an OS reboot the BeagleBone and NIC lights flashed normally, but the web browser couldnt connect to the receiver or admin system. Power off/on restarts didnt help.
That was a couple days ago and this is the first time I have had time to work on the problem. I suppose one fix might be restore from the supplied micro-SD card, but I am wondering whether that is the best alternative. I have never tried to connect to BeagleBone from an SSH client, so I can try that to see whether the Linux operating system is still running. If it is, it should be possible to uninstall the new cape overlay and reinstall the prior one. However, I have had no prior experience with BeagleBone and dont know the cape overlay version previously installed or where to obtain it.
Any advice will be appreciated.
sudo apt-get update
sudo apt-get dist-upgrade
Lots of upgrades have been installed that way without incident until the last time I did that. Since then the KiwiSDR wont start. I was distracted by something during that last upgrade and had just noticed that one of the upgrades was to a cape overlay as I was typing 'Y' to continue. I realized immediately that I should have investigated compatibility before installing, but installation had started and it was too late. After an OS reboot the BeagleBone and NIC lights flashed normally, but the web browser couldnt connect to the receiver or admin system. Power off/on restarts didnt help.
That was a couple days ago and this is the first time I have had time to work on the problem. I suppose one fix might be restore from the supplied micro-SD card, but I am wondering whether that is the best alternative. I have never tried to connect to BeagleBone from an SSH client, so I can try that to see whether the Linux operating system is still running. If it is, it should be possible to uninstall the new cape overlay and reinstall the prior one. However, I have had no prior experience with BeagleBone and dont know the cape overlay version previously installed or where to obtain it.
Any advice will be appreciated.
Comments
When I first did an "apt-get update; apt-get upgrade" to get 8.10 I had the same SPIDEV problems. After a while I figured out that "apt-get upgrade" is not the same as "apt upgrade" (apt upgrade includes --with-new-pkgs). Once I used apt everything was fine. Some packages were being kept back (e2fslibs e2fsprogs ifupdown libsystemd0 libudev1 systemd udev) when I was incorrectly using upgrade with apt-get.
The man page seems to imply apt-get dist-upgrade is equivalent to apt upgrade. So I don't understand why you had a problem. Did you do an apt-get update first? Is repos.rcn-ee.com in the package index list?
'apt-get upgrade' only upgrades to the latest versions of whatever is already installed. However, sometimes new versions have different dependencies and will fail to work correctly because one or more newly-dependent code modules are not already installed. Sometimes the reverse is true, where an old version was dependent something that is no longer needed and can be removed.
'apt-get dist-upgrade' was created to automatically deal with those two situations. It checks new-version vs. old-version dependencies, installs anything that isn't already installed that is now needed, and uninstalls anything that is no longer needed by anything installed. It is therefore by far the best of the two for most people to use. The only case I can think of where 'apt-get upgrade' would be better is where a developer who is very knowledgeable about dependencies and knows exactly what is already installed wants to temporarily leave currently-unneeded modules installed to facilitate quick comparative testing between versions.
I haven't had time to connect via an ssh client yet and so don't know whether repos.rcn-ee.com is in the package index list.
Prior to today I had only accessed the BeagleBone OS command line from the KiwiSDR admin console. Because KiwiSDR is no longer working, I connected today via a Bitvise SSH client running on a Windows 7 desktop.
I first re-ran the following to see whether "bb-cape-overlays didnt get upgraded for some reason:
sudo apt-get update
sudo apt-get dist-upgrade
This was the result:
========================================
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
bb-cape-overlays
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 65.1 kB of archives.
After this operation, 2048 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://repos.rcn-ee.com/debian/ jessie/main bb-cape-overlays armhf 4.4.20180223.0-0rcnee0~jessie+20180223 [65.1 kB]
Fetched 65.1 kB in 0s (208 kB/s)
(Reading database ... 25075 files and directories currently installed.)
Preparing to unpack .../bb-cape-overlays_4.4.20180223.0-0rcnee0~jessie+20180223_armhf.deb ...
Unpacking bb-cape-overlays (4.4.20180223.0-0rcnee0~jessie+20180223) over (4.4.20180221.0-0rcnee0~jessie+20180221) ...
Setting up bb-cape-overlays (4.4.20180223.0-0rcnee0~jessie+20180223) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.120+deb8u3-1rcnee0~bpo80+20170219) ...
update-initramfs: Generating /boot/initrd.img-4.4.9-ti-r25
cp: cannot stat '/etc/modprobe.d/*': No such file or directory
debian@kiwisdr:/$
========================================
Sure enough, bb-cape-overlays was shown as needing to be upgraded, so I expected that would fix the problem. Unfortunately, it didnt. KiwiSDR still didnt work.
Notice this line at the bottom:
cp: cannot stat '/etc/modprobe.d/*': No such file or directory
I have never noticed that before when doing upgrades and am not sure whether or not it is new. I went to /etc/modprobe.d/ and found the directory exists, but is empty.
Next I went to the OS syslog and found the following sequence repeated continuously:
========================================
Feb 27 05:52:49 kiwisdr systemd[1]: kiwid.service: Service hold-off time over, scheduling restart.
Feb 27 05:52:49 kiwisdr rsyslogd-2007: action 'action 17' suspended, next retry is Tue Feb 27 05:54:19 2018 [try http://www.rsyslog.com/e/2007 ]
Feb 27 05:52:49 kiwisdr systemd[1]: Stopped kiwi daemon.
Feb 27 05:52:49 kiwisdr systemd[1]: Starting Cape Manager Service...
Feb 27 05:52:49 kiwisdr systemd[1]: Started Cape Manager Service.
Feb 27 05:52:49 kiwisdr systemd[1]: Starting kiwi daemon...
Feb 27 05:52:49 kiwisdr kiwid[12935]: DEBIAN 8
Feb 27 05:52:49 kiwisdr kiwid[12935]: USE_SPIDEV
Feb 27 05:52:49 kiwisdr kiwid[12935]: LOAD_SPI = no
Feb 27 05:52:49 kiwisdr kiwid[12935]: Starting kiwid
Feb 27 05:52:49 kiwisdr kiwid[12935]: Start kiwid: OK
Feb 27 05:52:49 kiwisdr kiwid[12935]: Tue Feb 27 05:52:49 UTC 2018
Feb 27 05:52:49 kiwisdr systemd[1]: Started kiwi daemon.
Feb 27 05:52:49 kiwisdr kiwid: 0:00:00 .... KiwiSDR v1.173 --------------------------------------------------------------------
Feb 27 05:52:49 kiwisdr kiwid: 0:00:00 .... compiled: Feb 14 2018 03:35:11
Feb 27 05:52:49 kiwisdr kiwid: 0:00:00 .... -debian 8
Feb 27 05:52:49 kiwisdr kiwid: 0:00:00 .... /etc/debian_version 8.10
Feb 27 05:52:49 kiwisdr kiwid: 0:00:00 .... background mode: delaying start 30 secs...
Feb 27 05:53:19 kiwisdr kiwid: 0:00:30 .... reading configuration from file /root/kiwi.config/kiwi.json: 125 tokens
Feb 27 05:53:19 kiwisdr kiwid: 0:00:30 .... reading configuration from file /root/kiwi.config/admin.json: 67 tokens
Feb 27 05:53:20 kiwisdr kiwid: 0:00:30 .... serial number from EEPROM: 2010
Feb 27 05:53:20 kiwisdr kiwid: 0:00:30 .... reading configuration from file /root/kiwi.config/dx.json: 7052 tokens
Feb 27 05:53:20 kiwisdr kiwid: 0:00:30 .... 883 dx entries
Feb 27 05:53:20 kiwisdr kiwid: 0:00:30 .... listening on default port 8073/8073 for "openwebrx"
Feb 27 05:53:20 kiwisdr kiwid: 0:00:30 .... webserver for "openwebrx" on port [::]:8073
Feb 27 05:53:20 kiwisdr kiwid: 0:00:31 .... ### using SPI_DEV
Feb 27 05:53:20 kiwisdr kiwid: SYS_PANIC: "open spidev" (platform/beaglebone_black/spi_dev.cpp, line 72): No such file or directory
Feb 27 05:53:21 kiwisdr systemd[1]: kiwid.service: Main process exited, code=exited, status=255/n/a
Feb 27 05:53:21 kiwisdr kiwid[12945]: DEBIAN 8
Feb 27 05:53:21 kiwisdr kiwid[12945]: USE_SPIDEV
Feb 27 05:53:21 kiwisdr kiwid[12945]: LOAD_SPI = no
Feb 27 05:53:21 kiwisdr kiwid[12945]: Stopping kiwid:
Feb 27 05:53:21 kiwisdr systemd[1]: kiwid.service: Unit entered failed state.
Feb 27 05:53:21 kiwisdr systemd[1]: kiwid.service: Failed with result 'exit-code'.
========================================
Notice that the SYS_PANIC that causes the exit is reportedly due to a file not being found in a "beaglebond_black" directory. I purchased a seeed package with a BeagleBone Green card and the software pre-installed, so I am wondering if the "bb-cape-overlays upgrade that was just installed is for a BeagleBone Black card, but if so, why didn't you have that problem?
================================================
debian@kiwisdr:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 99M 2.7M 96M 3% /run
/dev/mmcblk0p1 3.6G 782M 2.6G 23% /
tmpfs 247M 0 247M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 247M 0 247M 0% /sys/fs/cgroup
================================================
The filesystem wasn't currently full, but I attempted to remove any /tmp/core* files anyway as you recommended.
================================================
debian@kiwisdr:~$ sudo rm /tmp/core*
rm: cannot remove '/tmp/core*': No such file or directory
================================================
I then went to /tmp, tried to list the files, and the total was 0.
================================================
debian@kiwisdr:/tmp$ ls -l
total 0
================================================
I am quite sure that the OS automatically deletes everything in /tmp when it boots and that is why temporary files that need to survive a reboot are supposed to be saved in /var/tmp.
About 20-minutes after first running 'df -h' I ran it again to see if filesystem usage had grown.
================================================
debian@kiwisdr:/$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 99M 4.0M 95M 4% /run
/dev/mmcblk0p1 3.6G 781M 2.6G 23% /
tmpfs 247M 0 247M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 247M 0 247M 0% /sys/fs/cgroup
================================================
As you can see, /run usage had increased 1%, but everything else was the same.
Following that I did this:
================================================
debian@kiwisdr:/$ sudo apt-get purge bb-cape-overlays
sudo: unable to resolve host kiwisdr
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
bb-cape-overlays*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 2273 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 25076 files and directories currently installed.)
Removing bb-cape-overlays (4.4.20180223.0-0rcnee0~jessie+20180223) ...
dpkg: warning: while removing bb-cape-overlays, directory '/lib/firmware' not empty so not removed
================================================
Then after running 'sudo apt-get update' to be sure the package lists were current I attempt to reinstall.
================================================
debian@kiwisdr:/$ sudo apt-get install bb-cape-overlays
sudo: unable to resolve host kiwisdr
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
bb-cape-overlays
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 65.1 kB of archives.
After this operation, 2273 kB of additional disk space will be used.
Get:1 http://repos.rcn-ee.com/debian/ jessie/main bb-cape-overlays armhf 4.4.20180223.0-0rcnee0~jessie+20180223 [65.1 kB]
Fetched 65.1 kB in 0s (173 kB/s)
Selecting previously unselected package bb-cape-overlays.
(Reading database ... 24914 files and directories currently installed.)
Preparing to unpack .../bb-cape-overlays_4.4.20180223.0-0rcnee0~jessie+20180223_armhf.deb ...
Unpacking bb-cape-overlays (4.4.20180223.0-0rcnee0~jessie+20180223) ...
Setting up bb-cape-overlays (4.4.20180223.0-0rcnee0~jessie+20180223) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.120+deb8u3-1rcnee0~bpo80+20170219) ...
update-initramfs: Generating /boot/initrd.img-4.4.9-ti-r25
cp: cannot stat '/etc/modprobe.d/*': No such file or directory
debian@kiwisdr:/$
================================================
Like before there were no files in '/etc/modprobe.d/'.
I rebooted and KiwiSDR still doesn't run.
There had been repeated warnings during Linux command-line operations that the kiwisdr host name could not be resolved. I wondered whether that might have caused something to go wrong during my Debian upgrades, but not yours, because (I think) you use Apple's non-standard Bonjour system for network host name resolution and I don't. So, I fixed the name resolution problem by simply pointing the kiwisdr host name to the localhost address in the '/etc/hosts' file, like this:
127.0.0.1 localhost
127.0.0.1 kiwisdr
That eliminated the warning messages and should be added to avoid potential problems, but it didn't fix my problem. I re-loaded the old Kiwi sd-card distribution again and once again the receiver worked normally until I upgraded to Debian 8.10.
'/var/log/syslog' records have contained the same 'kiwid: SYS_PANIC: "open spidev" (platform/beaglebone_black/spi_dev.cpp, line 72): No such file or directory' line after each Debian upgrade. This was copied from syslog following the latest upgrade:
==============================
Mar 2 22:29:19 kiwisdr kiwid: 0:00:00 .... background mode: delaying start 30 secs...
Mar 2 22:29:34 kiwisdr minissdpd[1742]: 3 new devices added
Mar 2 22:29:34 kiwisdr minissdpd[1742]: 1 new devices added
Mar 2 22:29:34 kiwisdr minissdpd[1742]: 1 new devices added
Mar 2 22:29:34 kiwisdr minissdpd[1742]: 1 new devices added
Mar 2 22:29:34 kiwisdr minissdpd[1742]: 1 new devices added
Mar 2 22:29:34 kiwisdr minissdpd[1742]: 1 new devices added
Mar 2 22:29:49 kiwisdr kiwid: 0:00:30 .... reading configuration from file /root/kiwi.config/kiwi.json: 125 tokens
Mar 2 22:29:49 kiwisdr kiwid: 0:00:30 .... reading configuration from file /root/kiwi.config/admin.json: 69 tokens
Mar 2 22:29:50 kiwisdr kiwid: 0:00:30 .... serial number from EEPROM: 2010
Mar 2 22:29:50 kiwisdr kiwid: 0:00:30 .... reading configuration from file /root/kiwi.config/dx.json: 7052 tokens
Mar 2 22:29:50 kiwisdr kiwid: 0:00:30 .... 883 dx entries
Mar 2 22:29:50 kiwisdr kiwid: 0:00:30 .... listening on default port 8073/8073 for "openwebrx"
Mar 2 22:29:50 kiwisdr kiwid: 0:00:30 .... webserver for "openwebrx" on port [::]:8073
Mar 2 22:29:50 kiwisdr kiwid: 0:00:31 .... ### using SPI_DEV
Mar 2 22:29:50 kiwisdr kiwid: SYS_PANIC: "open spidev" (platform/beaglebone_black/spi_dev.cpp, line 72): No such file or directory
Mar 2 22:29:51 kiwisdr systemd[1]: kiwid.service: Main process exited, code=exited, status=255/n/a
Mar 2 22:29:51 kiwisdr kiwid[1825]: DEBIAN 8
Mar 2 22:29:51 kiwisdr kiwid[1825]: USE_SPIDEV
Mar 2 22:29:51 kiwisdr kiwid[1825]: LOAD_SPI = no
Mar 2 22:29:51 kiwisdr kiwid[1825]: Stopping kiwid:
Mar 2 22:29:51 kiwisdr systemd[1]: kiwid.service: Unit entered failed state.
Mar 2 22:29:51 kiwisdr systemd[1]: kiwid.service: Failed with result 'exit-code'.
Mar 2 22:30:01 kiwisdr systemd[1]: kiwid.service: Service hold-off time over, scheduling restart.
Mar 2 22:30:01 kiwisdr systemd[1]: Stopped kiwi daemon.
Mar 2 22:30:01 kiwisdr systemd[1]: Starting Cape Manager Service...
Mar 2 22:30:02 kiwisdr systemd[1]: Started Cape Manager Service.
Mar 2 22:30:02 kiwisdr kiwid[1837]: DEBIAN 8
Mar 2 22:30:02 kiwisdr systemd[1]: Starting kiwi daemon...
Mar 2 22:30:02 kiwisdr kiwid[1837]: USE_SPIDEV
Mar 2 22:30:02 kiwisdr kiwid[1837]: LOAD_SPI = no
Mar 2 22:30:02 kiwisdr kiwid[1837]: Starting kiwid
Mar 2 22:30:02 kiwisdr kiwid[1837]: Start kiwid: OK
Mar 2 22:30:02 kiwisdr kiwid[1837]: Fri Mar 2 22:30:02 UTC 2018
Mar 2 22:30:02 kiwisdr systemd[1]: Started kiwi daemon.
Mar 2 22:29:51 kiwisdr kiwid[1825]: USE_SPIDEV
Mar 2 22:29:51 kiwisdr kiwid[1825]: LOAD_SPI = no
Mar 2 22:29:51 kiwisdr kiwid[1825]: Stopping kiwid:
Mar 2 22:29:51 kiwisdr systemd[1]: kiwid.service: Unit entered failed state.
Mar 2 22:29:51 kiwisdr systemd[1]: kiwid.service: Failed with result 'exit-code'.
Mar 2 22:30:01 kiwisdr systemd[1]: kiwid.service: Service hold-off time over, scheduling restart.
Mar 2 22:30:01 kiwisdr systemd[1]: Stopped kiwi daemon.
Mar 2 22:30:01 kiwisdr systemd[1]: Starting Cape Manager Service...
Mar 2 22:30:02 kiwisdr systemd[1]: Started Cape Manager Service.
Mar 2 22:30:02 kiwisdr kiwid[1837]: DEBIAN 8
Mar 2 22:30:02 kiwisdr systemd[1]: Starting kiwi daemon...
Mar 2 22:30:02 kiwisdr kiwid[1837]: USE_SPIDEV
Mar 2 22:30:02 kiwisdr kiwid[1837]: LOAD_SPI = no
Mar 2 22:30:02 kiwisdr kiwid[1837]: Starting kiwid
Mar 2 22:30:02 kiwisdr kiwid[1837]: Start kiwid: OK
Mar 2 22:30:02 kiwisdr kiwid[1837]: Fri Mar 2 22:30:02 UTC 2018
Mar 2 22:30:02 kiwisdr systemd[1]: Started kiwi daemon.
Mar 2 22:30:02 kiwisdr kiwid: 0:00:00 .... KiwiSDR v1.174 --------------------------------------------------------------------
Mar 2 22:30:02 kiwisdr kiwid: 0:00:00 .... compiled: Mar 2 2018 21:25:05
Mar 2 22:30:02 kiwisdr kiwid: 0:00:00 .... -debian 8
Mar 2 22:30:02 kiwisdr kiwid: 0:00:00 .... /etc/debian_version 8.10
Mar 2 22:30:02 kiwisdr kiwid: 0:00:00 .... background mode: delaying start 30 secs...
==============================
Do you have any new ideas about what is causing this?
Thanks for investigating and finding the problem.
There are lots of widely-publicized vulnerabilities in the old distribution. It is good to be up-to-date. Thanks again for your help.