DANGER: DO NOT do a manual Debian/Linux upgrade to your Kiwi! (update: but it's okay now)

jksjks
edited April 2020 in KiwiSDR Discussion
We're not talking about the Kiwi auto-update process. That feature is perfectly fine and will continue as usual.

We're talking about people who manually update/upgrade the underlying Debian/Linux system on the Kiwi by logging into the Beagle as root and running, for example, the "apt" command. Please don't do this. We only validate specific Debian distributions. If you update manually you run the risk of "bricking" your Kiwi server. That is, downloading something causing an unexpected side-effect which prevents the Kiwi server from running. This will stop further auto-updates and will require a manual intervention to fix. Possibly by re-flashing your Beagle using the Seeed SD-card and losing your entire Kiwi configuration (unless you've backed up).

This is exactly what has happened since a change was made in late February by one of the Beagle developers that made its way into the Debian update stream. We've identified the problem and have asked them to make a change. But it is not clear when this might happen. If you have a Kiwi in this condition then please use the following workaround:

Symptom: The Kiwi server won't stay running. The log shows it just crashes and restarts over and over again with the error "SYS_PANIC: open spidev".
Workaround:

1) ssh/PuTTY into your Kiwi Beagle as user "root" (no password).
2) "cdp" Change to Kiwi build directory.
3) "up" Manually pulls the latest version from Github (v1.175 or later) and builds it.
4) When that's done do "cd unix_env/bb.org-overlays"
5) Type "./install.sh" (note the "dot" before the slash)
6) When that's done type "reboot"

Now the Kiwi server should run after the reboot.

If you upgrade from here things will break again until the Beagle developers can implement a permanent solution.

KA7U

Comments

  • jksjks
    edited March 2018
    UPDATE: Thanks to some fast work by Beagle kernel guru Robert C Nelson the Beagle Debian update is working again with the Kiwi server.
    I.e. a newer bb-cape-overlay package should get installed if you do another upgrade:

    root@kiwisdr:~# apt-get update
    ...
    root@kiwisdr:~# apt-get dist-upgrade
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Calculating upgrade... Done
    The following packages will be upgraded:
      bb-cape-overlays
    1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Inst bb-cape-overlays [4.4.20180223.0-0rcnee0~jessie+20180223] (4.4.20180305.0-0rcnee0~jessie+20180305 rcn-ee.net:repos.rcn-ee.com [armhf])
    Conf bb-cape-overlays (4.4.20180305.0-0rcnee0~jessie+20180305 rcn-ee.net:repos.rcn-ee.com [armhf])

    So if your Kiwi server is bricked simply login as root and use the two commands above to do another upgrade. After rebooting the Kiwi server should run normally.


  • lou_lovolou_lovo
    Can it be a problem with installing the "OH1KK/KiwiSDR-antenna-switch-extension" ?
     I tried to do this install under the Console in the Kiwi but that does not work.

    When I connected to the Kiwi with SSH/Putty the installation run perfectly!
  • jksjks
    Did you post in the wrong thread? This problem has nothing to do with the antenna switch extension.

    This particular problem has been fixed, although people should still not randomly upgrade unless we have given the okay to do so.

  • lou_lovolou_lovo
    edited March 2018
    What I ment was that I was adding the antenna switch as :

    cd /root
    cd KiwiSDR-antenna-switch-extension
    bash ./ant-switch-extension-installer

    by logging in to the kiwi direct (with SSH/Putty) and not through the Console.

    I wounder if that could disturb the Kiwi/Linux installation ?

    Sorry if this ended up in wrong place :-(
    Wkr /lou_lovo

  • jksjks
    Running the antenna switch extension installation script while connected to the Beagle via the admin console tab doesn't work. This is because in the middle of the script it stops the Kiwi server to rebuild the sources. As soon as the Kiwi server is stopped it kills all the subprocesses including the script doing the installation!

    There is another script in the Kiwi distribution that has a similar problem. I added a check to tell when it was bring run in the console tab so it could print an error and exit. The same should probably be done for the antenna switch script. The correct thing to do in this case is to run these scripts while connected with ssh or PuTTY directly to the Beagle.

  • helioshheliosh
    edited February 2019
    Hi,

    Is it safe to update the linux kernel?

    https://elinux.org/Beagleboard:BeagleBoneBlack_Debian#Kernel_Upgrade
  • jksjks
    Depends on your definition of "safe". If you're going to do this I would strongly recommend using the "backup" function on the admin page to create a "flasher" sd card containing the complete Beagle distro including Kiwi customizations in the (likely) event you need to restore the Beagle filesystem. Don't use the sd card that came with the Kiwi. Use another one.

    Good luck, and please let us know what happens. Be careful you are updating to a kernel that is compatible with the Debian disto you're running.
    heliosh
  • HB9TMCHB9TMC
    I tried to update the kernel, because my usb wifi stick refuses to work on 5 GHz.
    Good news is, that 4.19.50-ti-r20 is booting and the usb stick works on 5 GHz.
    Bad news is, kiwid doesn't start: 
    root@kiwisdr:~# /etc/init.d/kiwid start
DEBIAN 7
ls: cannot access /sys/devices/bone_capemgr.*/slots: No such file or directory
USE_SPIPIO
LOAD_SPI = yes
Starting kiwid
    Then it hangs.
    Reason for it presumably being: https://elinux.org/Beagleboard:BeagleBoneBlack_Debian#Where_did_the_slots_file_go.3F

    It's not urgent for me. But in long term it might need an adaptation, as debian jessie support will end 30/6/2020.
  • jksjks
    Well, I told you so.

    Guess what I just discovered? For all of you who upgraded your Kiwis beyond the stock Debian 8.5, and then did backups to an sd card, all those backups are UNUSEABLE as a recovery re-flasher by ANY version of BeagleBone Debian. Those cards will not boot and re-flash your Kiwi like they're supposed to.

    Kiwi release v1.334 fixes this issue provided you make new backups using v1.334 or any future release. Out of the 400+ public Kiwis 60 of them are running Debian 8.6+ So this is not a small problem. PLEASE wait for the update and make new backups!

    What happened? Well, sometime after Debian 8.5 the mkfs.ext4 utility that creates "ext4" format filesystems used on sd cards and the Beagle on-board eMMC was changed to use 64-bit addressing to support huge filesystem sizes. That's fine. But the problem is that uBoot, the little program at the beginning of every bootable device, was not changed because that's not part of a Debian upgrade. That's a BeagleBone distro issue. All of this was discovered by the Beagle folks back in 2016. And Beagle distros that used Debian 8.6+ made sure to add an option to mkfs.ext4 preventing the use of 64-bit addressing as a workaround until uBoot eventually gets 64-bit support.

    But just upgrading Debian alone beyond 8.5, without the benefit of the associated Beagle distro, doesn't get this workaround. I spent a long time trying to figure out why sd cards built with Debian 8.6+ wouldn't boot. They looked fine when inspected using Debian itself. It wasn't until I learned how to inspect an sd card filesystem from uBoot that I saw they were corrupted from uBoot's point of view.
    PowernumptyWA2ZKDrz3dvp
  • ka9qka9q
    Are you tracking upstream Debian security updates? I consider frequent updates critical to system security, so if we're not supposed to do our own Debian updates then you'll have to take that over.
    HB9TMCdl7awl
  • jksjks
    edited April 2020
    So it appears the Debian project and Beaglebone.org have stopped making updates to the Debian 8 (Jessie) release. For the Beagle, Debian 8.11 seems to be the last release. It has been run by a number of customers for a long time and also by us. So it seems safe now for Kiwi owners generally to do an upgrade and get the benefit of whatever security patches there have been between 8.5 and 8.11.

    To do this login as root using ssh/PuTTY and use the following commands. (this can not be done from the Kiwi admin console tab)
    If you haven't logged in this way in a while remember that your Debian root password has likely been changed to either the Kiwi admin password or the Kiwi's serial number if no Kiwi admin password has been set. The Debian root password has not been touched if you had set it to something other than the default of having no password.
    See this post for details: http://forum.kiwisdr.com/discussion/1801/v1-354-security-improvements-debian-root-password-no-longer-unset-blank 
    cdp
mst      (stops the Kiwi server)
cd
pkup     (shell alias for "apt-get -y install debian-archive-keyring; apt-get update")
pkug     (shell alias for "apt-get -y dist-upgrade")

(lots of output, takes roughly 10 minutes)

reboot
    A few minutes after the reboot the Kiwi server should be responding again. If you login to the Beagle and type the "dog" command you should see: 
    Debian 8.11
BeagleBoard.org Debian Image 2016-05-13
Linux kiwisdr 4.4.9-ti-r25 #1 SMP Thu May 5 23:08:13 UTC 2016 armv7l GNU/Linux
    There is still the open question of how the Kiwi distribution, and thousands of Kiwi customers, will migrate to the current Debian 9 (Stretch) and/or Debian 10 (Buster) releases which are available for the Beagle. This needs time for research as there are many issues involved.
    Powernumptyrz3dvpjapmedia
  • vk2ktjvk2ktj

    Is this procedure still valid?

    During the 'pkup' step I received:

    Err https://deb.nodesource.com jessie/main armhf Packages                         
Err https://deb.nodesource.com jessie/main armhf Packages                            
Err https://deb.nodesource.com jessie/main armhf Packages
Err https://deb.nodesource.com jessie/main armhf Packages
Err https://deb.nodesource.com jessie/main armhf Packages
server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
...
...
W: Failed to fetch https://deb.nodesource.com/node_0.12/dists/jessie/main/binary-armhf/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
E: Some index files failed to download. They have been ignored, or old ones used instead.

    I have stopped at this point.

    fyi:

    root@kiwisdr:~# dog
Debian 8.5
BeagleBoard.org Debian Image 2016-05-13
Linux kiwisdr 4.4.9-ti-r25 #1 SMP Thu May 5 23:08:13 UTC 2016 armv7l GNU/Linux
-bash: lsb_release: command not found

    This is pretty much an "out of the box" Seeedstudio kiwi that has been configured in the usual way. I haven't done anything particularly adventurous yet.

  • jksjks

    It's being worked on apparently: https://github.com/nodesource/distributions/issues/1266

  • vk2ktjvk2ktj
    edited October 2021

    Thank you.

    On that same thread I have found and tested a safe work-around to this issue.

    The amended procedure to use is:

    cdp
mst      (stops the Kiwi server)
cd
pkup     (shell alias for "apt-get -y install debian-archive-keyring; apt-get update")
apt-get install ca-certificates    (manually update this package first)
pkup     (repeat, to take advantage of new certificates just installed, you should no longer see the certificate error )
pkug     (shell alias for "apt-get -y dist-upgrade")

(lots of output, takes roughly 10 minutes)

reboot

    I upgraded from Debian8.5 as shown above directly to:

    Debian 8.11
BeagleBoard.org Debian Image 2016-05-13
Linux kiwisdr 4.4.9-ti-r25 #1 SMP Thu May 5 23:08:13 UTC 2016 armv7l GNU/Linux

    Cheers

  • YO3IBZYO3IBZ

    I also updated my kernel to 4.4.155 to install RTL8811au wifi chipset, now all of my kiwi's working over wifi :)

    Thank you @jks for all your work!

    Aurel.

  • ratwolfratwolf

    v1.486 January 18, 2022 Support for Debian 10 (Buster) on BBG/BBB.

    How to proceed to migrate to Debian 10 (Buster) on BBG?

  • jksjks
    edited January 2022

    I'm working out a procedure right now. The upgrade will not be an easy "one-click" process. It will be painful, involving ssh/PuTTY access to the Beagle Debian, sd card backup/download, etc.

Sign In or Register to comment.