Is it possible for you to open an ssh port (22, via NAT) in your router so I could login and take a look? If so please send the details to support@kiwisdr.com Thanks.
However having ssh access to your updated AI-64 would really help me as I need to develop a fix that is backward compatible for people who haven't updated yet. And I'd like to keep my AI-64 as-is for that purpose.
Ugh, do not try setting your BBAI-64 Ethernet interface speed to 10 Mbps (admin network tab).
Doing this will brick the Ethernet interface. Although there is a small window after a reboot in which you can recover.
I don't know why this is. If you do an ethtool eth0 Linux claims 10 Mbps is supported. For v1.603 the 10 Mbps option will be greyed-out in the admin menu. Maybe this will be fixed in a later Debian release.
How to recover: After a reboot the Ethernet always comes up initially in auto speed mode. And so will generally negotiate to 100 Mbps with your router/switch (1 Gbps is also possible). The "downshift" to 10 Mbps only happens after the Kiwi server starts up. The server has a built-in 30 second delay when started after a Beagle reboot. So this, plus the Linux startup time, gives you a short window to ssh into the Beagle and disable the Kiwi server. Then you can manually edit the Ethernet speed configuration back to auto and restart the server.
Start a continuous ping to the Beagle. The pings will fail. Power cycle the Beagle.
As soon as the pings start going through immediately ssh into the Beagle. If you can't run a continuous ping just keep trying an ssh. Don't wait for it to timeout. If it doesn't go through immediately interrupt it and try again. You're trying to get an ssh connection at the earliest possible moment.
Then do:
cdp
mst
mdi
(now you can relax as the server is disabled)
cdk
nano kiwi.json
(change)
"ethernet_speed": 1,
(to)
"ethernet_speed": 0,
(write file and exit)
cdp
ku
The server should start back up and the Ethernet speed menu should show auto.
Thanks @engineercarl for discovering this. I'm really surprised we didn't run across this sooner.
Okay, I was just about to do a third AI-64. I take it from the above discussion that I'm safe provided that I don't set the enet speed to 10 mbps and follow the directions at the start of the thread? (Which I've done successfully twice.)
Oh, what the heck. I'll be impatient and try. What could possibly go wrong?
Err. Groan. Looks like the third time was unlucky. Everything was going fine, but "make install" exited with "Error 2". So I'm looking at this (accidentally hit control-c after the error trying to copy, so disregard that):
Per the directions I've stopped here and will wait for any advice.
Makefile is corrupted. Rather than a million back-and-forth forum posts, or emails, which eat up all my time, can you open/NAT the ssh port (22) on your router so I could login to this guy and have a look? Email pubic ip address (or domain) and Kiwi serial number to support@kiwisdr.com
That resulted in an unexpectedly short "make" command (just a few minutes). "Make install" then completed without error. But unfortunately, although halting the AI64 and rebooting seemed normal, when I removed power, attached the Kiwi and rebooted, the Kiwi server did not come up.
(This was a working Kiwi on a BBG earlier today, so I know it isn't the Kiwi. The AI64 is new.)
Rather than taking up more of your time, John, is there any reason why I can't follow the directions to flash the AI64 here on the "getting started" page (https://beagleboard.org/getting-started) and then pick up your directions at the beginning of the thread at the appropriate point? (which is....?)
Linux BeagleBone 5.10.168-ti-arm64-r103 #1bullseye SMP Fri May 12 23:47:21 UTC 2023 aarch64 GNU/Linux
Debian GNU/Linux 11 (bullseye)
I'll e-mail the lengthy MSL. I haven't knowingly opened a port since the 2000s and have brand new networking hardware, so let me look at that. In the meantime the cdp / msl will come by email, although I wonder if this couple of lines doesn't answer the question:
Jun 6 00:56:41 BeagleBone kiwid: 00:00:00.951 .... /root/kiwi.config/admin.json: required parameter not found: port
Jun 6 00:56:41 BeagleBone kiwid: PANIC: "cfg_int" (init/cfg.cpp, line 527)
Thanks for your help as always. Will send the complete msl output and IP separately (assuming I figure this new hardware out).
/root/kiwi.config/admin.json: required parameter not found: port
Oh boy, you're not the first to see this problem. That means there's a disastrous problem with v1.602. It was supposed to make config file reliability better, not worse.
Ugh. It's early, but this may actually be a bug in the Linux kernel (somewhere between 5.10.121 and 5.10.168) used by the recent BBAI-64 Debian 11.7 update. Need to verify. The other people seeing this problem all had new AI-64's running 11.7
If anyone else sees this problem, please, please, please, let me know. Whether you're running an AI-64 + Debian 11.7 or not. I've updated mine to 11.7 and I just can't seem to replicate the problem so far.
BTW, the procedure mentioned above for recovering admin.json doesn't work. It will be fixed in v1.603. We use a similar procedure to fix a corrupted kiwi.json all the time. But for admin.json the recovery file dist.admin.json is missing a key parameter.
I know you're busy, but thought you might want to tweak this eventually.
I just got my BBAI-64 in today, followed all of your instructions, but apparently, after installing bbai64-emmc-flasher-debian-11.7-xfce-arm64-2023-08-05-10gb and updating to 11.8 (according to /etc/debian_version), they renamed the directory:
Comments
Is it possible for you to open an ssh port (22, via NAT) in your router so I could login and take a look? If so please send the details to support@kiwisdr.com Thanks.
Okay, I think I know what's going on.
However having ssh access to your updated AI-64 would really help me as I need to develop a fix that is backward compatible for people who haven't updated yet. And I'd like to keep my AI-64 as-is for that purpose.
Thank you.
You got it. Expect an email shortly.
EC
A fix was found and will be in the v1.602 release.
Big thanks to @engineercarl for discovering the issue.
Ugh, do not try setting your BBAI-64 Ethernet interface speed to 10 Mbps (admin network tab).
Doing this will brick the Ethernet interface. Although there is a small window after a reboot in which you can recover.
I don't know why this is. If you do an
ethtool eth0Linux claims 10 Mbps is supported. For v1.603 the 10 Mbps option will be greyed-out in the admin menu. Maybe this will be fixed in a later Debian release.How to recover: After a reboot the Ethernet always comes up initially in auto speed mode. And so will generally negotiate to 100 Mbps with your router/switch (1 Gbps is also possible). The "downshift" to 10 Mbps only happens after the Kiwi server starts up. The server has a built-in 30 second delay when started after a Beagle reboot. So this, plus the Linux startup time, gives you a short window to ssh into the Beagle and disable the Kiwi server. Then you can manually edit the Ethernet speed configuration back to auto and restart the server.
Start a continuous ping to the Beagle. The pings will fail. Power cycle the Beagle.
As soon as the pings start going through immediately ssh into the Beagle. If you can't run a continuous ping just keep trying an ssh. Don't wait for it to timeout. If it doesn't go through immediately interrupt it and try again. You're trying to get an ssh connection at the earliest possible moment.
Then do:
cdp
mst
mdi
(now you can relax as the server is disabled)
cdk
nano kiwi.json
(change)
"ethernet_speed": 1,
(to)
"ethernet_speed": 0,
(write file and exit)
cdp
ku
The server should start back up and the Ethernet speed menu should show
auto.Thanks @engineercarl for discovering this. I'm really surprised we didn't run across this sooner.
Okay, I was just about to do a third AI-64. I take it from the above discussion that I'm safe provided that I don't set the enet speed to 10 mbps and follow the directions at the start of the thread? (Which I've done successfully twice.)
Oh, what the heck. I'll be impatient and try. What could possibly go wrong?
Yes, everything should be fine as along as you don't change to 10 Mbps. The default for that menu is
auto.Err. Groan. Looks like the third time was unlucky. Everything was going fine, but "make install" exited with "Error 2". So I'm looking at this (accidentally hit control-c after the error trying to copy, so disregard that):
Per the directions I've stopped here and will wait for any advice.
Makefile is corrupted. Rather than a million back-and-forth forum posts, or emails, which eat up all my time, can you open/NAT the ssh port (22) on your router so I could login to this guy and have a look? Email pubic ip address (or domain) and Kiwi serial number to support@kiwisdr.com
Or first maybe try this:
cd /root
df -h . (make sure there is at least 700 MB free space under "Avail" column)
mv Beagle_SDR_GPS B.bad
Then continue starting with step 5 of the instructions, "git clone" etc.
That resulted in an unexpectedly short "make" command (just a few minutes). "Make install" then completed without error. But unfortunately, although halting the AI64 and rebooting seemed normal, when I removed power, attached the Kiwi and rebooted, the Kiwi server did not come up.
(This was a working Kiwi on a BBG earlier today, so I know it isn't the Kiwi. The AI64 is new.)
Rather than taking up more of your time, John, is there any reason why I can't follow the directions to flash the AI64 here on the "getting started" page (https://beagleboard.org/getting-started) and then pick up your directions at the beginning of the thread at the appropriate point? (which is....?)
If Debian 11 runs you really don't want to risk re-flashing the AI-64. You could easily end up bricking the whole thing.
I really need to see all the output from the make as well as from the
dogcommand.And do this and let's see what the server is doing at the moment:
cdp
msl (paste, or email, the last few hundred lines of that)
Unfortunately I no longer have the output of the make.
Dog says:
Debian 11.7
BeagleBoard.org Debian Bullseye Xfce Image 2022-01-14
Linux BeagleBone 5.10.168-ti-arm64-r103 #1bullseye SMP Fri May 12 23:47:21 UTC 2023 aarch64 GNU/Linux
Debian GNU/Linux 11 (bullseye)
I'll e-mail the lengthy MSL. I haven't knowingly opened a port since the 2000s and have brand new networking hardware, so let me look at that. In the meantime the cdp / msl will come by email, although I wonder if this couple of lines doesn't answer the question:
Jun 6 00:56:41 BeagleBone kiwid: 00:00:00.951 .... /root/kiwi.config/admin.json: required parameter not found: port
Jun 6 00:56:41 BeagleBone kiwid: PANIC: "cfg_int" (init/cfg.cpp, line 527)
Thanks for your help as always. Will send the complete msl output and IP separately (assuming I figure this new hardware out).
/root/kiwi.config/admin.json: required parameter not found: port
Oh boy, you're not the first to see this problem. That means there's a disastrous problem with v1.602. It was supposed to make config file reliability better, not worse.
You can recover as follows:
cdp
cp unix_env/kiwi.config/dist.admin.json /root/kiwi.config/admin.json
It should come back up within a few minutes. Don't spend too much time customizing stuff until I can figure out what's going wrong.
Ugh. It's early, but this may actually be a bug in the Linux kernel (somewhere between 5.10.121 and 5.10.168) used by the recent BBAI-64 Debian 11.7 update. Need to verify. The other people seeing this problem all had new AI-64's running 11.7
If anyone else sees this problem, please, please, please, let me know. Whether you're running an AI-64 + Debian 11.7 or not. I've updated mine to 11.7 and I just can't seem to replicate the problem so far.
BTW, the procedure mentioned above for recovering admin.json doesn't work. It will be fixed in v1.603. We use a similar procedure to fix a corrupted kiwi.json all the time. But for admin.json the recovery file dist.admin.json is missing a key parameter.
John,
I know you're busy, but thought you might want to tweak this eventually.
I just got my BBAI-64 in today, followed all of your instructions, but apparently, after installing bbai64-emmc-flasher-debian-11.7-xfce-arm64-2023-08-05-10gb and updating to 11.8 (according to /etc/debian_version), they renamed the directory:
to just:
Kiwi's top-level Makefile, line 1107 calls for:
That was breaking make install. But when I edited that line to reflect the path change, make install worked and rebooted fine:
I have attached all of the make / make install logs in case you wanted to look.
Thanks.
P.S.: https://beagleboard.org/update-ai64 link (404) was replaced with https://docs.beagleboard.org/latest/boards/beaglebone/ai-64/update.html.
Thanks. I pushed a change that should allow compatibility with both situations. Also updated the URL of the AI64 update instructions.
Thank you, Sir.