v1.707
UPDATE: For now please do not follow the instructions below. The v1.707 version I just released seems to be updating okay. It is not using the binary update feature so will take 20 - 30 minutes to build.
----
Unfortunately, the github.com problems we've had with Kiwis using the original Debian 8 distro have now spread to all versions of Debian (i.e. 9, 10, 11).
A fix for this issue is in the v1.706 release. But the problem is your Kiwi running a version prior to v1.706 can't auto-update to get the fix. You have to manually force an update to get v1.706. After that auto-updates can continue (hopefully).
Proceed as follows:
Go to the admin update tab. Click the "Check now" button. If the status message says "Git clone damaged!" then click the button again. You should see a spinning icon and "Checking for software update". It may take afull minute or twofor the spinning to stop and the message "Available version: v1.706" (or later) to appear. It is possible you may have to click the button several times if you continue to get "Git clone damaged!" messages. Some have reported needing to click the button almost a dozen times.Now click the "Build now" button . The Kiwi will restart when the build completes in 20 - 30 minutes.
When the Kiwi restarts it should be running v1.707 (or later).
Comments
Kiwi V2 Debian 11.11the message about the available version appeared after 4 clicks, but now I see this:
I'm trying now with Kiwi V1 D11.11 but I'm clicking 9 times and still Git clone damaged!
-edit: Kiwi v1 worked after 11 clicks and started updating, Kiwi v2 still the same: Filesystem is FULL!
edit: 2 Kiwi v1 Installed 1.706 in about 5 minutes and works stably
edit3: the last one reacted after !27 or 28! clicks, I've already lost count
For the Kiwi-2, it's possible that the log files are huge. Possibly from update retries and also a bot that has been connecting repeatedly on a number of Kiwis filling up the logs.
In the admin console type "m clean_logs" and note the amount of before and after filesystem free space. Also type "d." to see the actual filesystem free space.
For the Kiwi-1 try in the admin console the command "git fetch origin". If that shows error messages try again a few times. Paste errors here or email to support@kiwisdr.com
clean logs in console solved the problem, Kiwi V2 started updating
Thanks for the report.
11 clicks for Kiwi-1 D11.11? That's not good. Means my workaround in v1.706 may not be good enough (I just retry 3 times which is what I observed to work).
I should probably do the log clean automatically the first time a full filesystem is detected.
I had to click six times to get a clean clone.
We're just going to have to move off of github.com onto our own hosted Git server. But that's a large amount of work and comes at a very, very bad time. I probably should have done that much sooner..
Okay, let me do a v1.707 release that addresses both these issues before too many people follow this procedure. Thank you!
Ugh. This situation is much worse than I thought.
I forgot that git keeps deltas of all changes to files so you can step to an arbitrary commit date and see the files as they were. But this means for the new ~30MB binary update files it simply keeps all the old ones. It doesn't delete them. Now the total git clone size is expanding at an alarming rate. This is almost certainly a contributor to the recent problems.
I'll need some time to fix this.
Is it worth standing up a private git?
I might be able to help as I have a proxmox cluster in Auckland.
I've already got two VPS that I'm paying for (website, proxy service, forum, file server, TDoA backend, ...) Plenty of storage, but the bandwidth costs are depressing (over 3.5TB/month now due to the included proxy service with Kiwi-2).
What's scaring the crap out of me are these tools that are supposed to help remove large blobs from a git repo (git-filter-repo et al). It seems you can do great damage to your repo if you're not careful. And I don't really understand the subtleties.
I think it might be simpler to just start over with a new repo without the large files. And change all the code that references large files to grab them off kiwisdr.com out-of-tree. Like what we do now for things like the latest IP blacklist and the various frequency lists. There are things like Git Large File Storage. But I'm not sure that has any real advantage.
After running git-largest-files I found there are big files that were deleted long ago. Compressed versions of these are getting pulled over every time you do a git clone. No wonder the current .git directory is 358MB in size.
are you able to rate limit any of that traffic?
No. But I get charged on bulk transfer amount, not rate. The rate is relatively low since it's just x-hundreds of Kiwi audio/waterfall channels. Averages roughly 20Mb/s (both in/out since it's mostly proxy traffic vs everything else).
How much storage do you need?
but the bandwidth costs are depressing (over 3.5TB/month
3.5 TB/month doesn't seem to be much.
My cloud server has 20 TB/month included and it's just 4€ a month. Maybe it's worth looking for an alternative.
Thanks guys, but my VPS service is not the priority at the moment. I have to get this disastrous git situation under control before every Kiwi on the planet becomes bricked.
If we can help in any way - reach out.
In my case i had to rebuild the Kiwisoftware completly via SSH as described here:
Took hours but now i am on 1.707.
73 Holger, DF6DBF
I get this:
Installed version: v1.702, built Oct 23 2024 20:33:26
Filesystem is FULL!
I tried this:
Well, your filesystem is full. See that "Avail 0, Use% 100%" in there? Have to figure out why that is.
Start by doing "cd; du" and see who is hogging the space.
Mine updated fine recently using the original method, check for update, rebuild. (actually before i read this post)
How to fix ?
How to fix?