Out of memory: Killed process 16929 (kiwid) [fixed in v1.645]
Every few days, one of my kiwis seems to be running out of memory.
All of my kiwis are running the same debian version, the same linux kernel and the same kiwid version, but it happens only to one of them.
I even added a small swap file (128 MB), but it still seems to be running out of memory.
What could be causing this?
The only thing different on this kiwi is, that I'm running remotely a script, which is using kiwirecorder to log the amplitude of several VLF transmitters continuously in a loop, so a lot of short kiwiclient connections in succession:
command = f"~/git/kiwiclient/kiwirecorder.py -q -s kiwi2 -f {frequency} -m am -L -100 -H 100 --S-meter=30 --tlimit=6 --connect-timeout=2 --nb"
Comments
How many connections are we talking about over the total runtime of kiwid before it crashes?
Maybe there's a small memory leak per connection that normally goes unnoticed. But in your case adds up because there are thousands of connections instead of hundreds for example.
If this is the case then it is something I could try replicating here. But I don't want to waste time doing that if that's not what's happening with your setup.
That would be about 13000 per day, so maybe 20000 in total before it runs out of memory
I'll try to write a script to see if I can trigger it faster tomorrow. But I can see now that the memory usage is steadily increasing.
No hurry.
Okay, well that's pretty definitive. I should be able to duplicate it here fairly quickly. Doing a "vmstat 1" is a pretty good way to see the free mem count go down if kiwid is leaking. That plus htop.
If you run it in a loop like this, it is making about 1 connection per second, using up about 1.5 MB per minute:
while true; do ~/git/kiwiclient/kiwirecorder.py -q -s kiwihost -f 28 -m USB --S-meter 1; done
The settings (modulation/bandwidth) don't seem to matter.
I found it. I just need to get a release together..
Excellent, thank you.
Okay, should be fixed in today's v1.644 release.
Good morning
after the latest update v1.644, by opening the DX tab of the administrator menu, the server closes automatically, not allowing any operation, and then repeats the operation endlessly while it is trying to load the list of broadcasters, not allowing me to load the list of labels I saved last time.
How is this problem solved?
and if I try to delete a label from the browser the device stops and restarts for any type of operation that is carried out on the labels
Anyone else having this problem? It worked fine with all my testing: BBG/AI64, Debian 8/11/12.
I can confirm the problem. Seems that server restarts whenever local DX labels are touched in some way.
The fastest way for me to fix this is to be able to ssh into someone's Beagle (that has the problem) and run the Kiwi server under the debugger. Would require ssh port to be opened on router and Kiwi serial number emailed to support@kiwisdr.com Also that the Kiwi already be publicly listed.
As soon as possible as it's already 9PM here.
Thank you
Unfortunately, my Kiwi is currently not publicly listed. Sorry. But I could arrange it if nobody else can help faster.
What I could do is temporarily put it on the proxy once I've connected via ssh. And then remove it when I'm finished. That wouldn't require you to do anything extra.
I sent the request via email
I still need someone (who has the problem) who knows how to add a NAT/port-mapping entry to their router to map their public IP address, using some port number (22 or similar), to the local IP of their Kiwi at port 22.
Example: 1.2.3.4:9022 (TCP, public IP) => 192.168.1.2:22 (TCP, local IP)
Your public IP address has to be able to accept incoming connections.
I can't simply connect as Kiwi admin to debug this problem. I have to connect to Linux and run the Kiwi server under a Linux debugger, something you can't do when connected as a Kiwi admin (chicken and egg problem).
John, you got mail. Sorry for the delay, I was interrupted.
Fixed in v1.645. No damage should have occurred to your dx.json file.
Thanks everyone for your help.