kiwid takes 100% CPU, occasionally starves system
Hello,
I recently set a wikisdr up and I have 2-3 issues I'd like to talk about. I will open a different issue for each one, I hope this is the right way to use this forum.
The first and foremost issue I see is that kiwid sometimes hangs and requires me to manually issue a
One thing I noticed is that when one or more users are connected to the site CPU usage is around 70%, when instead the website is idle the CPU is constantly at around 95-100% (maybe be normal or unrelated... still pretty strange).
I have no visitors except for myself and the /var/log/messages says nothing of interest.
Any suggestions?
Thanks!
I recently set a wikisdr up and I have 2-3 issues I'd like to talk about. I will open a different issue for each one, I hope this is the right way to use this forum.
The first and foremost issue I see is that kiwid sometimes hangs and requires me to manually issue a
service kiwid restart. Today, on the first halt day of operation, it already happened twice. On the first episode just the website went unresponsive while on the second both the website and sshd were extremely slow (ssh was so slow it took me 4-5 minutes to issue the kiwid restart command).One thing I noticed is that when one or more users are connected to the site CPU usage is around 70%, when instead the website is idle the CPU is constantly at around 95-100% (maybe be normal or unrelated... still pretty strange).
I have no visitors except for myself and the /var/log/messages says nothing of interest.
Any suggestions?
Thanks!
This discussion has been closed.
Comments
I wonder if the KiwiSDR was updating after the new install? If that isn't it, I'd take a hard look at the network cable connections from the router to the KiwiSDR. Sometimes the crimp connections on the RJ45 jacks and plugs back off or didn't complete and an intermittent joint will cause retrys endlessly.
Ron
KA7U
I believe there were no updates.
If the eth cable was the cause I wouldn't have been able to login via sshd. But I have...
Alain
Intermittent is the worst. Your KiwiSDR is currently working, so that is something to consider. Last night the noise on my dipole was horrible, and this morning it was still horrible. I checked the feedline out to the antenna, then started in on the last links to the KiwiSDR. I have a switch box that contains notch filters and lets me move the KiwiSDR from direct connect to the antenna to a connect through the ORION II so I can use it when transmitting. I disconnected the coax from the port that feeds direct and when I reconnected it, the noise went away. I can only believe that the center pin on the PL259 was not making a solid connection and unplugging and replugging cleaned it a bit. I've had slow downs on this computer I'm typing away on, and when I decided something was up, and replugged the ethernet cable connecting the PC to the switch, the DSL speed test went back to good again. Wiggled the cable at the jack and it went bad, made a new cable and life was good again. This stuff drives me nuts.
Ron
KA7U
ssh and other Linux commands being that slow (minutes!) is not normal. Even a fault in the Kiwi software cannot cause this as Linux time slicing (20 milliseconds) will always allow an unrelated command sufficient cpu cycles. It has to be some gross error like a network reliability error, as Ron points out, or filesystem/eMMC damage causing excessive retries during file read/write operations.
For now I am restarting kiwid twice per day (I really cannot afford sshd to go down on me as this is a remote setup!)
Is the SSH public port on 22?
A few things spring to mind but the way you connect will probably count out most of them but figured I'd ask.
Stu
I have used the same setup for years with a Rapberry PI, never had any issues with it
I suppose something like a tiny Mikrotik router (E.G. HAP Lite) could be added to enable you have many more route options that can't load the Kiwi.
I ran one of those off the Kiwi USB port for a while as the Wifi bridge (on the stock 1.5A PSU) as it has four ports you could easily use it for two or three devices.
You are probably right it's not in the method but I like to have a modular approach to be able to break it down if things are failing.
Does the WAN stay solid, Does the Router drop packets? etc. - get some nice little graphs or sniff traffic all for (in the UK) about £20 and 3W of power.
it's possible that a "bot" could be "brute forcing" port 22 if it's open to the public (not sure based on your comment).
this could cause the issue you describe.
I don't leave port 22 on the kiwi open to the public, instead, I access ssh to the kiwi from a remote pc using team viewer.
seeing the default login and password and the possibility that anyone can access it and do whatever they want (especially if the kiwi is connected directly to a WAN) is an eye opener.
I have tested some public kiwis and observed port 22 open to the public with default login and password, there's many out there configured in this manner.
a knowledgeable Debian/kiwi user with nefarious intent could really trash the Debian OS/kiwi project folder beyond what the backup uSD can provide.
I strongly recommend filtering port 22 from public access, else providing a randomized ssh password for each kiwi owner displayed in the admin web interface for those advanced users who want SSH access.
yes the default password can be changed by the knowledgeable user but does it revert back to default after an update ?
additionally, not all kiwi owners are knowledgeable enough to secure access to port 22 and many aren't even aware of it.
port 22 is constantly being scanned by bots.
If someone has purposefully configured NAT on their router to forward port 22 to the Kiwi without setting a Debian root password, well, they deserve what they get. I highly doubt there are "many" installations in this condition.
even with proper ssh configuration using port 22 on Kiwi, a bot will find the port and brute force it until it gets in.
the brute-forcing can cause network issues for the Kiwi.
just a suggestion.
you are right JKS, there isnt that many but there are a number of them configured in this manner.
in this case, the ones i know of, i will notify the owners of it (the ones that have SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3) seen upon straight connection)
Ron
KA7U
it's a strong possibility especially for those kiwi owners who have the port from the beagle green open/forwarded to the WAN.
I think changing the default port to something less common is an excellent step ahead of the curve.
I am surprised that anyone would actually leave the Kiwi open via SSH intentionally with no security, that is madness.
I expect the culprit is "Plug and Pray", if that is enabled through the admin option - "Auto add NAT rule on firewall / router?" and the feature is enabled on the router then suddenly things get very wild west.
It's also the case generally that most security gets outdated (or exploits found) which is why I try to stick to brands with decent after sales support. The Asus routers have Merlin or DD-WRT etc as an alternative firmwares so can be excellent value and updated well after some other brands.
Ron,
I did get from your comment that you were aware of the risk of public 22 so that was unlikely to be the case but, like Elitedata, that was my initial thought "is it getting hammered, and who else is using it".
even if an advanced kiwi owner installs a different flavor of SSH on the same port (I've seen this), the hardware saturation is still susceptible to saturation from botnets.
bots are programmed for common ports, the solution for an advanced user is to simply use a less commonly known port.
I prefer to remotely log into a PC that's on the same LAN as the kiwi and use SSH from there.
I'd rather not have any ports out to the WAN except port 80 (I use 80 instead of 8073)