kiwid takes 100% CPU, occasionally starves system

Hello,

I recently set a wikisdr up and I have 2-3 issues I'd like to talk about. I will open a different issue for each one, I hope this is the right way to use this forum.

The first and foremost issue I see is that kiwid sometimes hangs and requires me to manually issue a service kiwid restart. Today, on the first halt day of operation, it already happened twice. On the first episode just the website went unresponsive while on the second both the website and sshd were extremely slow (ssh was so slow it took me 4-5 minutes to issue the kiwid restart command).

One thing I noticed is that when one or more users are connected to the site CPU usage is around 70%, when instead the website is idle the CPU is constantly at around 95-100% (maybe be normal or unrelated... still pretty strange).

I have no visitors except for myself and the /var/log/messages says nothing of interest.

Any suggestions?

Thanks!

Comments

  • Alain,
    I wonder if the KiwiSDR was updating after the new install? If that isn't it, I'd take a hard look at the network cable connections from the router to the KiwiSDR. Sometimes the crimp connections on the RJ45 jacks and plugs back off or didn't complete and an intermittent joint will cause retrys endlessly.
    Ron
    KA7U
  • Ron,
    I believe there were no updates.
    If the eth cable was the cause I wouldn't have been able to login via sshd. But I have...

    Alain
  • Alain,
    Intermittent is the worst. Your KiwiSDR is currently working, so that is something to consider. Last night the noise on my dipole was horrible, and this morning it was still horrible. I checked the feedline out to the antenna, then started in on the last links to the KiwiSDR. I have a switch box that contains notch filters and lets me move the KiwiSDR from direct connect to the antenna to a connect through the ORION II so I can use it when transmitting. I disconnected the coax from the port that feeds direct and when I reconnected it, the noise went away. I can only believe that the center pin on the PL259 was not making a solid connection and unplugging and replugging cleaned it a bit. I've had slow downs on this computer I'm typing away on, and when I decided something was up, and replugged the ethernet cable connecting the PC to the switch, the DSL speed test went back to good again. Wiggled the cable at the jack and it went bad, made a new cable and life was good again. This stuff drives me nuts.
    Ron
    KA7U
  • jksjks
    edited December 2018
    It is normal for the Kiwi software to have higher cpu usage with no users connected. This is an artifact of the internal realtime process scheduling used.

    ssh and other Linux commands being that slow (minutes!) is not normal. Even a fault in the Kiwi software cannot cause this as Linux time slicing (20 milliseconds) will always allow an unrelated command sufficient cpu cycles. It has to be some gross error like a network reliability error, as Ron points out, or filesystem/eMMC damage causing excessive retries during file read/write operations.
  • It could be filesystem/eMMC damage, in my opinion. I was thinking the same.
    For now I am restarting kiwid twice per day (I really cannot afford sshd to go down on me as this is a remote setup!)
  • Just out of interest have you tried running a constant ping to the device? how do you get to the SSH? via a VPN or firewall that allows just one or more IP address?
    Is the SSH public port on 22?
    A few things spring to mind but the way you connect will probably count out most of them but figured I'd ask.
    Stu
  • The beaglebone creates an ssh tunnel to an host in the middle and forwards a remote port to port 22. I then connect to the middle host on that port.
    I have used the same setup for years with a Rapberry PI, never had any issues with it
  • That sounds good, I was just interested if there was anything else loading it.
    I suppose something like a tiny Mikrotik router (E.G. HAP Lite) could be added to enable you have many more route options that can't load the Kiwi.
    I ran one of those off the Kiwi USB port for a while as the Wifi bridge (on the stock 1.5A PSU) as it has four ports you could easily use it for two or three devices.

    You are probably right it's not in the method but I like to have a modular approach to be able to break it down if things are failing.
    Does the WAN stay solid, Does the Router drop packets? etc. - get some nice little graphs or sniff traffic all for (in the UK) about £20 and 3W of power.
  • edited December 2018
    just a thought.
    it's possible that a "bot" could be "brute forcing" port 22 if it's open to the public (not sure based on your comment).
    this could cause the issue you describe.
    I don't leave port 22 on the kiwi open to the public, instead, I access ssh to the kiwi from a remote pc using team viewer.
    seeing the default login and password and the possibility that anyone can access it and do whatever they want (especially if the kiwi is connected directly to a WAN) is an eye opener.
    I have tested some public kiwis and observed port 22 open to the public with default login and password, there's many out there configured in this manner.
    a knowledgeable Debian/kiwi user with nefarious intent could really trash the Debian OS/kiwi project folder beyond what the backup uSD can provide.
    I strongly recommend filtering port 22 from public access, else providing a randomized ssh password for each kiwi owner displayed in the admin web interface for those advanced users who want SSH access.
    yes the default password can be changed by the knowledgeable user but does it revert back to default after an update ?
    additionally, not all kiwi owners are knowledgeable enough to secure access to port 22 and many aren't even aware of it.
    port 22 is constantly being scanned by bots.
  • The Kiwi uSD "flasher" card is bootable. It can recover a Kiwi with an eMMC in any condition besides an outright hardware failure.

    If someone has purposefully configured NAT on their router to forward port 22 to the Kiwi without setting a Debian root password, well, they deserve what they get. I highly doubt there are "many" installations in this condition.
    elitedata
  • edited December 2018
    aside from default access, changing the port to something less common on the Kiwi is a much better practice and mitigation method against bots.
    even with proper ssh configuration using port 22 on Kiwi, a bot will find the port and brute force it until it gets in.
    the brute-forcing can cause network issues for the Kiwi.
    just a suggestion.
    you are right JKS, there isnt that many but there are a number of them configured in this manner.
    in this case, the ones i know of, i will notify the owners of it (the ones that have SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3) seen upon straight connection)
  • I use openVPN for remote access. It is amazing how many bots hit the openVPN trying multiple ports. It has been going on for a few years, but so far the lock has held. So I still trust openVPN running on an Asus NT-16 router.
    Ron
    KA7U
    elitedata
  • edited December 2018
    im simply suggesting, since the beagle green is low profile hardware, its hardware could easily become over saturated with multiple incoming SSH TCP packets, considering how bot-nets operate with multiple connections to brute force the password, essentially causing a DoS to the beagle green, rendering external TCP connections to the device extremely slow or unresponsive.
    it's a strong possibility especially for those kiwi owners who have the port from the beagle green open/forwarded to the WAN.
    I think changing the default port to something less common is an excellent step ahead of the curve.
  • Did anyone say Shodan?
    I am surprised that anyone would actually leave the Kiwi open via SSH intentionally with no security, that is madness.
    I expect the culprit is "Plug and Pray", if that is enabled through the admin option - "Auto add NAT rule on firewall / router?" and the feature is enabled on the router then suddenly things get very wild west.
    It's also the case generally that most security gets outdated (or exploits found) which is why I try to stick to brands with decent after sales support. The Asus routers have Merlin or DD-WRT etc as an alternative firmwares so can be excellent value and updated well after some other brands.

    Ron,
    I did get from your comment that you were aware of the risk of public 22 so that was unlikely to be the case but, like Elitedata, that was my initial thought "is it getting hammered, and who else is using it".
  • "Auto add NAT rule" only does so for the Kiwi port -- 8073 or whatever has been configured. It does not do so for port 22 under any circumstances.
    Powernumpty
  • Correction noted, so people are manually port forwarding 22 while credentials are at default, OK (facepalm)
  • edited December 2018
    still, even with 22 secured and open to the internet, why would one do this considering the low profile of the beagle green hardware and it's potential to become saturated from multiple connections from a botnet with brute force activity ?
    even if an advanced kiwi owner installs a different flavor of SSH on the same port (I've seen this), the hardware saturation is still susceptible to saturation from botnets.
    bots are programmed for common ports, the solution for an advanced user is to simply use a less commonly known port.
    I prefer to remotely log into a PC that's on the same LAN as the kiwi and use SSH from there.
    I'd rather not have any ports out to the WAN except port 80 (I use 80 instead of 8073)
This discussion has been closed.