My KIWI's drop their users
Approximately a week ago, my 3 public KIWI's http://72.172.110.98:8173 plus 8174 and 8175 have been dropping users after just a few minutes, haven't seen anyone survive more than 10m and the audio is on and off. They had been fine for years, so I am trying to discern what has changed. I'm on a local Wireless ISP with approx 35 down and 10 up, 25ms latency and that has been fine, even with many users. Nothing has changed on my local side and I talked to my ISP and they said nothing had changed from their end and that my internet radio connection looked fine. On this same internet, I can browse and play Utube channels in HD without any issues. Only the KIWI's have the issues. I have rebooted the router and the KIWI's but nothing has helped. Other remote users on the Perseus and SDRanywhere servers haven't had issues. Anyone have any suggestions on what I can do to fix this, or any good questions to ask my ISP in case they have inadvertently changed something that has affected what the KIWI's do? I am far from an expert on any of this....
Thanks, Don
VE6JY
Comments
Do you share your antenna with a transceiver and have a protection switch in place that is switching your antenna in and out? If not, it looks like you may have a bad connection to your antenna. I had a similar issue with the center pin of the female BNC connector on my loop amp loosening up and and creating an intermittent open when the wind moved the loop.
However,
At about 2 minutes in, the Kiwi froze up (audio underrun, no audio and frozen waterfall), but came back after about 3-5 seconds. It did it a number of times before freezing up entirely at 6 mins 51 secs. I would check every connection, make sure you don't have any bad ethernet cables or poor connections. Is it connected via wire or do you have a WiFi adapter in between?
That sounds exactly like what has been happening for the last week or so.... Yes I have relay protection for when I transmit. All are wired direct with ethernet. All 4 kiwi's behave the same, yet the internet for everything else is just fine. Thanks for the comments.
what happens with local connections (on the LAN)
Hi Don,
It looks like some sort of networking issue.
In situations like this I usually ask myself "what has changed ?"
Some initial thoughts.
Have you moved any cables or wiring ?
Have you altered your network configuration or added any new devices that could for example cause a clash of IP addresses ?
Does this occur on your local network without the internet connection working ?
Has your IP service provider changed anything ?
Are you using IPV4 or IPV6 mapped to IPV4 ?
Have you tried any of the network route tracing tools to see where packet loss or corruption is occurring ?
Regards,
Martin
I tried it too, and after a few seconds audio and WF stopped. Often it continued after a break, but not always. Seems very similar to nitroengine's experience. So I also think it's some kind of network problem.
Yes, the first thing to check is the last thing one touched.... but in this case I can't think of anything I did. I'm leaning to suspect the ISP did something but I'd like to be able to give them an area to look into first. I was planning to give them the link to one of the KIWI's and with their network access, perhaps they can see what is going on. Network tools (other than ping and tracert) are above what little I know so that's why I was hoping the ISP could chase that idea.
To answer other questions - they work fine on LAN. perfect. No IPv6 here. Nothing new on the network.
Thanks, Don
I never had a connection dropped to the Kiwi URL listed in the first post. Only these long pauses which always eventually recovered.
Interestingly, if you look closely at a completely zoomed out waterfall you'll note the WF always "catches up" when the pause ends. So the Kiwi is buffering all the WF data. It's just the network connection that has temporarily stopped.
Are the Kiwis are the only devices that use wired Ethernet connections to your router? You other connections, which do work, use wifi from the router? Then I would suspect a failing Ethernet hub inside your router. I would have said a failing Ethernet cable but you have multiple Kiwis exhibiting the same behavior.
I have other computers that serve multiple SDRAnywhere, Perseus and various video camera servers, all wired to a 24 port Cisco dumb switch, which go to the Asus RT-AC68u router. I have changed ports on that, in trying to eliminate commonality somewhere..
what model is your Cisco switch?
SRW2024 v1.3
that's not so dumb... I see it can do VLAN so it can also do some interport routing...
You have a lot of kit running.
Is it possible that something is hogging a lot of bandwidth ?
If it works on the local network, I suspect your ISP is doing something with your upstream traffic, maybe throttling it back for some reason. Usually because they don't have enough capacity for all of the users that are trying to connect, (contention ratio) and they don't want to spend money upgrading their plant until too many people complain and they have to do something about it.
Can you access the switches stats, to see if one port is particularly busy ?
Alternatively unplug everything else except one KiWi and see if that behaves itself. The gradually introduce more items until it falls over again.
Regards,
Martin
I got the switch from a friend who set it up (at my request) as a dumb switch as I would be too dumb to mess with it.
Yes there is quite a bit running, but it has been that way for a long time and the internet is very reliable so it does very well. These guys are proactive and in the past have expanded their backbone at the first sign of congestion. They are not your average greedy over subscribed ISP. They do have a 150 tower on my property and I have been hosting them for many years.
I keep a close watch on traffic and there was nothing going on to cause these issues.. and speed tests showed plenty of bandwidth in both directions.
A friend just emailed me and said that I fixed it! He has been on for 25 minutes. I see other users on too. I don't want to jinx it - I did nothing other than some resets on the router and isp radio, which had been done many times before. And on a weekend, doubt the isp was tinkering too much either.
Fingers crossed....
several days later and all is behaving fine....
I'm very sorry, but I can't confirm that - a quick test at :8173 showed the same symptoms for me as I described earlier. Of course this does not necessarily mean that your problem is back - maybe it's another problem or something on my side which nobody else has. Just my feedback.
I just noticed the URL in the first post has
/?abuf=0.25,0.25That is an invalid audio buffer setup. The two numbers are min and max buffer times. If you set the max equal to the min I don't know what effect that will have. Probably nothing good. Using/?abuf=0.25and the default max will still give you what you want (a minimal audio buffer delay).So try that, or remove the option entirely.
Well I spoke too soon.... jinxed it! Issue returned this afternoon - likely right around the time DL7AWL checked! nothing I did (that I know of).
I've been using that URL for over a year or so, but I will correct it. Altho it has worked fine with it so maybe the effect isn't serious. Most users don't add that so the problem is elsewhere. Internet speed was as before - 32 down 11 up, 26ms ping and everything else seems to behave properly.
And the users emails complaining have started....
If it works OK on your local LAN, and other kit can be remotely accessed without any issues, then that rules out your network, switch and router.
So it has to be something that your ISP has done recently, despite their denials, such as plant, hardware or software upgrades. Quite often changes can be made at the local level that the folks at the main centre don't know about (and vice versa), even though they all should be kept informed.
Network Jitter and Latency, were issues we faced when I was working and we streamed video, and this could well be a buffering issue, but without access to the diagnostic tools the ISP is likely to have, it is difficult to pinpoint the exact problem.
Maybe a remote user could do a trace to see if there are any obvious packet losses or latency issues that could indicate any problematic nodes in the path ?
Regards,
Martin
Are there other users when it drops?
If you lock it down to just one user and test that externally does is still choke?
Can you enable ICMP ping from outside you network and from a remote location see what sort of packet size survives the trip? (ping -f -l 1430 72.172.110.98) incrementing the 1430 up and down to see what it will take without fragmenting the packet?
Are there any geographical commonalities to the users that see issues?
Ideas to consider:
The packets are being truncated and some are not making the trip (being black-holed on the internet).
Some local (or ISP) firewall is chopping the packets or rate limiting (do you have an ISP firewall enabled?) do you have logs of traffic or interface loading on your router, to be able to see if it is being DOS'ed or heavily loaded? Some routers if you turn on all the features can struggle.
I'd probably look to sniff the traffic around your router, replace network cables, check for re transmits or errors on ports, even swap ports on the switch, power cycle it.
Thanks for the comments. It happens when only 1 user is on... from anywhere, even if that user is me (on the external IP). Can barely get thru 2 minutes of audio/waterfall, sometimes less. Just now, got 26 seconds and then freeze on http://72.172.110.98:8173/ Fine on LAN. Other internet at the time is fine and the speed is normal. Any firewall is in my router, not the ISP. Other remote radio servers are fine at the same time, somehow it seems to be unique to the KIWI. They worked great again from Fri to yesterday and then just quit behaving again. I will be at another location tonite and will try the ping commend mentioned, thanks....
better late than never, I guess
Don, I think it was friday afternoon that I was playing with your misbehaving kiwis (about 4ish local) and your internet presence, at least to the kiwis, disappeared entirely. Knowing you had thought about a different switch/router, I waited for them to reappear which they did a few minutes later and all seemed fixed. They seemed fine all weekend but did fail eventually.
Now knowing that you had nothing to do with it, I will note also being in a teamviewer session with ve6egn who lost his internet at exactly the same time and duration.
His ISP is different so I cant expain the connection between the simultaneous outages. What happens around 4PM on a friday?
Ross
Latest update... the provider ran a ping test all weekend and things were fine (as expected), but the regular internet was always ok, just not the KIWI's. So I changed routers to an IQRouter. No difference. I went back to the Asus. Mor troubleshooting by process of elimination - Putting one KIWI into the router directly and going in from an outside IP showed the KIWI was behaving fine. I then started connecting all the other devices back in and it still was working fine. It and the other KIWI's have run fine for the last day, but it has done that once before so we'll keep our fingers crossed.
One bit of collateral damage - one KIWI lost its marbles and IP during all the rebooting etc... so I reflashed it to the original card with v1.2. It's working fine and I can connect locally. restarted it as per instructions but the auto update didn't happen - left it on all night. Is there something else I need to do?