Broken Beagles
Hi,
This morning I noticed that my main KiWi had stopped working with no network connectivity.
The blue LEDS on the board were flashing as normal but both the yellow and green LED's on the ethernet port were on, even immediately at power up and with no network cable plugged in. The yellow and green LED's do not flash or go out (unless the power is removed) they just remain on all the time
I tried reflashing the KiWi from a previous good backup, but it remains the same. I also tried connecting via the USB port, but I get no reply to a ping and can't establish a connection with Putty.
I then tried swapping the power supplies over and swapping network connections around. During this process my second KiWi failed in the same way (They are not from the same batch).
I've checked and swapped the PSU's and network hub, so I'm fairly sure that these are OK. I've also tried reflashing the second Kiwi, which like the first one seems to fire up and work with the correct blue LED sequences. But despite trying all the obvious tests and fixes I can't get either of them to respond to network or usb port communications.
I have a third KiWi which is working, but I'm somewhat reluctant to mess with this one until I have a better idea of what's happened to the other two.
The main symptom seems to be the permanently on yellow and green network port LED's, which I don't think is a good sign.
Any ideas ?
Regards,
Martin - G8JNJ
This morning I noticed that my main KiWi had stopped working with no network connectivity.
The blue LEDS on the board were flashing as normal but both the yellow and green LED's on the ethernet port were on, even immediately at power up and with no network cable plugged in. The yellow and green LED's do not flash or go out (unless the power is removed) they just remain on all the time
I tried reflashing the KiWi from a previous good backup, but it remains the same. I also tried connecting via the USB port, but I get no reply to a ping and can't establish a connection with Putty.
I then tried swapping the power supplies over and swapping network connections around. During this process my second KiWi failed in the same way (They are not from the same batch).
I've checked and swapped the PSU's and network hub, so I'm fairly sure that these are OK. I've also tried reflashing the second Kiwi, which like the first one seems to fire up and work with the correct blue LED sequences. But despite trying all the obvious tests and fixes I can't get either of them to respond to network or usb port communications.
I have a third KiWi which is working, but I'm somewhat reluctant to mess with this one until I have a better idea of what's happened to the other two.
The main symptom seems to be the permanently on yellow and green network port LED's, which I don't think is a good sign.
Any ideas ?
Regards,
Martin - G8JNJ
Comments
I have been able to connect to one of the KiWi's using a USB cable and kiwisdr.local:8073/ as the URL.
It all seems to be running as usual apart from the ethernet port.
Any tests I can try ?
Regards,
Martin - G8JNJ
After you connect to the KiwiSDR with the USB cable, enter the command ifconfig . That will list your network connections and assigned IP addresses for each one. You are looking for eth0. You can also reboot the KiwiSDR and then enter this command:
root@kiwisdr:/var/log# cat /var/log/kern.log |grep -i eth0
Nov 3 17:41:09 kiwisdr kernel: [ 7.308684] net eth0: initializing cpsw version 1.12 (0)
Nov 3 17:41:09 kiwisdr kernel: [ 7.312452] net eth0: phy found : id is : 0x7c0f1
Nov 3 17:41:09 kiwisdr kernel: [ 7.322075] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Nov 3 17:41:09 kiwisdr kernel: [ 11.341755] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Nov 3 17:41:09 kiwisdr kernel: [ 11.341822] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
root@kiwisdr:/var/log#
If it looks right, it probably is, if not, then the network port has likely failed.
Ron
Jul 23 14:26:31 kiwisdr kernel: [ 149.672992] net eth0: initializing cpsw versi on 1.12 (0)
Jul 23 14:26:31 kiwisdr kernel: [ 149.680956] net eth0: phy found : id is : 0x7 c0f1
Jul 23 14:26:31 kiwisdr kernel: [ 149.696344] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jul 23 14:26:31 kiwisdr kernel: [ 152.696871] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Jul 23 14:26:31 kiwisdr kernel: [ 152.715456] IPv6: ADDRCONF(NETDEV_CHANGE): et h0: link becomes ready
Jul 23 14:26:31 kiwisdr kernel: [ 149.437243] net eth0: initializing cpsw versi on 1.12 (0)
Jul 23 14:26:31 kiwisdr kernel: [ 149.445037] net eth0: phy found : id is : 0x7 c0f1
Jul 23 14:26:31 kiwisdr kernel: [ 149.460355] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jul 23 14:26:31 kiwisdr kernel: [ 153.477669] cpsw 4a100000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Jul 23 14:26:31 kiwisdr kernel: [ 153.485827] IPv6: ADDRCONF(NETDEV_CHANGE): et h0: link becomes ready
Jul 23 14:26:36 kiwisdr kernel: [ 169.431621] net eth0: initializing cpsw versi on 1.12 (0)
Jul 23 14:26:36 kiwisdr kernel: [ 169.443459] net eth0: phy 4a101000.mdio:00 no t found on slave 0
Jul 23 14:26:36 kiwisdr kernel: [ 169.460322] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jul 23 14:26:36 kiwisdr kernel: [ 7.290040] net eth0: initializing cpsw versi on 1.12 (0)
Jul 23 14:26:36 kiwisdr kernel: [ 7.296521] net eth0: phy 4a101000.mdio:00 no t found on slave 0
Jul 23 14:26:36 kiwisdr kernel: [ 7.313306] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Jul 23 16:29:04 kiwisdr kernel: [ 7.178262] net eth0: initializing cpsw versi on 1.12 (0)
Jul 23 16:29:04 kiwisdr kernel: [ 7.184737] net eth0: phy 4a101000.mdio:00 no t found on slave 0
Jul 23 16:29:04 kiwisdr kernel: [ 7.201451] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
I guess the problem may have been caused by some external issue, as it's odd they have both failed within a day of each other, even though they were from different batches. However I don't know what it could be as I've got extensive protection on the DC and RF feeds to the boards and the network switch is local and all the ports are functioning OK.
The first Beagle was an original kickstarter supplied KiWi kit, the second was purchased via Massdrop about a year later.
I'll probably buy some boards from Farnell in the UK as that's the cheapest option and I may pursue Seeed if that fixes the faults.
Is the Farnell supplied BBG now OK - I seem to vaguely remember there was an issue with some boards ?
Regards,
Martin - G8JNJ
A bit of internet surfing suggests that the BBB is even worse with regard to the ethernet port not initialising.
There seem to be various hardware mods suggested to strap a pin to a GPIO port so that it can be kicked into life with a watchdog if required.
It doesn't sound too good to me.
Maybe the BBG wireless or a USB wireless dongle to resurrect the current boards ? or a USB to ethernet dongle ?
Regards,
Martin - G8JNJ
Is it physically compatible to replace the BBG Will it fit the metal case and are the GPIO pins the same etc so that the antenna switch would still work ?
I think the metal case fan plugs into the BBG grove port. Could I just wire it to the switched 5v rail instead?
Regards,
Martin - G8JNJ
If you haven't already, try a new network cable. I've had good network cables quit or become intermittent. Sometimes an unplug/replug gets it going, or changing the cable to another known good one. The pins on the plugs seem to corrode a bit overtime. The port on a network hub or switch is also a suspect point of connection failure. A good test is to use the network connections with another known good device and see if it connects.
Ron
KA7U
Swapping cables and such like was the first thing I tried.
Unfortunately it looks like the LAN8710A chips on both Beagle boards are fried.
I'm not sure what has happened. The first board was definitely not my doing, but the second could have been whilst I was swapping cables and boards around. Either way, I think it would be hard to pin the failure on Seeed after they have been running OK for this length of time.
At least I've been able to confirm that the KiWi boards still work, which was my main concern, so that's good news.
I've ordered up another couple of BBG boards to quickly get me up and running again (I had just ordered some metal cases prior to the failure, so I've held off from buying BBB's in case they are not compatible) and also a few LAN8710A chips to see if I can repair the existing BBG boards and keep them as backups, as I've nothing to loose if I mess them up during the rework process.
I was thinking that if I get the old BBG's working again I could just buy some KiWi boards to use with them. But actually it seems that it's probably cheaper to wait and get a whole kit via Massdrop than buying just the KiWi boards from Seeed.
Fingers crossed that I can get it all up and running again without too much hassle :-)
Regards,
Martin - G8JNJ
When I looked at the 0-30 MHz noise floor, since that is one of my current interests. I found that the BBB had a 'hump" out in the middle of HF that the BBG doesn't. Perhaps this is the HDMI circuits, I don't know and didn't pursue it. Instead I ordered a BBG and am now waiting for it.
I would say use the BBB at your own risk...
Glenn n6gn
I ask as my first Kiwi is still going but does not fire up the network in some circumstances, may take a few goes.
All I have inboard of the Kiwi now is about a 1m CAT7 cable and a copper to SFP adapter to a router with a separate switching PSU (plugged into the same mains socket as the Kiwi PSU).
Before at various times I did fire it up with all copper and my small antennas connected so may have stressed something.
I did try feeding the fibre router with the same PSU as the active antenna but when plugging it in there was a visible spark as the negative went in (can't see the positive pin) and the glitch would reboot the rooter.
I suspect voltages between the external feed and local ground (whopping 9m of semi-buried cable) are to blame, not sure what sort of current could be induced if the Kiwi had a lot more copper hanging off it inside and out. There are lots of mobile, pager and tetra signals here and that is only in the frequency range I can detect.
I think it is safe to say you have more induced voltages on your antennas than mine (those pesky "signals" which I don't suffer), maybe there should be some powering sequence that involves grounding out the antenna via a low value resistor, powering the BBG then connecting the antenna (in some surge arresting way).
May be interesting to know the situation in which the items fail.
Stu
I agree you have to be careful with ESD paths, especially if you have big external antennas and live on top of a hill :-)
The first KiWi was obviously connected to the antenna etc. when it failed. However the second one only had the ethernet and DC connectors plugged in when it broke.
I have taken great care to ensure that the RF ground on the antenna and mains / DC supply ground are all at the same potential, including adding supplementary ground rods and suitable bonding cables.
My KiWi's have a spark gap (GSD), ESD discharge, 30MHz LPF filter, BC band notches, RF limiter and transformer isolation. All contained in a custom built module that I use to provide additional protection on the RF input.
You also have to be careful with DC supplies (especially the double insulated Switched Mode types - the ones with only a two core mains cable). These often contain an RF filter network consisting of a Y capacitor network across the incoming AC supply and to the supply ground, which is also often the DC ground. The net result is that the DC ground 'sits' at 1/2 AC supply potential (but at very low current).
However when you first plug in the DC connector to any kit that already has a ground return path, an appreciable current can flow from the Y capacitors to the ground of the kit the supply is being connected to for a fraction of a second. This can be enough to pop any sensitive devices.
I always make sure any DC supplies have a proper connection between the AC ground and DC ground to try and prevent this problem. However this can result in additional RF interference unless the RF, AC and DC grounds are all at the same RF potential (not the same as supply potential) or are isolated by means of RF choke baluns (or a combination of both).
It takes a lot of effort to try and maintain safety and ESD protection whilst at the same time trying to minimise RF interference paths. If you also have UPS's in circuit it becomes even harder :-(
I think my second KiWi popped when I was repeatedly plugging and unplugging the network cable whilst trying out different tests. I can only think that I must somehow have cross connected some of the pins or got the screen of the RJ45 shorted against something it shouldn't have.
Regards,
Martin - G8JNJ
Glenn n6gn
Would be nice to see a full spectrum (with actual spectrum not just waterfall) and a close in look at the signal. Some network kit here causes a rise around 18-21MHz but that is visible zoomed as signals at equal intervals.
The first one fired up perfectly after I had loaded one of my backup images of a recent version from the SD card.
The second one loaded up with a backup image from my second KiWi. It booted OK and the LED's indicated that the correct static IP address that I had previously allocated to the KiWi had been loaded. However although the green and yellow network LED's flashed initially, as soon as the image had loaded and the IP address had been allocated, they both went out and didn't light again.
I spent most of the afternoon updating SD backups on two working KiWi's, verifying that they worked OK and then tried loading them onto the new Beagle that wouldn't connect to the network.
Finally, I swapped boards around again to verify SD backup images and then reloaded a third KiWi's backup image onto the new Beagle that wouldn't connect. This one has DCHP set in the SD image, but the router had been setup to allocate to a specific IP address to a specified MAC address, so I knew that the router would try to allocate a new DCHP address when it first saw the new KiWi. Sure enough the router showed a new DHCP address and listed the MAC address, so I was able to copy the new MAC address into the router in place of the old one and get the router to allocate a designated IP address. Phew.....
The green and yellow Ethernet port LED's flashed, and I could now connect via the browser and Putty, so all was well.
I just don't understand why the second Beagle was so difficult to bring up. I'd tried Bonjour, Putty, Pings and all the rest but it just wouldn't talk. I couldn't get anything to connect to the USB port either. I don't know if the old routing tables had been stored somewhere and it took a while to flush them out, but if that was the case I'd have expected to have had similar problems with the first KiWi too.
Very odd....
I'm now wondering if it is safer to set DHCP on all the KiWi's rather than having static addresses, but define specific IP's against MAC addresses in the router so that they always come up on the same IP's. I'm not a network Guru so I don't know what best practice is. Maybe others could advise ?
My next challenge is to solder the new surface mount LAN8710A chips back onto the original broken Beagles, to see if I can resurrect them to keep as backups.
Regards,
Martin - G8JNJ
I'd be tempted to fire it up without the Ethernet connected or connected to a spare DHCP server/router.
I'm used to seeing network checks on Linux as it comes up, I wonder if, in this case, it asks the network "if an IP is in use" and gets a "yes" and that stops it completing some part of boot.
Depends what happens in the error condition interface connected but won't come up what is the fall back situation.
I fixed the address on my first Kiwi (can't remember exactly why) but the DHCP option (with IP mapping at the router) does seem a better bet now.
I wonder also if the hardware MAC is used for identifying the right interface and that is no longer is correct when loading from a backup?
I will be interested to hear how you get on with the LAN chips as I fear mine may be marginal.
Cheers
Stu
Edited later in the day simply as I should never be let near a keyboard in the morning (minimum).
For the attached screen shot, the KiwiSDR was freshly calibrated from a reliable signal generator. In the 2400 Hz USB bandwidth, far away from the peak of the bump, the S meter reads about -127 dBm. -127 - 10log(2400) = -161 dBm(1 Hz) for a ~13 dB noise figure. At the bump it reads about -122 dBm. This is actually a somewhat lower noise figure than is measured on other Kiwis so it is possible that the lower noise has revealed a 'bump' that exists in other receivers. I haven't had time to investigate this possibility.
Glenn n6gn
Attachments:
https://forum.kiwisdr.com/uploads/Uploader/54/f7010a44ef77a185bf228a53ece6c3.png
I had a bit of a problem with the second board as the LAN8710A is a 32-pin QFN/SQFN surface mount package with a fairly large heatsink pad on the underside. I thought I'd got it hot enough with the hot air gun on the rework station, but unfortunately three of the pads lifted off the PCB from one corner when I removed the chip :-(
Fortunately I managed to repair the missing tracks and the chip works OK :-)
However the second new Beagle board I got from Farnell still refuses connect. I've tried just about everything with it, but it just won't cooperate. I've checked the same image I've loaded onto it on two other Beagles, which have both started up and connected to the router, so I don't think it's a MAC address conflict as the same problem would have occurred. So I'm 95% certain there is a problem with the new Beagle board.
I've still got enough spare chips to try swapping the LAN8710A, but I think it's probably better just to send it back to Farnell and get a replacement board.
So it looks like the Ethernet port is the Beagles weak spot.
Regards,
Martin - G8JNJ
I wonder if the heatsink pad is marginal or is that too simple a failure mode? I always think of NVidia BGA re-ball kits on ebay and printers that will come back to life if you bake the control board, I had a couple of HP1320's that went that way no point fixing them as they just do it again in six months.
Not sure if I dislike SM packages because they can fail from hidden design/build faults or just because it's so tricky to replace the components without fancy kit.
Might have to borow the company thermal camera and look at the chip when working.
Still very useful to know they can be raised from the WEEE pit.
The fact you posted the LAN8710A details with package allowed me to quickly search Mouser while ordering other items, just had to double check it was 3V3 from the Beaglebone schematic.
Not sure if I'll give it a go myself or request the guy at work who is hot on SMD (excuse the pun) to save me lifting pads.
I did order another BBG anyway.
If you are repairing the Beagle board, be sure to also check R136 on the PCB underside of the Ethernet socket.
The whole sorry saga is told in the thread that this post form part of.
http://forum.kiwisdr.com/discussion/comment/5531#Comment_5531
BTW The faulty board I got From Farnell was exchanged and the new one worked perfectly.
Regards,
Martin - G8JNJ
Now I'm more confused as this DDG/Kiwi was powering up but the LAN connection, link light on the switch, were going out after a few minutes and not recovering.
I didn't have a current backup so I decided to split it down and initialise backup, of course it ran while apart and is still running.
While it was initially failing I swapped out power supplies, switch ports and cat5 cables, all with no affect. It was in a metal case with my other Kiwi on separate supplies so maybe there was some interaction between linear to DC-DC modules, switch PSU ground etc., not sure but I needed to split it down again so may never know.
I'll leave it running today and get on with my chores. Mouser want an export form filled before they'll send the BBG (as mentioned in other threads) so I'll complete that and have a spare. This Kiwi/BBG has had more bad use by me than any kit deserves so a safe retirement is only fitting.
Cheers
Stu
Don't have time to look (at work) but needed to bookmark for further consideration...
Later that day-- 33pf on the RX clock marked on the schematic as "C162 33pF,4 EMI"