Troubleshooting after lightning strike
Hi,
A lightning strike just hit the power line which connects to my house. The RCD and fuses tripped.
I had disconnected the antennas before the storm but i left the kiwis powered on. Now there seem to be some damage on all 3 of them.
Two kiwis don't show any signs of life when i connect to power. On the 3rd kiwi, a LED on the BBG lights up.
On all 3 kiwis, the common mode choke gets very hot. The voltages after the common mode choke are 4.4V on the one where the LED still lights up. On the other two kiwis, the voltages are 3.5V and 4.4V.
I separated the BBG boards from the kiwi board. I tried to power the BBG via the micro usb connector. But nothing seems to happen, there is no current draw at all. If I understand the manual correctly, I'd have to hold down the user button while applying power.
Any tips for further diagnosis are welcome.
Edit: If I understand correctly, the BBG should be powered through the 5VE pins? They have 2-3 Ohm on the BBG input (VDD_5V) which doesn't seem right.
Comments
The CMC on the Kiwi is a bit undersized and can act as a poor fuse in some cases. "Poor" because it doesn't always cleanly blow open, but rather develops high resistance and the heating as you've observed. 4.4V is much too low. The Beagle won't power up with this voltage (4.8V minimum).
But the CMC is not strictly necessary for operation. You can remove it and jumper over the pads with a bit of wire. A painful modification for sure. But it can be done.
Alternative: If you decide the Beagle needs to be replaced buy a BB-Black instead of a BB-Green and use the DC barrel jack of the BBB, completely bypassing the Kiwi DC jack. As you know, the BBG doesn't have a DC barrel jack for 5V input, only micro-USB. But this can't handle the current of the BBG and Kiwi both. Hence why the Kiwi DC barrel exists in the first place. Note that using the BBB DC barrel may be effected by your enclosure situation. The BBB and BBG are about the same price these days (wasn't true back in 2016!)
We have some evidence over the years that the Beagle is the thing that gets blown up and often not also the Kiwi board. This is because the Kiwi DC input connector goes across the Kiwi board, into the P9 connector, and down to the Beagle and its power supply. Then power after the Beagle voltage regulators comes back up to the Kiwi board. So the Beagle is first in the path and takes the damage. Beagles are relatively inexpensive to replace.
You did the correct thing by separating the boards and trying to run the Beagle alone. If the software on the filesystem (eMMC) is okay then it should power up and the 4 LEDs show a pattern without needing to hold any of the buttons down. If this doesn't happen you could try re-flashing from a Kiwi sd card image.
The single blue LED on the Beagle is from the Beagle power supply. It can light up even though the rest of the Beagle is dead and will never boot.
Thanks for the explanation @jks
I think the BBGs are dead. They won't power up through the USB port. If I apply 5V on the VDD_5V, it draws 0.7-1 A, the TPS65217C gets hot and nothing else happens.
I'll replace them and hope for the best.
Can the BB-Black be cooled passively if the PCBs aren't in an enclosure?
Can the BB-Black be cooled passively if the PCBs aren't in an enclosure?
Yes. I run BBB or BBG plus Kiwi board sitting horizontally on a shelf all the time without any issues. In an enclosed box it's a different story of course since there's roughly 5 watts that has to go somewhere.
It would make sense that the PMIC (TPS65217C) got fried. I would replace with BBB (not BBG) and only have to remove the CMC and not have to bridge the pads. That's much easier to do. Presumably the heating comes from the windings (5V, gnd) having a low resistance between them.
Maybe buy just one BBB and see if any of the Kiwi boards survived rather than invest in all three Beagles at once.
Remember that Mouser and DigiKey (USA) have free international shipping for orders over USD $50. It looks like okdo.com in the UK has BBB also. Probably lots of other places too.
It's also possible to pop the plastic top of the CMC with a thin flat blade screwdriver, cut the winding connections two at each end and jumper lengthwise in place of the disconnected windings.
This avoids having to heat up the pads to remove the entire CMC - a rather difficult process without the right desoldering equipment.
Thanks Glenn. I did not realize that.
Thanks for the tip. My guess is that the CMC could be ok.
The BBG have very low resistance on the 5V input so there would have been a bit of power dissipitation on the CMC. I measure 0.1 to 0.2 Ohm across the CMC but the resolution of my multimeter isn't good at that low range.
That's a good point. If the PMIC on the Beagle was effectively shorted then the CMC could certainly get hot. It's spec is DC resistance is 0.15R (0R15).
A bit off-topic, but my signalink USB radio interface has also seen better days.
It was connected via a 10m long active usb cable (which also doesn't work anymore) from my PC to the IC-706. It was turned off and no antenna was connected. But there was possibly a ground loop from the IC-706 power supply via USB cable to the PC.
So, two of the kiwis appear to be running fine.
On the third, the waterfall is blank, the S-Meter shows 0 dBm and it is not possible to select a frequency or demodulator.
GPS appears to be working.
I'm afraid it might have affected the LT2248?
I think if you can't select a frequency then I suspect the firmware may be corrupted.
Have you tried restoring it from an SD card backup ?
I had a problem with one of my KWi's that affected the ADC, but I could still tune to different frequencies, even though I couldn't receive anything.
Have you tried using the signal generator extension ? This injects a signal partway along the receive chain and will help determine if the problem is before or after the ADC.
Regards,
Martin
Regarding the Firmware: It runs fine with a different kiwi PCB. So I think the firmware should be good.
I tried now the sig gen extension but it shows nothing:
And the status from the admin page:
Datapump: 0, 0, 0, 0, 0, 0, 0, 0
SoundInQ: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
GPS doesn't show any satellites anymore, but I'm pretty sure that it worked previously.
Ah OK, that doesn't look good.
Assuming all the supply rails are good, and clock signals are present, I can't think of what else to try other than piping clock & ADC output signals into the dud board from a good working one to see where the signal path ends.
Maybe John can suggest some test points or or other diagnostics to try.
Good luck,
Martin
It's extremely strange that the Kiwi server is running (the web interface shows) and yet there is no waterfall. The server won't even start unless it successfully programs the FPGA and downloads the code into the eCPU on the FPGA and gets a response that it's running. So it's difficult to imagine why the sig gen wouldn't show something. That all happens internal to the FPGA.
Is it possible to put this Kiwi on the web so I could connect and have a look? If so, please email public ip address (or domain) and temporary admin password to support@kiwisdr.com
Thanks
Oh wait, I can think of something. The eCPU and SPI logic for communicating to the Beagle run off the 16 MHz GPS clock. But all the RF stuff (waterfall and audio DDCs) run off the 66 MHz ADC clock.
So check the ADC clock on the board with a scope. See the "65M" test point mentioned here: http://www.kiwisdr.com/ks/troubleshooting.pdf
You're right, @jks
I have only a 60 MHz scope, so take the measurements with a grain of salt.
On a good kiwi I measure 930mV RMS on the 65M test point, on the bad kiwi 3 mV.
I don't have a signal generator, so I used the tracking generator of my spectrum analyzer to feed a 0 dBm 66.666 MHz signal to J5. On the 65M test point I measure 650 mV not-very-sinusoidal.
When I enable the external ADC clock setting, I have a waterfall with a -120 dBm noise floor and the OV warning is occasionally flashing.
But unfortunately no signal from the antenna input. The sig gen extension does now produce a signal though.
The voltages are all within spec.
Okay. So it seems the 66 MHz XO (U402) is not running. Very strange that would fail.
But it also sounds like the FPGA is okay given that the sig gen works. But the ADC is probably dead. It would be interesting to know if any of its 14 outputs are switching. If you're getting a -120 dBm waterfall only the bottom bits would be non-zero. But it's also troubling that the S-meter OV is flashing. That shouldn't be happening.
The oscillator appears to be running, on the output pin I measured 670 mV (1.39 V on a healthy kiwi). But it doesn't make it to the test point.
I'll look into it again tomorrow, Anything particular that i could test on the ADC?
Well that doesn't quite make sense. Look at the schematic (http://kiwisdr.com/docs/KiwiSDR/kiwi.schematic.pdf)
The oscillator and external clock both go through U406 on the way to the FPGA. If the FPGA sig gen worked when you applied an external clock, and you're measuring output from the oscillator, then the oscillator should also make the FPGA work. Make sure the admin setting for external clock on the config tab is set correctly.
As I mentioned before, look at the 14 output data lines of the ADC. I'd like to know if the low bits (D0, D1, etc) are switching. Probe on the series resistors R404,405. The schematic has the same orientation as the board for the ADC and R404,405. So D0 is at the "bottom/right" of the lower resistor pack R405. It would also be good to know if OVFL is switching.
Probe the right side (e.g. D0 not AD0) of the R pack so in case you short adjacent pins together you're not shorting the ADC output drivers. It's difficult because these parts are extremely tiny.
When the kiwi is powered off, i measure 30 Ohm from the XO output to ground, which seems to be low. On the good kiwi it is several 100 kOhm.
With "external ADC clock off", I measure 670 mV on the input of U406 and 3 mV on it's output. U406 has good VCC.
But today I can't get it running with the external ADC clock (and proper software setting). The external clock doesn't seem to make it across U403 (OE(inverted) low, VCC good).
I don't understand why it was working yesterday. The only thing i changed since then was resoldering the coax to J5 because it was a bit flimsy yestarday.
Suggestions? I can try to replace U403/U406, but is there anything else I could do before that? It seems a bit odd that they would fail.
Edit: The low resistance on the XO output appears to be coming from the ADC input, not U403/U406.
Do you actually use the external clock input ?
If not then you could just remove U403/U406 and replace the switched RF paths with wire links.
But either way it would be a quick method of testing for the fault.
Regards,
Martin
No, i've only used the external clock for diagnostics.
I just wanted to ask for the options, before i put my hands on the U403/U406 because I'm not very experienced with SMD soldering and don't have the right tools currently here.
Unsoldering: yes, resoldering: perhaps :)
I usually find re-soldering the easy part.
Removing multi pin parts without damaging the PCB tracks can be challenging.
Sometimes you have to destroy the part in order to make it easer to remove the individual pins.
Once you have cleared the board. Just tack one of the new device pins to the correct pad, check it is aligned correctly and then work your way around the rest.
Regards,
Martin
My only other suggestion is to check all the other voltage test points (see the troubleshooting guide). "1V" and "1.8V" for the FPGA. "3.3A" for the RF. "3.3G" for the GPS.
Also, check that you have a GPS clock on the "GCLK" test point (about 0.9V).
The DC voltages and GPS clock are good.
The only thing which is definitely not good is the input resistance of the ADC clock (30 Ohm).
I think it is pulls the clock voltage to a level which is too low. Or the input is completely broken. Bypassing U403/U406 didn't help.
The ADC is definitely broken, but out of curiosity I set it up again with a proper signal generator as external clock source.
Sometimes I get a waterfall with -120 dBm, sometimes it's black, but tunable and it shows a -127 dBm level with overflow flashing.
I probed the ADC output and the bits 0-12 were always high. bit 13 and OVFL were always.. semi-high .. 1.68V
Worth a try to just replace the ADC?
Check the DIL resistor packs between the ADC output and FPGA before you do that.
A "semi-high" reading with a DVM can be an indication that the pin is switching (the DVM gives a somewhat averaged DC reading). Your scope, even though limited in bandwidth, should show the switching. Bit 13 and OVFL always being high seems very bad though. And if the ADC is bad the ADC preamp is likely as well since that's the probable path the surge followed.
Replacing that part is going to be tough unless you've got experience doing it. Hot air gun mandatory of course. But that's the least of your problems. Prepping the pads for new solder paste and not blowing off the surrounding components are the biggest issues.
But it can be done. I know of one Kiwi owner who replaced the GPS front end chip themselves (incredible). And another who took a dead Kiwi to a professional company (run by hams) who were able to do the SMD rework to replace the ADC preamp chip (this was recently, so no possibility of simply purchasing a replacement Kiwi).
Yes, I would give the kiwi to a company which does repairs. But I think having a full diagnostic done by them would be too expensive.
I measured the semi-high with the oscilloscope, there is no switching on any output. Resistors are good.
So I will have to test the function of the preamp first.
The ADC has been replaced and everything works again :)
congrats and <applause>