Help Needed: How can I access a remoted Kiwi somewhere in 169.254/16 ? [fixed in v1.254]

n6gnn6gn
edited January 2019 in Problems Now Fixed
I don't know what I'm doing with networking, perhaps I should stick to RF. But I have a problem that may be solvable. Here's the situation:

I have a Kiwi located at a remote site having only solar power, an hour's drive away on a very snowy/icy poor quality mountain road 800m higher and 22 km from my home QTH. It's very quiet there. I access it via a Ubiquiti IP 5 GHz radio bridge and it normally sits on my private home subnet, served by a home gateway/router which supplies DHCP. This configuration has all been running fine until the remote site lost power last night. After several days of clouds and no charging the battery voltage dropped below threshold, the solar controller shed the loads and both the Kiwi and the link went down. When power returned (with the sun) the microwave link came back up but the Kiwi did not appear.

I believe what may have happened is that the Kiwi is actually powered up and running now but in the process of coming up, it came up before the microwave link. Since the Kiwi was set for DHCP ( I think this may have been a BIG mistake on my part! ) it looked for its normal network and DHCP server but found neither. At that point it gave up and issued itself a random address somewhere in 169.254/16.

I've been able to remotely access and modify the routing table in the microwave link host to point 169.254/16 packets to the local eth0 interface/LAN where the Kiwi resides. But that's all I can do, I don't know the Kiwi address since I can't see the LEDs on the BB in order to SSH into the Kiwi and restart the server now that DHCP is accessible. Thus far I'm unable to flood ping or install nmap to snoop the 169.x.x.x network. Even if I did restart it, I'm not sure what would happen with DHCP under that situation (a mac address already associated with a 169.x.x.x IP, perhaps I'd want to restart the Beagle?)

Pinging the broadcast address 169.254.254.254 gives me a reply that appears to be the Ubiquiti itself.
I've run tcpdump looking for any likely 169.254 activity but haven't had any success, likely I don't know how to configure it.

I suppose this problem would not have occurred if I had realized that DHCP might not be available upon restart and used a static address, but I didn't. I'm left with a number of questions:
  1. Would a static address have been the correct fix for the problem?
  2. Once the Kiwi has failed to find a DHCP server does it ever try again or does it just wait for a non-networked host e.g. laptop, to find it?
  3. If the answer to the above is 'no', would there be any positive value in having the Kiwi look periodically for those situations where it boots before its network does, perhaps even some home gateway/routers might not come up very soon after a power fail.
  4. Am I missing an obvious configuration or solution that someone here can point me toward?
  5. Should I just give up, drive to the mountain when the weather permits, restart it all and set the IP to static?
Thanks for any helpful advice.

Glenn n6gn

Comments

  • elitedataelitedata
    edited January 2019
    if the kiwi set its IP address to a private non routable IP different than your home network (eg. 169/192), you will need to connect your PC or another PC directly to the Ubiquiti, let the PC set its own non routable IP (it should be on the same subnet as the Kiwi after your PC auto-sets) and then scan for MAC addresses (https://www.solarwinds.com/topics/mac-address-scanner) - you should then find the IP address the Kiwi is at and administer it from there.
    not knowing the exact private IP subnet the Kiwi will set itself to when it does not receive a DHCP response is something i dont know - if the above suggestion doesnt work, i'll take my Kiwi, connect it to my PC directly (with Kiwi in DHCP mode) and see what private IP address it will range itself to after not receiving a DHCP response, you can then use that information to set your PC to the same subnet in order to find your Kiwi.
    yes, definitely set your Kiwi to a static IP address of the same subnet as your home network.
    the ID MAC for the Kiwi is 40BD32
    edit - you might get results trying the MAC ID tool on your network before connecting the Ubiquiti directly to the PC.
  • n6gnn6gn
    Thanks John, I was afraid of that :'(
    I think a trip to the hilltop is in my future...
    Glenn
  • njcnjc
    @n6gn FIrst off, when you get this working again, definitely configure a static IP.

    If you can restart the remote end of your link, do it. It might take the NIC down, and when it comes back up the kiwi should do a DHCP discover and hopefully get a real IP.

    If that doesn't work:

    If your link system is layer 2 (not-IP aware, only MACs) then assign a computer on your network a 169.254.x.x IP with a MASK of 255.255.0.0 (/16). Then use an IP scanner to look for active hosts on that entire range (169.254.0.0 - 169.254.254.254). You may have to make some changes to your firewall and router(s) in order to pass/direct the 169.254 traffic over your link. You might try MAC scanning as mentioned previously also.

    If your link system is layer 3, then first assign it (or add a virtual) IP in 169.254.0.0/16 and do the same as above.

    Good luck!
    Nick W1NJC
  • KA7UKA7U
    @n6gn
    Glen, you no doubt have this accounted for, but I'll throw it into the mix. When the power comes back up, do you have a switch to prevent a low power start on the KiwiSDR? If the KiwiSDR startup voltage sags, it boots but doesn't complete the load and doesn't initiate the network. It just sits there until the power is recycled to it. So a mountain top needs a power reset switch to the KiwiSDR and probably a charged capacitor to account for initial current draw if the power supply isn't at full power.

    Please disregard if this is accounted for.
    Ron
    KA7U
    njc
  • PowernumptyPowernumpty
    edited January 2019
    Ron's comment is worth looking at as power supplies that are a bit light (surge current) have been an issue here (detailed in other threads).
    The Ubiquiti kit normally has some pretty useful info so I'd look for the "stations" listed at the remote end, also try rebooting the remote end if you can, that should kick the DHCP into gear if it not in a locked condition (I had forgotten about any DHCP bugs - that may not poke DHCP but worth a try).
    Also as stated in other threads having the supply voltage a bit over 5V won't hurt (E.G. 5.4V) as that gives it a more solid 5V at the Beagle when running and less chance of a brown out on boot. The little DC-DC I'm trying now is set to 5.3V but I don't think it will cleanly boot so needs a cap (reluctant to keep rebooting to test).

    So in summary
    Remote power switch won't hurt.
    Slighty higher supply voltage
    Static IP address.
    Get to know the admin interface of the Ubiquiti (if you have access) they have some nice hidden gooodies (like the SDR waterfall site survey).
  • n6gnn6gn
    Thanks to all for ideas and suggestions. I expect to have a little more detail once we actually visit the remote site again (-2F/-19C here this morning!). In the past, the Kiwi has always restarted, so I don't suspect the power supply. On all my Kiwis I am using simple LM2596 buck converters to get from 12VDC to 5V. These have ample peak current capability such that startup has mever been a problem. In addition, once or twice in the past when on-site, the Kiwi was seen to have the "double beat" LED pattern indicating no DHCP after power was interrupted. I should have taken the "one error is a pattern" adage more seriously.

    We don't know for certain but it looks from this end like the problem is one of a bridge/adaptor not coming up before the Kiwi needed it. This problem could also plague other Kiwi users, particularly anyone who might be using a WiFi adaptor as a means of eliminating all antenna feedline (and associated common mode noise currents!) by placing all the radio hardware right at the antenna. Separately I've created a very light active dipole/Kiwi/Wifi module that is light enough to be lifted by helium balloon. Such a device becomes a sort of portable e-field probe for measuring both signal and noise as a function of height and distance from residential/commercial noise sources. Perhaps more on this in another post.

    I was able to scan the 169.254/16 address space using nmap (it takes a while even with only ICMP/ping probing!) but discovered only the Ubiquiti answering. A visit to the hilltop will answer the question of whether the Kiwi is actually powered and without DHCP-provided address or whether it is a power supply issue after all. This is a first test in very cold weather so anything is possible. It certainly would have been nice to have a remote power/restart control on the Kiwi. We presently have APRS delivering panel/battery/load status, separate from the microwave link, but no way to actually restart the Kiwi remotely.

    Again, thanks all, I'll try to report anything else I discover that might be useful to others.

    Glenn n6gn
    Fort Collins, CO
    KA7U
  • n6gnn6gn
    N6GN/K2 is back on the Internet. It took a trip to the (remote) site to bring it back up. Upon arrival the Kiwi did have power and looked normal but of course was still unreachable. Status of the 4 LEDs didn't get noted before its input power was cycled. Upon reboot, everything worked once more. It found the DHCP just fine. At that point I (remotely) set the IP address to static and restarted the kiwi server, but not the BB. Everything came back up.

    To further test it, a simulated power fail/load shed was performed. The 12V fuse serving both microwave link as well as Kiwi was pulled and replaced after 10 seconds. The microwave link came back up. The kiwi did NOT. It sat with "double heart beat" and not much else. Perhaps we didn't wait long enough for it to grant itself a 169.254/24 address? There appeared to be network activity, judging by the lights at the RJ45, but no response. Maybe had we waited longer a local.network address would have been self-assigned and "nmap -sn" could have been used to discover it.

    This isn't what I expected. I thought that because the static address had been assigned that it would immediately select that upon power cycle/reboot.

    It took another power cycle, like the very first, of just the Kiwi in order to get the expected address to respond. From the admin page, it still claimed to be static and that address so I don't know why it didn't work.

    If we lose power again, I expect that it will fail again and, unless I can remotely find a way to talk to it, require another trip to cycle power.

    Perhaps the addition of a periodic re-probe for DHCP, even after an IP address has been self-assigned would still be the best answer. Barring that, I think we'll need to put an IOT sort of device on the network up there and give it the ability to cycle DC power to just the Kiwi.

    Glenn n6gn
  • njcnjc
    Interesting. If the NIC is set to static it should never get a 169.254 address. That should only happen when DHCP fails to obtain an address. When configured for a static IP the NIC should always have that IP whenever there is a link present. Therefore I think you might have a different problem going on. Perhaps it's that issue where the power supply cannot source the initial start-up current resulting in a failed network? You would need to do more testing like unplugging and replugging the network cable, power cycling just the microwave link equipment, etc to narrow down the issue. I'm sure some of the other guys have ideas too...
  • PowernumptyPowernumpty
    edited January 2019
    I am prepared to bet the Kiwi power goes too low momentarilly during boot, if you have a USB/portable scope watch the supply at power on both devices.
    Another option is delay the boot to say thirty seconds after link power on.
    Many of us have seen this supply rail drop issue and just because on paper the supply should do it doesn't mean there isn't some 10ms 6A load on the supply.
    Also what voltage is the DC-DC set to?

    If you are sure the rail is solid and you still get it - Just had a daft idea, the Ubiquiti's normally have a way to alter the link LED's so that you can fine tune the orientation for link strength, replace the highest value LED with the input of an optocoupler that switches the Kiwi supply.
    Unless they flicker on boot up (I don't remember them doing that) the Kiwi would only get power once the link is established. If you need to power down the Kiwi alter the LED's so that last goes out.
    I know it is impractical as you won't want to go digging into the Ubiquiti but thought I'd throw that out there in case it inspires another solution.
  • njcnjc
    I agree it's likely a power issue, at least until you rule that out. Heavier gauge/shorter wires, use Vsense lines, higher supply voltage (I'd keep it <5.5V), higher amperage (or dedicated) supply, staggered startup sequence. Any or all of those should help if that is indeed the issue. I've found that even if the V at the kiwi drops to around 4.7V or so the BB/kiwi doesn't like it.
  • n6gnn6gn
    I truly don't believe this is a power supply problem. I have seen the power supply problem previously with different PSes (e.g. HP bench supply that didn't have sufficient maximum current). But the part in this buck converter can source 3A @ 5VDC continually. The leads from it to the Kiwi are short. There has been no evidence of shutdown. I use the same exact thing on two other Kiwis with no evidence of a problem What there *has* been on this Kiwi is evidence of lack of appropriate IP address.

    I agree that I wouldn't expect it to ever go to a 169.254/16 address, perhaps it no longer does. The symptom was that there was LAN activity but no connection noted and in the time we waited no advertisement of an IP address. Perhaps I should have monitored traffic down at this end to see if I could see evidence of what it was and where it was going, but a very short look with tcpdump from the microwave host didn't reveal anything. I may have missed it though.

    Glenn n6gn
  • elitedataelitedata
    edited January 2019
    you indicated that upon a simulated power fail and when power was restored, that the Kiwi had network activity indicated on the RJ45 LED's and the blue light was making a "heartbeat".
    did it do this (heartbeat) immediately following the power being applied to the Kiwi ?
    i ask because if that is the case, its normal as the software is being compiled, the Kiwi does this anytime it is power cycled and it takes about 2-3 minutes for the software to finish compiling before access to the web interface is active.
  • PowernumptyPowernumpty
    Wireshark it locally then, I often do that if I don't know the device IP address range, should only take a few seconds.
  • n6gnn6gn
    Fine to do all those things when you're local, not so easy when 20 km distant and 800m up a bad road, covered in ice/snow and at negative degrees C!!

    It came up normally, lights did the normal thing until about the time the server should have arrived. Heartbeat didn't happen until things were up. Then the normal sending of the IP address did not happen - not until the second power cycle.

    Glenn n6gn
  • KA7UKA7U
    @Glenn
    The lazy boy way, change out the KiwiSDR and see what happens with a different unit in the position. Same failure, then issues with power at the site. Works fine, then issue with the old BBG network port or whatever.
    Ron
    KA7U
  • jksjks
    edited January 2019
    You need to login to the Beagle and use the "i" command alias to check that the static address assignment was properly recorded in the Linux networking configuration file (you can also do this using the "console" tab of the admin interface). "i" is an alias for "ifconfig -a; cat /etc/network/interfaces" the later of which should be setting the ip statically rather than using DHCP.

    The behavior you describe of the Kiwi needing two power cycles to be reachable can be explained by it still being on DHCP, and the Kiwi booting up faster than the router / DHCP server.

    Also, is the static address you're trying to set outside the range of DHCP-assigned addresses?

    ----

    Okay, I just looked and it's totally screwed up. Somehow you've managed to set static mode in the Kiwi admin interface (network tab). But /etc/network/interfaces still shows DHCP configured for eth0. Also, the gateway field is blank. It should not be possible to do either of those things. There is a ton of error checking code in that interface that attempts to prevent bad configurations from being made. I guess it's still not good enough.
  • jksjks
    edited January 2019
    So it is possible after all to set an inconsistent configuration (i.e. Kiwi configuration says static but Linux configuration still says DHCP).

    If after selecting static and filling-in the address fields you fail to push the confirmation button that pops-up at the top that says "Are you sure? Click to update interface DHCP/static IP configuration" then the Linux configuration change will not be recorded even though the Kiwi configuration is. If you just refresh the admin page or restart the server the button never reappears and the inconsistency is not undone.

    This only harm this causes is making you think you've got a static assignment set when you actually don't. I fixed it by pushing DHCP, then back to static, then fixing the gateway field and then pushing the confirmation button. But then I did not go ahead and reboot the Beagle as the dialog then requires. I didn't want to risk bricking it again in case there is some other problem. So it is actually currently still in DHCP mode. But it should properly use a static ip at the next reboot / power-fail.
  • n6gnn6gn
    John,
    Thanks, I think I followed that :)
    Your description does indeed fit my experience and clearly what you saw: said static, I thought 'static' but actually was DHCP. I did the same thing on a local Kiwi here but I'm not sure whether I hit the "are you sure" or not. I think I did. Today when I performed the move to static on the remote kiwi, I did it really quickly and could easily have skipped answering that step - thus creating inconsistent states as you describe.

    I think I'll leave it as it is with any test of whether it's now fixed (no pun intended) only being performed when either we lose power again or else next time someone is present to rescue it in case it doesn't come up.

    Glenn n6gn
Sign In or Register to comment.