The KiwiSDR 2 online store is open for orders! Please visit kiwisdr.nz

TDoA UI improvements

2»

Comments

  • No update today. I need to get more stuff working.

    @martin: Good idea.
  • @G8JNJ: I will have to think how this can be done (figure of merit per KiwiSDR).

    @jks: it would be helpful to be able to download the IQ files used for a given TDoA analysis.
  • @Christoph: Okay, I'll add a download icon/link to each sampling Kiwi in the list.
  • Hi Christoph,

    "@G8JNJ: I will have to think how this can be done (figure of merit per KiwiSDR)."

    Thanks for considering the suggestion.

    I think it may also help towards implementing any further automation of the overall process and allow John to analyse the server stats in order to determine the 'best' KiWi's to use for TDoa.

    Regards,

    Martin - G8JNJ
  • FOM might be a score derived from successful results in a previous TDoA run. I am frustrated how poor many stations are in CONUS, that is ones that lack adequate sensitivity and relative freedom from spurs. The problem with too many is a bad antenna system. Marco added a feature to his scheme that allows graying of poor ones.
  • Sorry for the delay. I've been chasing a Firefox-specific problem for days now. There should be a new update tomorrow with quite a few improvements.
  • Hi John,

    Would it be possible to load up the list of KiWi's and have an option so that it will wait until all the listed SDR's are available for TDoA use then automatically trigger the sampling ?

    This would save a lot of hassle waiting for specific 'key' KiWi's to have a spare slot.

    Regards,

    Martin - G8JNJ
  • jksjks
    edited July 2018
    Alright, it was really rushed, but v1.208 is out. I hope not too many things are broken. From the change log file:
    v1.208  July 21, 2018
        Commas in frequency entry field work again (broken since keyboard shortcuts added).
        Can now quickly change passband via entry in frequency entry field:
            Type "freq/pbw" or just "/pbw" where pbw is the passband width in Hz
            or the low and high edges with "low,high".
            Examples:
            1234.56             set f=1234.56 as usual
            1234,56             set f=1234.56 using comma as decimal separator
            234.56/1200         set f=234.56 and pbw=1200 around current pb center
            /600                set pbw=600
            /300,3000           set pb=(300,3000) i.e. usb 2700 Hz wide
            7255/-2600,-300     set f=7255 and pbw=(-2600,-300) i.e. lsb 2300 Hz wide
            [/nnn doesn't adjust about current pb center if center is non-zero -- will be fixed in v1.209]
        Facility added to load certain javascript asynchronously (e.g. Google maps).
        
        TDoA extension improvements:
            New URL parameter "submit:" that will automatically start TDoA on webpage load.
                e.g. ...?ext=tdoa,submit:,F1JEK,DL6ECS,SWL/JO21JN,DCF77
                There is a 6 second delay allowing the initial status inquiries to complete.
            6 character grid square used for ID instead of old "KiwiNNN" (Thanks Martin).
            Duplicate IDs distinguished by appending /N to the ID (N = 2,3,...)
            Multiple Kiwis reporting identical locations on map dithered slightly so all are visible.
            More reliable communications protocol with server should prevent hangs.
            Submit button changes to a stop button while TDoA running (feature may still have bugs).
            "Most likely position" marker now shown on result maps.
                This in addition to reference position if one has been specified.
            Most likely position lat/lon shown at panel top along with cyan Google map link.
                Triple-click on lat/lon value to copy/paste.
            Green link icon at panel top has URL with current TDoA extension configuration:
                Right-click while hovered on link to create configuration bookmark or
                click to open in a new tab/window (less useful).
                Link is continuously updated as map is changed, host list changed etc.
            After sampling each host shows a download icon for the recorded .wav IQ file.
    
    WA2ZKD
  • Hi John,

    The latest additions are really great :-)

    The map is looking a bit crowded now that more TX sites have been added.

    Would it be possible to be able to tick box switch these on / off as required ?

    Regards,

    Martin - G8JNJ
  • Marco added a scheme in his SNR plot to save the state of some "bad" stations in the local cookies. Perhaps something similar could be added in the TDoA scheme to save preferred stations with a Preferred/All toggle
  • @Martin: Yes, I had been thinking the same thing.
    I have been working on your auto-retry idea. It wasn't quite ready for the last update. Lots of "corner cases" to get right.
  • Hi John,

    "I have been working on your auto-retry idea"

    Great that's really good.

    However thinking about it a bit further.........

    I think all KiWi's that are on-line need to be shown on the map, but the ones that are fully occupied or don't have the required number of GPS fixes should be perhaps grayed out or similar, so that they can still be selected and placed on the list ready to auto run when they finally become available.

    When the TDoA map is available could it be shown as default instead of the original KiWi selection map ? This would help indicate that the results are back. also it would facilitate the next suggestion.

    When a run has been completed could a 'snapshot' of the KiWi GUI including the control panel and waterfalls be grabbed and made available for local download (like the FAX images) ?

    This would give confidence that the target signal was present when the TDoA ran as it could be recalled later. This is because I've found that sometimes when I've started a TDoA run and gone away from the PC then returned later, I've not been able to recall the results. I don't know if this is because of a server or web browser session time out, but if a screen grab is stored locally on the KiWi at least I'd be able to examine it and recall the parameters etc if I wished to repeat the run at a later stage.

    I've also noticed that the TDoA map takes a long time to display (or not at all) but the TDoA combined map displays almost instantaneously.

    One final thought. Would it be possible to combine all the I/Q wave files together into one zip file and also include a text file with the URL string for download ?

    Regards,

    Martin - G8JNJ
  • jksjks
    edited July 2018
    the ones that are fully occupied or don't have the required number of GPS fixes should be perhaps grayed out
    There are different ways of doing that, some of which I don't like. Mass periodic polling from the extension is no good. Sending out hundreds of status queries every so often (one to each Kiwi) is a bad idea. That's why its only done as each Kiwi is selected for addition to the list (and would be done again on an auto retry). The data fetched from kiwisdr.com to initially populate the list is derived from the same data used to feed kiwisdr.com/public and sites like rx.linkfanel.net This list is rebuilt every two minutes, but each Kiwi updates info used by the list every 15 minutes (randomly distributed over time). So this info is not timely for short-term changes to channel occupancy or GPS recency.

    This is a difficult problem. The solution for kiwisdr.com/public has to scale for ever-increasing numbers of Kiwis. Maybe the Kiwis should push status updates on more than a fixed schedule? Like when transitioning from "all channels full" to "channels available" (similar for GPS) but rate-limited so as not to overwhelm kiwisdr.com. Lots to think about..

    When the TDoA map is available could it be shown as default instead of the original KiWi selection map ?
    Great idea. Can't believe I don't already do that.

    When a run has been completed could a 'snapshot' of the KiWi GUI including the control panel and waterfalls be grabbed and made available for local download (like the FAX images) ?
    Interesting idea. Not sure how I'd combine all those pieces (multi-page PDF?)

    I've not been able to recall the results ...
    I see your point about storing the waterfall at the moment of TDoA completion in case the target signal is transient. The TDoA result files are removed from the server one hour after they are created (arbitrary). So this is why when you return after a long absence the "broken link" icon might be displayed. Remember the green icon link should always contain all information necessary to repeat the run.

    I've also noticed that the TDoA map takes a long time to display (or not at all) but the TDoA combined map displays almost instantaneously.
    For me they load at the same speed. What happens if after a run you try loading the combined map first? The first one has a new Google map generated that can receive markers and the result overlays and contour drawings separately from the Kiwi selection map.

    Would it be possible to combine all the I/Q wave files together into one zip file and also include a text file with the URL string for download ?
    Could be done, yes. I have resisted making the control panel larger and more complex. But it might be unavoidable.
  • Is anyone still having problems with clicking on new hosts going to the bottom area of the list leaving empty slots at the top? Or blank entries with "id_N" (N=0, 1, ...) when submitting? I have never seen these problems myself, but have made some fixes which may prevent them. If you have failure case I would like the know the exact repeatable sequence. Thanks!
  • jksjks
    edited July 2018
    v1.209 is out. From the change log file:
    v1.209  July 22, 2018
        TDoA extension improvements:
            When TDoA complete automatically switch to result map.
            Switch option bar to 'off' (and restore) when TDoA running. Obscures the map less.
            Option checkboxes moved to right of map display.
            Added checkboxes to show/hide reference stations and day/night overlay.
            Added more references stations.
        Fix passband width adjustment when passband is offset from carrier (i.e. ssb, cw)
        Use pointer cursor for menus and checkboxes.
    
  • Hi John,

    "Maybe the Kiwis should push status updates on more than a fixed schedule? "

    That seems like a sensible idea.

    "When the TDoA map is available could it be shown as default instead of the original KiWi selection map ?"

    That change is great. However I now find that I think the result map is the KiWi selection map, so when I start a new run, I sometimes find that the zoom level I'm seeing on the results map what I'd expect to get back, but the selection map is still at it's original zoom level and center position. Maybe the selection map and results maps all have to be synchronised, so that a change in one affects the settings for all of the others. This would save having to flip backwards and forwards between selection and results maps, when you are trying to center up the map position and scale for a second (or more) run to locate the source if it was off the edge of the map on the first run.

    "Not sure how I'd combine all those pieces (multi-page PDF?)"

    I don't think everything needs to be grabbed, just the result map and the waterfall or a snapshot of the whole KiWi GUI.

    As you say the URL can still be recalled, so that's good too.

    "What happens if after a run you try loading the combined map first?"

    The TDoA map still doesn't display, however in some instances this may be due to not having an clear result. I've occasionally noticed that the combined map may have multiple 'cross over' points and not just one distinct hot spot, in which case the combined map may not indicate anything. However under these circumstances it would still be worthwhile having some sort of indication that the overlay has loaded correctly, even if there is nothing to be seen.

    Ideally when viewing the combined map, it would be great if you could 'switch' off the contribution from individual KiWi's so that it's easier to identify which ones are providing good clear plots and which ones are not really adding anything at all.

    "Is anyone still having problems with clicking on new hosts going to the bottom area of the list leaving empty slots at the top?"

    It was still doing it occasionally with v1.208.

    I'll let you know if I see it again with v1.209 onwards.


    Other stuff

    "Then TDoA complete automatically switch to result map."

    That works great :-)

    " Option checkboxes moved to right of map display. Added checkboxes to show/hide reference stations and day/night overlay."

    That looks a lot better and makes more sense.

    Other other stuff...

    I'm still not sure about the circles indicating KiWi's when they are located close together, I think the zoom level to see individual KiWi's is perhaps a bit too far zoomed in. However I realise that getting this right every time based on the KiWi spacing is difficult to achieve.

    If a selected KiWi is enclosed within a circle then the circle should change to yellow, as is the case with individual KiWi markers.

    Overall the TDoA extension is really impressive.

    My only major complaint is that I'm spending far too much time trying to ID the source of many signals that I've wondered about over the years :-)

    Brilliant stuff - Thanks John & Christoph - please keep up the excellent work.

    Regards,

    Martin - G8JNJ
  • There are different ways of doing that, some of which I don't like. Mass periodic polling from the extension is no good. Sending out hundreds of status queries every so often (one to each Kiwi) is a bad idea. That's why its only done as each Kiwi is selected for addition to the list (and would be done again on an auto retry). The data fetched from kiwisdr.com to initially populate the list is derived from the same data used to feed kiwisdr.com/public and sites like rx.linkfanel.net This list is rebuilt every two minutes, but each Kiwi updates info used by the list every 15 minutes (randomly distributed over time). So this info is not timely for short-term changes to channel occupancy or GPS recency.

    This is a difficult problem. The solution for kiwisdr.com/public has to scale for ever-increasing numbers of Kiwis. Maybe the Kiwis should push status updates on more than a fixed schedule? Like when transitioning from "all channels full" to "channels available" (similar for GPS) but rate-limited so as not to overwhelm kiwisdr.com. Lots to think about..

    I like this. I think that ideally, each KiwiSDR should push a status update whenever any of the /status info changes at all, even.

    In order to scale, and avoid mass periodic polling which doesn't scale, each KiwiSDR should maintain and push all of its info itself - including SNR metrics, that are still polled and generated externally.

    There's a number of ways to make sure the kiwisdr.com/public endpoint scales up: load balancing, DNS round-robin... Or, the URL of the status push endpoint could include a (partial) hash of the ID field - MurmurHash, MD5, merely to ensure even distribution, which could be used to pre-balance all the push calls. Then it's a matter of backend to merge, but at least the hash trick ensures balancing where each node is always stored in the same place.

    An additional, unrelated suggestion: could there be a field to manually enter the URL of a KiwiSDR into the TDoA list? That way, people can make use of unlisted KiwiSDRs, and the extension stays partially functional if kiwisdr.com/public goes down.
  • jksjks
    edited July 2018
    Could there be a field to manually enter the URL of a KiwiSDR into the TDoA list?
    Maybe not obvious, but there is already. Both the ID and URL fields in the list table are editable. Hit return after a new/modified entry. Status check occurs when URL field changed. Add to an empty slot or edit an existing one. At submit time any slot without a checkmark won't be used.
  • jksjks
    edited July 2018
    v1.210/211 is out. From the change log:
    v1.210  July 25, 2018
        Correct dictionary-order sort of extension names in lists and menus.
        TDoA extension improvements:
            Added reference station categories.
            Admin page extensions tab lets you limit number of channel used by TDoA service.
            Admin page security tab lets you disable sending GPS timestamps completely.
            Moved "TDoA ID" setting from sdr.hu tab to TDoA tab on extensions page
            Updated help info.
    
    Note the channel limitation is not yet ignored when you use Kiwis on your own local network. That's a little difficult to implement.
  • Hi John,

    The new map categories are good.

    I notice that when using Chrome on my iPad the black map background that the key is displayed against is entirely covered by the map when it opens up, making the key difficult to read against map background.

    OK here's my first lot of suggestions for the day.............

    Would it be possible to place co-ordinates in the DX tags so that the station would appear on the TDoA ( and possibly a future KiWi selection map) ? Duplicates would need to be filtered for sites with multiple transmissions / frequencies. A new map category would need to be provided for DX tags.

    Would it be possible to place the TDoA url in a DX tag so that double clicking on the tag would bring up a previously stored TDoA for that station. Maybe some form of indicator (bold surround or colour spot) could be added to the tag to indicate that it has an embedded TDoA capability.

    Regards,

    Martin - G8JNJ
  • TDOA DF: UI Idea for quick location determination

    Let me preface this by saying I dont have the IT or programming skills to implement what follows. But I do have the communications experience to know what follows is do-able with a dense enough sensor network.

    Firstly, I will say I experimentally went through the process defined here: https://www.rtl-sdr.com/kiwisdr-tdoa-direction-finding-now-freely-available-for-public-use/ in order to try to determine the source of some intentional QRM on the 40m band. I found the process cumbersome in extreme, and the target signals were gone after I spent 30 odd minutes manually hunting for sites that could detect them.

    Here is how I would lay it out.

    1) I would capture the location of the user. Simply ask them or do a locate using the usual internet libraries for determination of approximate geographical coordinates. This is based on the simple assumption that a signal of interest may be received at the users location.

    2) I would accept a frequency from the user.

    3) Using the frequency provided, the user's location, and the Solar Flux Index of the moment, I would do a VOACAP calculation to get high probability propagation bands from the user in order to automatically select some likely KiwiSDR receivers, TDOA-capable preferred, but for the first pass, any receivers in the high probability band will do. Alternatively, the user can select 1 receiver where the signal is present in lieu of their own location if they are hunting a signal that is not available at their location. Do VOACAP analysis from the selected receiver.

    4) I would generate a list of 5 or 10 receivers showing a 1 inch high waterfall maybe 5 kHz wide centered on the frequency selected. Preferentially select receivers that are a single-hop from the user's location or initial receiver. I would show them muted by default, and allow the user to select receivers by listening to them one at a time independently based on whether the signal is present or absent in each (visualized modulation will also suffice). When some receivers are locked in, if enough TDOA receivers are present in the selection, go do the rest of the TDOA calculation.

    5) If not, repeat the VOACAP analysis from receivers that do detect the signal to provide further replacement receivers in the select list for any receivers that do not. At this level, there are probably enough propagation rings to select only TDOA capable receivers to add to the list. The user does another pass through the list identifying good receivers. Again, if enough are selected, do TDOA.

    6) Repeat 5 until there are enough receivers, dropping single hop limitation, or until it is obvious that there arent enough receivers to do a successful computation.

    7) A receive array configuration developed as described above could be saved for future use to bring on line instantly in case favorable propagation appears at a later date, or to repeatedly police troublesome frequencies and/or emitters. Having a partial solution ready to go might significantly improve the efficiency.

    8) The manual selection process could be automated to a degree by use of correlation. A user selects the best example of the target signal, and correlation with other receivers could autoselect receivers that are good and then go through the total process of selecting further receivers.

    A few ideas that might improve things from an operator's perspective...

    73;

    Bob KV4PC

    Powernumpty
  • Interesting points, Bob, thanks!

    1) I would capture the location of the user. Simply ask them or do a locate using the usual internet libraries for determination of approximate geographical coordinates. This is based on the simple assumption that a signal of interest may be received at the users location.

    2) I would accept a frequency from the user.

    The TDoA extension and signal to analyze are driven by the currently open receiver, and what it's tuned to. You probably want to TDoA a signal because you've found it on the current receiver, because it's received there. So for this approach, I think it might make more sense to use the current receiver's location instead rather than the user's; and for example have the TDoA extension map center on it when it opens.

    3) Using the frequency provided, the user's location, and the Solar Flux Index of the moment, I would do a VOACAP calculation to get high probability propagation bands from the user in order to automatically select some likely KiwiSDR receivers, TDOA-capable preferred, but for the first pass, any receivers in the high probability band will do. Alternatively, the user can select 1 receiver where the signal is present in lieu of their own location if they are hunting a signal that is not available at their location. Do VOACAP analysis from the selected receiver.

    As far as I understand, VOACAP coverage predictions are calculated from a time and date, frequency, and the origin location of the signal; I'm not sure VOACAP can accomplish much with just the location of somewhere a signal is well-received, so that wouldn't really work.

    7) A receive array configuration developed as described above could be saved for future use to bring on line instantly in case favorable propagation appears at a later date, or to repeatedly police troublesome frequencies and/or emitters. Having a partial solution ready to go might significantly improve the efficiency.

    It is true in my experience that even if you know exactly which good receivers you want to use, it can be cumbersome and time-consuming to reselect them all one by one - which can be a serious hinder for time-critical signals.

    8) The manual selection process could be automated to a degree by use of correlation. A user selects the best example of the target signal, and correlation with other receivers could autoselect receivers that are good and then go through the total process of selecting further receivers.

    If I understand correctly, the TDoA algorithm basically computes such a correlation between each two selected receivers. I don't remember for sure, but I guess it even already takes into account how well it correlates when calculating results. So the point here is that you need to have already taken samples from selected receivers before you can judge if they're good and whether you want to use them.

    But it is true that we could use a two-step approach, one where we first select and probe receivers, maybe analyze a sample and display a small waterfall or compute a quick SNR measure, and choose which ones we keep, before we launch the TDoA sampling and calculation for real on all the selected ones.

    Also, now that KiwiSDRs compute themselves and report a general SNR score beforehand, that data is available to display in the TDoA extension and help choose receivers too.

Sign In or Register to comment.