TDoA UI improvements

edited July 2018 in TDoA topics
This discussion was created from comments split from: Direction Finding and linking existing KiwiSDRs.
«1

Comments

  • edited July 2018
    "For example if the kiwirecorder run launched by the kiwisdr.com server doesn't act as expected there is no timeout detection."

    Maybe it should just time out after say 40 seconds on the basis that if it's not complete by then, it isn't going to be ?

    Thinking out loud........

    Does the sampling period always need to be 30 seconds long or could it be made adjustable ? I'd have though that trying to find correlation over a longer period would take up a lot more processing, and that a lot of signals don't need that much 'effort'. It's also problematic if you are trying to pick out one transmission from an amateur net or aircraft comms, as each 'over' tends to be quite short. In a 30 second period you are likely to capture more than one transmitter. Short burst 'once in a while transmissions' like ALE are difficult to capture, so I wonder if the record function could incorporate some sort of 'squelch' type trigger ?

    "If you're zoomed out too far then the Octave TDoA process will run for a long time generating the long coastlines for the maps."

    Ah that makes sense - would it be possible to give a countdown or thermometer indication, hopefully not with a 'Microsoft' scale :-) of the likely duration based on the level of zoom, so that you know you have time to go away and make a cup of tea ?

    Regards,

    Martin - G8JNJ
  • Hi John,

    The new zoom arrangement is a bit problematic if you are trying to perform a first 'fix' on the HF bands when you need to use KiWi's that are some distance away from the likely source and the map is zoomed out to show a 1/2 world view.

    Also would it be possible for the initial map to be centered at a moderate zoom level on the KiWi location that is using the extension ?

    This would save having to re-zoom each time the extension is opened.

    Regards,

    Martin - G8JNJ
  • jksjks
    edited July 2018
    @Martin Minimum zoom requirement: The latest change requires a map zoom >= 4. The initial map shown the entire world with zoom = 2. The Octave script fails for zoom = 2 and = 3 takes a long time to plot. But I see your point about needing a large-scale view for an initial search. I'll reduce the minimum required from 4 to 3.

    This becomes moot if we manage to figure out how to plot the TDoA result on a Google map rather than the plain coastline maps returned now. This would have the added benefit of allowing you to immediately zoom in (in satellite mode) and start searching for the antenna field. A button could toggle the TDoA solution map overlay (or change it's transparency value) so it wouldn't obscure the map.

    Initial map centered at mid-zoom of current Kiwi: Does this really make sense? Like other Kiwi settings (e.g. freq/mode) what about saving the last position and zoom of the map in a cookie? First time users would see a full world-view which I think is the right default behavior.

    Sampling period always 30 secs: A question for Christoph. I don't understand if that is arbitrary or not.

    Timeout/cancellation: I'm working on it..
  • Hi John,

    Yes a stored cookie would be good.

    How about storing the last used list of KiWi's for each quick map zoom region ?

    My thinking is that if folks are trying to locate a 'mystery' signal they would probably want to check out different areas of the globe before 'homing in' on a particular continent. Having the map quick zoom bring up the appropriate map and a pre-loaded list of SDR's (chosen during previous sessions) would save a lot of setup time, especially if you are trying to track down a 'fleeting' signal.

    Most of the time Id probably just be interested in the UK or Europe, so having my 'favourite' KiWi's set-up and ready to go would be really useful.

    Having some other pre-sets that could be pre-loaded with stored values of the KiWi's to be used, the map zoom, the frequency and bandwidth would be really handy for capturing elusive signals such as ALE.

    Maybe primary and secondary KiWi's could be defined so that if the first choice is not available a second one would automatically be used instead when the sampling starts.

    Regards,

    Martin - G8JNJ
  • Really great developments around this extension.

    I find myself cycling between the computed TDoA maps (and sometimes td plots) to find which sets of Kiwi nodes work best.

    It might be an idea to show these maps for a brief period sequentially before showing the final combined map. If there was a re-compute function for the server run octave program after deselecting node(s) but still using the already sampled data the results would appear faster.

    Some further ideas can be glanced from the link below which is a specialized software program.


    CRFS Software: TDOA Geolocation

    Best regards, Ben.
  • Hi John,

    Thanks for the latest updates, I'm having great fun :-)

    Some further observations / suggestions.

    The basic TDoA is working extremely well.

    My main problems are mostly associated with choosing the best KiWis to use.

    When I double click on a KiWi from the map in order to take a listen before adding it to the list, when I return from the new tabbed webpage that opens with the selected KiWi in it, I often forget which KiWi I have chosen when I return to the TDoA extension, as the descriptions on the map don't always tie up with the description on the KiWi.

    Would it be possible to:-

    1. Highlight in a different colour the KiWi's already chosen and added to the list on the map, so that it's easier to decide where the next ones should be located.

    Maybe some range circles could be drawn around each selected KiWi location, so that it's possible to get a better idea of the coverage and overlap between KiWi's ? The size could be defined by the zoom level and possibly the frequency being used ?

    3. Highlight in a different colour the last KiWi to be double clicked on the map, so that it's easier to add that one to the list if it's suitable.

    Also......

    If the map extents are too great for Octave to handle, would it be possible just to re-run just the Octave submission part rather than having to take a further sample ? If the transmission was fleeting, it may be necessary to change the map parameters, without trying to obatain another capture.

    Regards,

    Martin - G8JNJ
  • The offline version prints most likely DF'd coordinates. Are there any plans to bring this over to the online version?
  • Hi Daniel,

    Not sure if your question about on-line version was related to the CRFS software. If so, I really don't know but looking at their website it seems either DFs, TDoAs or a combination of both is used and available right after processing. Seems actually that professionally nowadays sigint / comint / geo-location is often done "after the fact" using IQ recorded transmissions of large chunks of the spectrum. There are of course situations that need almost instantaneous results.

    Another major manufacturer of signal analysis and geolocation equipment is TCI with some quite interesting material at their website. One of their videos is inked below.


    NextGen HF Detection, Intercept and Direction Finding


    Previously Kiwi user WA2ZKD mentioned also the Corsair products with link http://www.entropixllc.com/

    I noted sometimes that good looking examples show operation at vhf and higher frequencies where unlike HF ionospheric propagation effects are almost absent.

    Of course it is totally unfair to compare products from companies with large man power resources / budgets and systems costing easily 100 times as much with our Kiwis having only few developers. I just mentioned those high-end products as examples and perhaps for ideas of how geo-location results are presented elsewhere.

    Already amazing what has been accomplished so far with the Kiwi TDoA extension and linkz directTDoA project. Hopefully Christoph's newer models will also be implemented in future.

    Best regards, Ben
  • I was thinking of Christoph's code. I don't know about the CRFS software.
  • Sorry everyone, I am compiling all your ideas and suggestions. But I can't reply in detail because I'm drowning in emails and posts and there's hardly any time to get real work done.

    @daniel I have not had time to get linkz's directTDoA project running on my laptop. I've just looked at his screenshots. I also still need to integrate Christoph's latest code which includes the likely solution center marker.
    Tonda
  • Today's v1.203 update defaults to plotting the results on a Google map. There is an option to view the old maps as well as one for toggling the "heat map" overlay on the new maps. It is now far easier to match a TDoA solution with a sighting of a potential antenna field on the Google satellite map. A distance ruler is shown on the map to help give an idea of the scale at a particular zoom level.

    Many thanks to Christoph for figuring out how to do the map projection conversion.
  • jksjks
    edited July 2018
    It turns out v1.204 is what you need. If you already have v1.203 you won't get v1.204 for another 24 hours. But you can force an update to v1.204 immediately by just restarting the Kiwi. This only applies to Kiwi owners of course.

    Everyone else, just look on sdr.hu or rx.linkfanel.net for a Kiwi running v1.204. It doesn't really matter what Kiwi you use to run the extension.
    Christoph
  • Look great!

    The composite map is very helpful.

    For the composite map it looks like the overlay of one of the KiwiSDR combinations is off. I will check the octave code. This may be a rounding issue.

    10222 kHz

  • edited July 2018
    Hi Christoph & John,

    I've updated to v1.204 and the new maps are great :-)

    However the TDoA plots seem to be miles out ?

    I've tried repeating several of the signals with known locations I'd tried over the past few days and none of them seem to be even close.

    It may be strange propagation or some other factor, but up until now I've have about 75- 80% success rate.

    Is it just me ?

    Edit - It's just me (I think) most of the stations I tried were around 5MHz.

    I've just checked propagation for the UK and find that for the first time in a while the foF2 value is above 5MHz, so it's likely that NVIS propagation is kicking in and messing up the delays.

    Regards,

    Martin - G8JNJ
  • @jks: I understand why the overlay is off: when there are gaps in the overlay (low resolution and very sharp area), pixels in the gaps do not get set in the png for the overlay. I committed a fix for this.

    @G8JNJ: I also noted E and F-layer propagation at 5-6MHz yesterday -> cross correlations have two peaks and this biases the TDoA maps. There was no such problem >10 MHz.
  • Indeed version 1.205 seems to help with the TDoA processing.

    Appears that more often than previous days after using submit "sampling started" is announced but never completed.
    I tried sampling requests from different Kiwis worldwide with similar results. Could be internet bottlenecks or temporary high loads at the server.

    A time-out xx seconds after submit if the sampling can not be completed and the possibilty to re-submit would save the need to set-up again for another run.

    Of course a way to store TDoA session set-ups would also help when making further attempts.

    BTW no degradation of the kiwi proxy service I am using has been observed.

    Best regards, Ben.
  • Hi John,

    When using my own KiWi for TDoA I choose my SDR, then change the port number in the list to that of my 'Private' KiWi on 8077.

    This allows me to use my second KiWi without tying up the main 'Public' one.

    It's worked well up until today, but I now find that when I do this I just get a "Kiwi is offline" error message.

    Regards,

    Martin - G8JNJ
  • jksjks
    edited July 2018
    @Ben, I have noticed this also. It's not supposed to happen. One advantage of running the backend processing on a central server (that I have access to) is that I can see the failure messages in the log and then "replay" the run since the recorded sample files are not removed for one hour.

    On the "todo" list is an abort/timeout feature, configuration saving of some kind and "re-compute without re-sampling" when only the map settings have changed or when a sampling station has been dropped from the list.

    @Martin, Sorry, didn't implement the new "offline" detection quite right. Will be fixed in today's update.
  • An unfortunate thing can happen for us folks living around metropolitan areas where we can get more than few Kiwis running. On the TDoA map, the little name box for one Kiwi can cover up others within the area around it. See Northeastern US for an example - WA2ZKD/KD2OM, ALA1520NLP/WB4KLJ, and AB1LD/W1EUJ/N2LAW/AB1KW. Hard to see the difference until you zoom in.

    Maybe the WSPR style markers could use a change, maybe just to a dot - keep the mouse-over detail when needed.
  • @W1EUJ: Fix coming in tomorrow's update, and it's pretty cool.
  • edited July 2018
    Hi John,

    A very minor point with so many items on the to-do list: the map choice either map or satellite on the Kiwi map should probably initially also be carried over to the results pages. Finding a small island in the Pacific with night overlay like Tahiti will be a lot easier. Best regards, Ben



    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/b6/b90769a0255f478e4b718b2f213778.png
    https://forum.kiwisdr.com/uploads/Uploader/a4/de4529b44873135bf7e2a7b2623d2b.png
  • @benson: Done -- two line fix (code to support that UI is over 1k lines now, :#).

    Nice FUM capture. I've been struggling to get a very strong MS-188-110 here in NZ. I know where it is, but can't get good results no matter what I try. It is possible the waveform doesn't have good properties. I don't really understand how important the waveform is to the correlation function. So much to learn..
  • v1.207 is out with lots of changes. Use the URL parameters to create browser bookmarks with your favorite presets. From the change log:
    v1.207  July 17, 2018
        TDoA extension improvements:
            Allow extension parameters in URL, e.g.
                ...?ext=tdoa,lat:49,lon:7.5,z:6,F1JEK,DL6ECS,DCF77
                Will zoom the map to specified lat/lon/zoom and select specified Kiwis and ref station.
                lat/lon/zoom is displayed at control panel upper right for reference.
            Cluster markers when they are too close together:
                Hover mouse over cluster for list of markers it contains.
                A cluster single-click zooms +1.
                A cluster double-click zooms in to reveal all the markers.
            Show Kiwi markers on new maps (just like they are shown on old maps).
            Second double-click of reference marker goes back to last map position/zoom.
            Kiwi map type selected is carried to result map type.
            Added additional reference stations:
                WWV/B, NMC, KPH, JJY/40/60, NDT/JJI, NDI, JXN, VTX3/4, DCF49/HGA22/DCF39, DDH47
    
  • Hello John,

    the cluster markers are very good, but my Kiwi's are so close together, so i cannot click them. Hm?
  • There are a few problems like this that need to be fixed. I think I'll have to dither the lat/lon a little bit for Kiwis that report identical locations. I had to do this manually for the DCF77/DCF49 reference markers since DCF49 supposedly transmits from the same facility as DCF77.
  • @jks Some observations using v1.207:

    a. Sampling started but not completed cases. Tried various browsers, cleaning caches, etc but find often the thing that works best in those circumstances is opening a new browser tab. Of course not so convenient if wanting to try the same run with other kiwis selected or deselected.

    b. To do the above as you indicated an URL can be made up manually as a preset to speed up another session. If later there is a possibility to generate,save and recall these from the TDoA panel that would be even better.

    example of such an URL 192.168.1.22:8073/ext=tdoa,lat:40,lon:30,z:5,F1JEK,Qatar,OE5EAN,SM2BYC&f=26.70iqz14&pbw=200

    c. MS-188-110. There is an 8-PSK signal on 12924 kHz and the only way to get a ToDA solution seems to be to narrow down to a single PSK channel. Not sure if that can done for the variant of the 188-110 signal you tried. In the specs there is an unmodulated doppler reference tone which might be a good one to zoom in on. If you can give the frequency & time others can have a go at it.

    I guess the difficulty using the normally selected bandwith which can contain up to 39 parallel PSK channels it is hard to come up with a single correlation value needed for each of those 1 sec snapshots.
  • Hi,

    I also agree with Ben, it would be useful to be able to generate the URL from the TDoA extension so that it's easy to cut and paste if you wish to repeat the run.

    Would it also be possible to generate a Lat & Long value for the TDoA solution, that can easily be cut and pasted into Google Earth / Maps or similar.

    Could the last used I/Q bandwidth setting be saved as a cookie, so that I don't have to set it all up again, if I have to perform a browser refresh

    The clustered KiWi's being shown as a single numeric blob is a good idea, but it's still a pain to have to zoom in to be able to individually select them. Would it be possible to select from the drop down list that appears when you mouse over the blob ?

    Incidentally, I've also had difficulties trying to TDoA one of the 'High Frequency Trading' test transmissions that are popping up around 14900KHz I'm not sure of the modulation scheme in use, but unlike most digital modes it seems to resist TDoA correlation.

    Regards,

    Martin - G8JNJ
  • @benson, @Martin: I was working on some of these issues yesterday after the release. Today you'll get a little green "link" icon like the one in the main control panel. It has the parameters just as Benson suggests and is updated immediately as you change the map or add/remove Kiwis from the list. While hovered over it you then right-click and your browser should present a menu that includes saving the link as a bookmark or copying it. I'll also save this as the extension cookie value that is used to setup the extension if URL parameters are _not_ present.

    "Started but never completed": The TDoA tasks on the server-side actually finish okay in the cases I've looked at. It's just the extension isn't being notified properly. I'm still looking at this.

    TDoA solution lat/lon cut/paste: I'll do this today.

    Marker cluster individual selection: This is a tough one. The little list presented is an HTML "tooltip" and can't be made a selectable list without using a custom object. So this would lake some effort. I already have code for the Kiwi custom right-click menu, so maybe I can reuse that.

    The other thing the marker cluster icons should really do is turn yellow/red (vs blue/red) when a marker "inside" them has been selected for the list (assuming you have zoomed-out so the marker has now become inside a cluster).

    The list of fixes needed goes on and on.. The "KiwiNNN" IDs need to get replaced by something else which is constant and guaranteed unique. Right now the NNN part is simply the Kiwi's position in the list of available Kiwis. But this changes when the list is rebuilt (every two minutes currently). That is no good if you are saving bookmarks containing them!

    Declaring a Kiwi "good" for TDoA only if it's had at least one solution in the last minute may be a little too harsh. This tends to make Kiwis with marginal reception (even temporary) appear and disappear from the map. I'm still thinking about this.
  • Hi Christoph & John,

    Would it be possible to provide some sort of numerical 'figure of merit' or percentage confidence estimate of the TDoA result, and contribution per KiWi.

    I realise that the size of the plot performs this function, but sometimes when I review the separate pairs of plots one KiWi is either not contributing much to the end result or is actually making it worse. If it was possible to quickly see which KiWi's were providing the best plots, it would be easier to cut the under performing KiWi's and make a better selection.

    Regards,

    Martin - G8JNJ
  • The "KiwiNNN" IDs need to get replaced by something else which is constant and guaranteed unique

    How about using the QRA locator ?

    Perhaps with a/1 /2/ 3/ ect. if the locator square is shared ?

    Regards,

    Martin - G8JNJ
Sign In or Register to comment.