HFDL extension
[updated 22 nov 2021]
Thanks to the recent excellent open source work of Tomasz (github.com/szpajder/dumphfdl) the Kiwi now has an HFDL extension. Although it will always be less capable and up-to-date than what the original dumphfdl offers (this is true of practically all Kiwi extensions). Note that in the dumphfdl README and kiwiclient README/Makefile are detailed instructions for using dumphfdl directly with a Kiwi IQ stream via kiwi_nc ("netcat"). This is similar to how some people prefer to use Dream DRM + Kiwi IQ stream rather than the Kiwi DRM extension directly due to Dream's features and their familiarity with that application.
There are definitely some problems with the extension. But I wanted to make it available on more Kiwis to make testing easier. In particular there seems to be trouble getting any message decodes initially. For stubborn cases re-opening the extension and clicking the "test" button until you get a decode helps. There may also be issues when multiple channels are running the extension although it works fine in some cases.
Since the EiBi database has labels for most of the HFDL channels a good way to visualize activity on a particular band is to switch to using EiBi (control backslash), select only HFDL, and adjust the zoom until all the HFDL labels of that band fit the waterfall. In the "Bands" menu clicking on the blue band entries selects EiBi labels and automatically sets the optimal zoom. The same happens when clicking on one of the ground station active frequency table (AFT) icons on the map [as of v1.478]. This makes it easy to see which channels have activity. Then you can shift-click a particular HFDL label to bring up the extension on that channel. Shift-click on other labels or use the HFDL extension menus from there.
The "display" menu changes the format of the decoded messages displayed. "DX" shows an abbreviated two-line message per transmission (similar to the ALE 2G extension). "Squitter" is an HFDL term for ground stations transmitting the dynamic "active frequency table" (AFT). Squitter mode will limit the messages to the AFT so it's easier to see which ground stations are using what frequencies. "ACARS" shows only messages containing ACARS message data.
You might also want to reduce the waterfall update rate to "slow" so that you can see more evidence of channel transmission.
The help button has more information on URL parameters. So a compact URL to monitor a particular band might be: my_kiwi:8073/?ext=hfdl,8977,d:1,l:10&wf=s Adjust as required.
A Kiwi that has poor frequency calibration (because it's not using GPS correction and is in a particularly cold/warm room) will exceed the ability of the HFDL decoder to lock to the signal. This has been observed. In the absence of GPS you can do a Kiwi manual frequency calibration against a time station (e.g. WWV/H, BPC etc.) as described in the Kiwi documentation. The header of each HFDL message (highlighted in yellow) shows the frequency correction. Anything more than about 50 Hz won't decode.
From the CHANGE_LOG file:
v1.471,472 November 12, 2021
HFDL extension preliminary release.
Because the decoding happens entirely on the Beagle, and due to the buffering delay of
delivering audio and waterfall to the browser, you will see the decoded text before the
audio/waterfall occur.
BBAI performance improvement. Specify correct compiler cpu type (cortex-a15) and
vector floating point version (vfpv4) for the TI AM5729 cpu chip used by the BBAI.
This causes the fused multiply-add instructions available in vfpv4 to be used.
By comparison the BBG (TI AM3359 cpu chip) uses cortex-a8 and vfpv3.
Comments
Wow! From first suggestion to a working implementation within only ~3 weeks (of much work, I guess). Respect and many thanks for all your work, John!
My first build process failed (like many others), and I had to do this fix. But after that, it worked from the first go. At my location (near Berlin) I was, for example, able to successfully decode:
I can confirm it was sometimes necessary to restart the HFDL extension. (But in my case, it may have only been due to unintended change from IQ to USB by carelessy clicking a label without shift.)
Thanks again and 73, DL7AWL
Edit: clarification
Hi John,
Excellent it works very well.
I'm not an aviation enthusiast, but on monitoring a HFDL channel for a while it seems that the majority of the traffic is the ground stations transmitting what is basically the same uplink information over and over again.
However once in a while there is a downlink signal from a plane in transit, with co-ordinates, which to me seems much more interesting.
Would it be possible to add a further filter in addition to All, DX and Squitter. to just display downlink signals ?
It would be nice to be able to see the tracks of the aircraft on a map, but I realise that I'm getting ahead of myself :-)
Another great addition to the KiWi, which reinforces my view that it's probably been one of the best investments in radio 'kit' that I've ever made.
Thanks again for all your hard work and perseverance.
Martin
Question from the Extension it seems to select IQ but the DX tags are USB, I assume IQ is the correct mode?
IQ is what works here!
Yes. According to instructions in opening post, the (EiBi-)labels have to be shift-clicked in order to get/keep the correct settings. (I also made normal clicks by mistake.)
Excellent thanks, set it going while doing housework but didn't read the first post properly!
@dl7awl Wow! From first suggestion to working implementation within only ~3 weeks
Yes, but remember nearly all the extensions are based on the open source work of others. HFDL could never have been an option without the recent work of Tomasz: github.com/szpajder/dumphfdl Now it was not trivial to port his code to the Kiwi environment. And there was never any guarantee it would work (still some issues to debug). But we got lucky.
@G8JNJ Martin: I've started to look into that. There is no reason not to add the aircraft info to the one-line DX output. I'd also like to create the "active frequency table" from the squitter messages and perhaps look into some mapping. Also, I need to find an open source version of the gigantic database that maps the various aircraft codes into their actual names. That's why this release was called "preliminary", lol.
@Powernumpty Stu: I need to fix the EiBi database to make the HFDL labels select IQ mode and maybe the special optimized passband the extension uses. I did incorporate the changes from the latest version 51 HFDL system table into EiBi (I'll send those changes back to them).
Works well with my Kiwi. Thank you!
When time is available, another humbly requested project is to have an extension to decode the Ham HF APRS on 30 meters!
73,
-Zyg- AF4MP
v1.473 November 13, 2021
HFDL extension improvements:
Additional processing of the messages appearing in the text window. The abbreviated 'DX'
display now contains aircraft info. The 'squitter' display is more compact.
Separate upload/download checkboxes to limit the message direction ('upload' are messages transmitted from the ground stations). Thanks Martin.
EiBi label database:
Clicking HFDL labels now selects IQ mode and the special USB passband used by HFDL.
Labels which automatically open an extension on shift-click now mention this fact in the
mouse-over popup (as a reminder). These are currently ALE, HFDL, Time, CW, FSK and Fax.
Clicking ALE, HFDL and Fax labels set the special passbands used by the associated extensions.
Clicking an HFDL label should also select the corresponding menu entry of the HFDL extension.
Corrections from the HFDL version 51 system table were applied to the EiBi database.
Hi John,
Thanks for the latest update. The message direction tickboxes make a world of difference (for me anyway).
I took a look at some of the aircraft calls and co-ordinates on some of the flight tracker sites, and found that a lot of them were not shown as airborne, as the flights were in remote areas where the ADS-B data had not been captured by hobbyist receivers and fed into one of the aggregation sites.
It struck me that maybe if the KiWi could output HFDL data in a suitable format such as 'dump1090' or 'Beast' provide for ADS-B data then it may be possible to feed existing flight plotting software or an aggregation site, directly with real time data. This may also bring some commercial benefit, from sales to aviation enthusiasts.
Dump HFDL seems to do something along the lines of what I'm thinking (in terms of output data format).
Edit - sorry, ignore the last point as it had already been discussed in the original HFDL thread.
http://forum.kiwisdr.com/index.php?p=/discussion/2405/dumphfdl
Regards,
Martin
Is the SNR something that the extension calculates? I believe that the HFDL stations themselves use it to determine best data rate
@G8JNJ Martin: Although uploading position data to one of aggregation sites might seem like an interesting idea it is specifically warned against by Tomasz in the dumphfdl FAQ for various reasons. Go to https://github.com/szpajder/dumphfdl and search for "Important Note!" toward the end of the page.
@WA2ZKD Jim: Not sure. I need to spend more time understanding the details of the protocol.
@jks re: SNR a quick look at the documentation shows that the GS calculates it and uses it to set the uplink data rate. I don't think the value gets set as data. The decoder in dumphfdl would have to calculate it. I use may limited skills to explore its source!
Understood regarding HFDL data corrupting ADS-B tracks.
Pity as it seem that the problem is simply that the aggregation sites don't know how to deal with HFDL, as they have only been setup for ADS-B. Personally I'd have thought it would be relatively easy to include HFDL data if there were no ADS-B tracks logged for a particular flight.
Despite this, I'm sure the HFDL decoding will still be of great interest to many.
Regards,
Martin
v1.474 November 15, 2021
HFDL extension improvements:
New entries in the "Bands" menu that select an entire HFDL frequency band.
Automatically sets the center frequency, zoom and switches to displaying only
the EiBi database HFDL labels. Eliminates tedious manual setup. Can be set
via URL parameter (see HFDL help panel).
EiBi label database:
Updated to B21 published 14 Nov 21 on eibispace.de
|| menu that select an entire HFDL frequency band.
That option is very helpful - Thank you!
Well done, thanks. The HFDL extension is working quite well with these latest updates.
It would be great if a fourth display selection for ACARS could be added (Up and Download LPDU's containing both "ACARS:" and "message:")
73 Ben
[update: have one now, thank you]
I am now in desperate need of a Kiwi in the EU I can do HFDL development on. It needs to be BBG-based (i.e. not BBAI) and have an open port on your router for ssh access. It needs to have GPS setup, good sensitivity and low noise to maximize HFDL stations received.
Development will mostly happen at night since I am 12 hours out-of-phase with the EU. Please contact me at support@kiwisdr.com
Thanks in advance.
v1.477 November 21, 2021
HFDL extension improvements:
Added map of ground stations with active frequency table (AFT) icons sourced from
decoded squitter messages. It can takes a number of minutes before sufficient squitter
meesages are received to populate all the ground station icons.
Small screens (laptops): because the area available for the waterfall is smaller than
ever use the 'x' and 'y' keys to hide the various control panels and label bars.
Click the "2" button at upper right to restore the map zoom/positioning.
The "split" show menu option lets you view the map ans a small message window simultaneously.
When you specify a band or frequency as an HFDL extension URL parameter EiBi labels
and band zooming are automatically selected. No need to specify "dx=none,hfdl" anymore.
The Kiwi icon shown when the "Show Kiwi" button is pressed does NOT reveal the precise
location of the Kiwi. The lat/lon is rounded to the nearest 2 degrees which is approx.
200 km of latitude and 100 - 200 km of longitude depending on the latitude location
(recall that latitude lines converge at the poles). The purpose of showing the Kiwi
location is to help understand what propagation might be in effect given the time-of-day
at the Kiwi and transmission frequencies of the ground stations.
No aircraft flight plotting on the map yet.
Interesting to see the active frequency table presented colour coded on a map. The limited screen real estate is still manageable on a 13 inch laptop using the x and y keys.
With aircraft positions shown as well the display could become quite crowded though in areas with heavy traffic, unless zoomed in.
Edit: what I didn't notice at first is that clicking the band coloured icons also tunes the Kiwi to the appropriate frequency for a particular ground station, which is really helpful.
Hi John
You may use my Kiwisdr oz1aef.ampr.org (http://85.191.35.166:8073). it is located in Denmark.
Look at it and see if it it useable for you, and I will give you SSH access
when extension is open not all of the right-click menu is visible
Very nice!
I couldn't resist to try an overlay of the HFDL extension map with the MUF3000 data derived from the Digisonde network. Both are almost real time and show some interesting aspects of the prevailing HF propagation.
73 Ben
@WA2ZKD when extension is open not all of the right-click menu is visible
Don't know what you mean by "all". The right-click menu is not displayed in any areas related to extensions (you get the default right-click browser menu instead). Only the waterfall, frequency scale, band scale and spectrum display show the Kiwi right-click menu. Maybe this should be fixed, but that's currently how it is.
I didn't mention it but this map should behave just like the TDoA map: the mouse wheel will zoom in/out. On most laptops you should be able to configure your trackpad to emulate a mouse wheel with e.g. two-finger dragging etc. That's what I use on my Mac.
@benson Is that MUF3000 data available as raw data (i.e. not just an image file) someplace? If so it might be possible to include it as an overlay option on the map like the current day/night overlay. There is the issue of map projection translation, but it should be possible.
@jks
"Don't know what you mean by "all". The right-click menu is not displayed in any areas related to extensions (you get the default right-click browser menu instead). Only the waterfall, frequency scale, band scale and spectrum display show the Kiwi right-click menu. Maybe this should be fixed, but that's currently how it is."
I think he meant that due to the reduced screen space for these areas when the HDFL extension is open and the map and/or text is displayed, the right-click is only possible near the bottom of the screen, at least on smaller screens, so that lower parts of the right-click-menu are outside the screen and some items are therefore invisible and unclickable.
(BTW, this can sometimes be workarounded by multiple hits of CTRL together with - to change the screen scale; the normal setting can afterwards be restored by CTRL in combination with 0.)
Okay, I see. So what I should be doing is shifting the menu up in that case so the entire thing is visible. I can probably do that. For example if you right-click in the S-meter area (near bottom of screen) the browser menu is displayed correctly.
there or may not be an issue..... I may have looked too early today (pre coffee)
v1.478 November 22, 2021
HFDL extension improvements:
Added ACARS display menu option (does not show ACARS data without message content).
Map AFT icons: highlight yellow on mouseover. Auto EiBi label / band zooming when clicked.
Kiwi right-click menu no longer clips when positioned near screen bottom or right edges.