dumphfdl
Hello all together,
after the great success of ALE 2G, could you add HFDL to it? There is a new Linux software called dumphfdl. Here is the link with more information:
https://www.rtl-sdr.com/dumphfdl-a-multichannel-hfdl-decoder-for-sdr/
Thanks and 73
Josef
DE3JGA
Comments
I'd have to analyze the waveform demodulation code and try and gauge how intensive it is in terms of Beagle compute resources.
The next problem is that you guys won't be happy just seeing screens full of ACARS messages. You'll want some sort of mapping like all the desktop packages have. That's much more difficult, although having existing mapping in the TDoA extension makes this a bit easier (the mapping framework is there at least).
I'll give you an example: The ALE waveform is quite old and simple. Should be easy to demodulate on the Beagle, right? Well, it took me a long time to figure out it was very sensitive to the quality of the sample rate conversion used. Worse, the portion of the code that does signal acquisition was a huge processing bottleneck given how it was written (brute force + floating point). This is completely unnoticeable on a desktop pc. It wasn't until I figured out I could rewrite it as integer code that it ran in a reasonable amount of time on the Beagle. The HFDL code is likely to have a similar issue (if I had to guess).
Nils (DK8OK) paper is an interesting read: http://www.udxf.nl/HFDL-some-ideas.pdf
Just curious..
The HFDL transmissions that I see are mostly BPSK with 300 or 600 bits/s data rates and on the busier ground station frequencies the messages are exchanged in rapid succession. Occasionally on good connections a switch is made to 1200 or 1800 QPSK. Kind of makes me wonder whether the Beagle could process this fast enough, in addition to other running tasks, to keep up with transmissions even from a single HF channel.
Ben
The counter argument is that something as complex as DRM actually works with its fancy COFDM modulation and an MPEG xHE-AAC audio codec thrown in as well.
Like DRM (and ALE) it's just one of those things that you have to try -- and see if you can get it to work. There is the possibility you get to the end and realize it's just not possible. But you can't tell for sure without putting in the work.
"But you can't tell for sure without putting in the work" True enough John, I can't argue with that. Sometimes taking up a challenge by itself may have greater value or significance than the outcome..
Best regards, Ben
Does anyone have a Kiwi IQ file recording that is known to decode with any of the HFDL decoders?
After some trouble I was able to get dumphfdl to build on Mac and Kiwi Debian. And it will decode an HFDL signal from a wide-band IF IQ file (from an SDRplay demo recording).
But I can't get it to work with a zero-IF (audio) IQ file recorded on a Kiwi. Looking at the dumphfdl code I don't see why it doesn't work. But it would help to have a file that is known to work with other decoders. Thanks.
Just recorded an IQ file from Kiwi at 13270 KHz and play back looks to be decoded normally by Multipsk. The forum engine does not accept my zip or wav file , so sending it to you via email jks@
bregards, Ben
Edit: Also tried to decode this with dumphfdl as well. No detail info on the cs16 file format that is used.
Thanks. I couldn't get this file to work with dumphfdl either. So there is some kind of issue with zero-IF files. I tried the usual suspects (big/little endian reversal, I/Q swap) with no change.
When you did this the Kiwi dial frequency was 6721 with the HFDL signal appearing 1440 Hz "above" the IQ carrier point (vertical line of Kiwi IQ passband drawn in frequency scale), right? (as seems to be shown in the Multipsk waterfall).
If so, then it isn't really true that this is a zero-IF situation. The channelizer code in dumphfdl will be shifting the channel by that small amount (1440 Hz) to baseband. The same way it does for much larger shifts involved when a file containing a wide IF passband is used.
John, reply sent by email due to attachments.
Okay, I found the dumphfdl bug. Now to fix properly and move forward..
FWIW, Tomasz has released v1.1.0 of dumphfdl which now builds/runs on MacOS (I sent him some trivial fixes).
https://www.rtl-sdr.com/dumphfdl-a-multichannel-hfdl-decoder-for-sdr/comment-page-1/#comment-222998
Thank you so much for your effort!!!
73 Josef
DE3JGA
As of today things have moved from "don't know if it will work" to "it might work".
Now able to decode a test signal with the HFDL code running as an extension on a BBG. There is still lots of work to do before it is in a usable form. It was a real struggle to get this far. Performance questions are still open although initial testing looks promising. I haven't even thought about UI issues yet.
Good to see you got the HFDL decodes running. A busy single frequency like the one you used 13270 KHz for the Thai GS can already yield a lot of aircraft transmissions and position reports over a wide geographical area.
For other freqs are you considering to use a dropdown list based on table 51 or scanning similar to ALE2G extension (and perhaps limited to the active freqs collected from the squitter messages) ?
KG-HFDLhas a feature called "automatic frequency control", where the GS active frequencies can be followed with several options. Seemingly quite handy, but depending on propagation it is possible to end up on a "dead" channel when using this.
The UI considerations are somewhat complicated by the following dumphfdl feature. Given sufficiently powerful client hardware, dumphfdl is capable of decoding multiple channels in parallel. An example: You can feed dumphfdl a SoapySDR IQ connection that has a 30 MHz bandwidth centered on 15 MHz (i.e. the entire SW spectrum). Then tell it to decode two HFDL channels, say 13270 and 8886 kHz. It will do the digital downconversion (DDC) of those two 2300 Hz wide channels to baseband and process them in parallel. The number of channels is limited only by the power of the client hardware.
Now in certain circumstances this might be applicable to the Kiwi as well. You can obviously run the HFDL extension on multiple Kiwi connections. But if the HFDL channels are spaced close enough together to fit in the audio passband (12 or 20 kHz) of a single Kiwi channel dumphfdl should be able to handle it. For example 8921 and 8927. The user interface needs some convenient way of handling this situation.
Of course all of this is dependent on how much HFDL processing the Kiwi can handle.
Yes indeed, I tried out the multichannel capability of dumphfdl and likewise gr-ale decoder for a given set of frequencies within a wide receiver bandwidth. This goes pretty well even with a simple RTL-SDR at 2 MHz bandwidth.
As you mentioned with the Kiwi we are limited to a 20 KHz bandwidth. It is worth a try first to see if that would work.
Having active frequencies that close together and with propagation open to a particular QTH, all in the same time period does not happen too often as far as I have observed, hence my question about channel selection.
73 Ben
Don't camp this Kiwi for long periods of time, but you can try an extremely rough version of the HFDL extension here: http://jimlill.com:8075/?f=z7&dx=none,hfdl&ext=hfdl,11387
Lots of the packets are not decoded for unknown reasons. They seem strong enough and without selective fading. So I think there is an issue someplace.
@benson I build the "Stations" and "Bands" menus/dropdowns by parsing the system table (v51) using some code recycled from ALE. I haven't looked at the ACARS messages in detail yet (such as the ACARS frequency tables you mention).
At the end of Nils' (DK8OK) paper he shows freq vs ToD schedules for the ground stations that were built from ACARS decoded data: http://www.udxf.nl/HFDL-some-ideas.pdf These are reminiscent of the DRM extension ToD schedules. It might be possible to generate something similar based on, for example, any public Kiwi running the HFDL extension reporting back to kiwisdr.com ACARS data for aggregation and schedule creation. All Kiwis would then download this schedule on HFDL extension startup similar to what DRM does now.
@jks That is a good idea to have the latest table downloaded from kiwisdr.com. For the uploads, the latest and complete table from any KiwiSDR location will be fine. It looks that no major changes in frequency assignment happen within a 3 hour period.
Having the current frequencies in use (a subset from the stations and bands dropdowns) is of course no guarantee that propagation will be favourable to a particular location.
From a DRM like schedules page, the station/frequency combination could be selected easily, with the added benefit of visually getting some insight into the worldwide propagation conditions from the ground station choice of frequencies.
working with the hfdl in the kiwi webpage ui nspired me to look at using kiwiclient tools along with dumphfdl to get similar results without using a browser or loading the BB with demodulation etc. I asked Tomasz about adding SNR reporting and he immediately put it into the dev branch. Maybe that something that could added to the extension also.
There were some recent fixes to kiwi_nc (Kiwi netcat) in the kiwiclient repo, and the dumphfdl repo, that allow direct piping of Kiwi IQ samples into dumphfdl. I have not tried this myself, but Tomasz says it works. Use dumphfdl options
--iq-file - --sample_rate 12000 --sample_format CS16(I think, I haven't actually tried this myself).thanks John... very helpful
does tlimit not work in kiwi_nc ??
I did get kiwi_nc recorded IQ file to decode using dumphfdl !! Cleanup work to follow
does tlimit not work in kiwi_nc ??
A quick look shows the code is there to do it. But it doesn't work for some reason.
Also note: Tomasz has added a detailed section to the dumphfdl readme describing the kiwi_nc pipeline procedure: https://github.com/szpajder/dumphfdl
Okay, pull kiwiclient repo to get kiwi_nc tlimit fix.
kiwiclient Makefile now has working dumphfdl example. At the moment requires you install/use the "devel" branch of dumphfdl. This will not be required after the next dumphfdl release.
v1.475 November 17, 2021
HFDL extension improvements:
Now works for Kiwis using 3-channel/20.25 kHz mode.
when I try the kiwi_nc pipe to dumphfdl I get a broken pipe error. If I use kiwi_nc to create a file then use dumphfdl on the file, it works great.
Make sure you have the latest kiwiclient and dumphfdl ("devel" branch) repos. I just did a "make dumphfdl" (using python3) and it works fine.
You may be using an argument (or typo) to kiwi_nc that is being hidden because of the use of the pipe (this really shouldn't happen as all errors should be written to stderr which should bypass the stdout pipe). To check use the
--progressoption to kiwi_nc (which will disable all stdout streaming output) running it without the pipe and see if you get any errors. See "make nc" in the Makefile for an example.