Suggestion - FT8 decoder?
Hi,
This suggestion/request stems off of work that's already been done by Pavel for the Redpitaya (FPGA+dual core ARM) to create a multi-band FT8 decoder. I (Andy/K1RA) posed the question/suggestion and started an on going discussion here
https://github.com/pavel-demin/red-pitaya-notes/issues/689
A more detailed description of Pavel's running end product is here
http://pavel-demin.github.io/red-pitaya-notes/sdr-transceiver-ft8/
In short his FPGA code creates .c2 complex data in a memory map for each band. The ARM processor reads the memory map and create files, which in turn are handed off to the ft8d command line decoder that produces decoded packets written to an text log file. From that log file there are a few options to upload to PSKreporter.info , ReverseBeaconNetwork, send LAN WSJT-X UDP broadcast packets, as well as a local telnet DXcluster capability.
Since Pavel has already modified the WSJT-X FT8 command line decoder to compile standalone, located here
https://github.com/pavel-demin/ft8d
I was wondering what it would take to get this to work in conjunction with one or more of the KiwiSDR channels. The wish is to run self contained on the Beagle/KiwiSDR and not through the browser.
Doable on the KiwiSDR? Thoughts, comments, suggestions?
Thanks & 73
andyz - K1RA
https://k1ra.us/
http://kiwisdr.k1ra.us:8073/
This suggestion/request stems off of work that's already been done by Pavel for the Redpitaya (FPGA+dual core ARM) to create a multi-band FT8 decoder. I (Andy/K1RA) posed the question/suggestion and started an on going discussion here
https://github.com/pavel-demin/red-pitaya-notes/issues/689
A more detailed description of Pavel's running end product is here
http://pavel-demin.github.io/red-pitaya-notes/sdr-transceiver-ft8/
In short his FPGA code creates .c2 complex data in a memory map for each band. The ARM processor reads the memory map and create files, which in turn are handed off to the ft8d command line decoder that produces decoded packets written to an text log file. From that log file there are a few options to upload to PSKreporter.info , ReverseBeaconNetwork, send LAN WSJT-X UDP broadcast packets, as well as a local telnet DXcluster capability.
Since Pavel has already modified the WSJT-X FT8 command line decoder to compile standalone, located here
https://github.com/pavel-demin/ft8d
I was wondering what it would take to get this to work in conjunction with one or more of the KiwiSDR channels. The wish is to run self contained on the Beagle/KiwiSDR and not through the browser.
Doable on the KiwiSDR? Thoughts, comments, suggestions?
Thanks & 73
andyz - K1RA
https://k1ra.us/
http://kiwisdr.k1ra.us:8073/
Comments
Thanks!
andyz
See the image from PSKReporter below.
Glenn n6gn
For my earlier work with Pavel on the Redpitaya, where I compiled WSJT-X for that ARM processor, I found I was able to take the binary executable apps from that platform and run them directly on the KiwiSDR. That saved me a ton of time as I didn't have to go through the time intensive process of recompiling all of WSJT-X, which took >1 hour on the Redpitaya. I need to investigate the use of cross compiling tools for ARM on my Intel i7, since I'm currently using an older 1.8.0 release of WSJT-X on the Kiwi since I compiled it some time ago. I would like to update to 1.9.0 and eventually 2.0.0. Nevertheless I used the 'jt9' WSJT-X command line app with the '--ft8' command line option, along with kiwirecorder.py and initially proved I could capture a 15 second WAV file from one of the FT8 calling frequencies and decode the WAV file packets right on the KiwiSDR. I ran a UNIX 'time' test on the decoder and it takes ~4 seconds to decode the 15 second WAV window on the BeagleBone ARM. By comparison it takes ~1 second on the Redpitaya ARM processor.
After several trials of trying to run 2 kiwirecorders & 2 FT8 decoder apps on different bands, I found the ARM processor bogged down too much to keep up. Maybe I could kill GPS or something else to help free up some CPU cycles? Anyhow, to accomplish my goal of monitoring and reporting across all 10 FT8 bands (160-10m) I hacked together a shell script and a crontab and used my PSKreporter uploader written in PERL for the Redpitaya and have been successfully uploading reports to the map server for the last hour or two under my club call KW4VA, since my Redpitaya is uploading under K1RA. Here are my spots on the map
https://pskreporter.info/pskmap.html?preset&callsign=kw4va&txrx=rx&mode=FT8&timerange=86400&hideunrec=1
Since FT8 decoding is very time sensitive I needed to create an app to wait an exact number of seconds to hit the 0, 15, 30 & 45 second start time windows, since crontab can only be scheduled to a single minute resolution. I found I had to kick off the kiwirecorder at 12, 27, 42 & 57 seconds after the minute to hit the timing correctly. I decided to use crontab to rotate through monitoring each of the 10 bands. I monitor a band for one full minute every 10 minutes, which is four 15 second sequences. That would allow me to see odd and even or 0/2, 1/3 pairs to catch most user's transmissions. I will eventually change this as 160-60m has little to no activity during the day and 15m and above none at nite. The shell script kicked off by crontab is here
https://www.dropbox.com/s/8in6gfgykvyomp8/ft8.sh?dl=0
and you can see my grossly verbose crontab used to accomplish the band rotation and sequence timing here
https://www.dropbox.com/s/cd838fpd4aql0fn/crontab.txt?dl=0
So, the basic process is run kiwirecorder in USB mode for 14 seconds with a 0-3000 Hz bandwidth and save to a unique kHz WAV filename. That filename is passed to 'jt9 --ft8' and the output is piped through a few 'awk' scripts to take the original decoder output that looks like this
000000 -10 -0.6 273 ~ KA2F KE0AOO RRR
and since GMT time is always 0000000 I replace with local HHMMSS and there is no band info so I replace ~ with kHz which get me a log line that looks
190330 -10 -0.6 273 7074 KA2F KE0AOO RRR
I save and append all that decoder output text to /dev/shm/decode-ft8.log and along with a line break I insert makes it look like this
Decoding
190415 11 -1.2 756 7074 ND8G N4TKW -24
190415 -14 -0.6 1215 7074 K4VBM WA3YZD RR73
190415 8 -0.5 1650 7074 CQ VA3HP EN92
190415 -8 -0.4 1828 7074 WD5ACR K9AZH EN62
190415 10 -0.5 2436 7074 CQ VE3MZD EN82
190415 15 -0.6 2825 7074 CQ K4WW EM78
Decoding
190430 -15 -0.4 1498 10136 CQ SV1LHZ KM18
190430 -15 -0.7 684 10136 VE3LTG/W4 73
190430 2 -1.9 853 10136 CQ VO1SA GN37
190430 -2 -0.5 954 10136 KJ4EG AA5AU R-01
190430 1 -0.6 1174 10136 N4TB W1TMG -02
I modified my PERL PSKReporter uploader for the Redpitaya to handle the KiwiSDR format and it now watches the KiwiSDR log file and uploads the latest decodes to PSKreporter. The uploader can be seen here
https://www.dropbox.com/s/sdqpwm507f5wv0i/pskr.pl?dl=0
The 'sleepx' executable app I use for timing can also be found here
https://www.dropbox.com/sh/stoi61cxpoyjzuh/AAAzNIGeUKMU-ts2TFmpAMlFa?dl=0
Also there is a usr-local-lib.tgz file there that needs to be untarred into /usr/local/lib to support the execution of the 'jt9' app.
So if anyone else wants to play with this feel free. If you have any comments, questions, suggestions, updates, etc. please let me know. I'll most likely clean this up and make it more efficient over time, but the proof of concept seems to work.
73
andyz - K1RA
You might want to try the squelch options of kiwirecorder.py. I do not know down to which S/N FT8 Mode can go. But you can see what happens if you use -T 10 (10dB over running average) --squelch-tail 0.2.
And if the CPU power of the beaglebone is not sufficient you can do the processing and recording on another computer, e.g. a RPI, like some people do for WSPR.
Christoph
Thanks for the suggestions. FT8 can detect signals down to an SNR of -24.
I do understand I can run the recorder off box, but my goal is to run everything on the KiwiSDR/BeagleBone self-contained like I do with the Redpitaya. Maybe converting kiwirecorder.py to C is an option to speed things up too.
All,
Please consult my _README-KiwiSDR-FT8.txt file for step by step instructions for running the FT8 scheduler, decoder and PSKreporter uploader directly from a KiwiSDR/BeagleBone. See my Dropbox here:
https://www.dropbox.com/sh/stoi61cxpoyjzuh/AAAzNIGeUKMU-ts2TFmpAMlFa?dl=0
I've cleaned up and improved my crontab and ft8.sh shell script for scheduled band decoding, as well as added a scheduling example to account for N.A. east coast band conditions. All that is in my Dropbox above. There are comments in the .sh, .pl and cron files to help those understand what's happening. Finally I put a copy of the WSJT-X (jt9) command line decoder used by the ft8.sh script in the Dropbox so everything is available for others to try on their KiwiSDR's.
Please let me know if you attempt this and if you have any successes or failures.
Thanks & 73
andyz - K1RA/KW4VA
KW4VA KiwiSDR FT8 24 hr spots here:
https://pskreporter.info/pskmap.html?preset&callsign=kw4va&txrx=rx&mode=FT8&timerange=86400&hideunrec=1
73!
- Kevin, N3LGA
Yes I've compiled and been running the new FT8 v2.0 decoder on the KiwiSDR now for over a week using that and my on board PSKreporter upload script. With that I have been placing in the Top 20 world spotters on the stats page here
https://pskreporter.info/cgi-bin/pskstats.pl
Look for my club call KW4VA. That is running one instance of kiwirecorder and FT8 decoder on the KiwiSDR in a band hop arrangement, where I watch a different band every minute. I'm also posting under my K1RA call, but those spots are from the new decoder running on my Redpitaya which supports watching, decoding and posting 8 bands simultaneously.
If you wish, you can grab the latest jt9 binary/executable for the KiwiSDR below. You can just copy that over the old one on your KiwiSDR
https://www.dropbox.com/sh/stoi61cxpoyjzuh/AAAzNIGeUKMU-ts2TFmpAMlFa?dl=0
I've also made an update to the README. Thanks to Yuri RZ3DVP, he pointed out I failed to add instructions for acquiring and installing kiwiclient software onto the KiwiSDR which is required for this all to work.
Please let me know if anyone else has success or issues with these instructions.
73 & Merry Christmas / Happy Holidays
andyz - K1RA
https://k1ra.us/
I use cron to schedule which bands I concentrate on, mainly 160-17m at nite and 40-10m during the day. I can usually revisit a band once every 5 minutes and that has produced daily copies over the last few weeks that look like this
https://pskreporter.info/pskmap.html?preset&callsign=kw4va&txrx=rx&mode=FT8&timerange=86400&hideunrec=1
I've recently started to experiment with 630m, but down there most people are running the WSJT-X JT9 mode. Fortunately I can use the same technique as above, though instead I use kiwirecorder to capture a full minute of WAV audio, or 1 JT9 transmission sequence and then handing that off to the WSJT-X "jt9" command line decoder I compiled earlier, just with a different command line option (-9 instead of -8) to decode that mode. I had to tweak my PSKreporter upload script slightly to handle this, but the result is good and can be seen
https://pskreporter.info/pskmap.html?preset&callsign=k1ra&txrx=rx&mode=JT9&band=200000-600000&timerange=10800&hideunrec=1
Its nice to have everything self contained on the KiwiSDR/BeagleBone, but I will experiment with running all this on a multi-core RPi to see how many simultaneous bands I can run using the KiwiSDR that way.
73
andyz - K1RA
It strikes me that using the same methodology for FT that has already been applied to WSPR monitoring would be the most sensible solution i.e. "no point in reinventing the wheel"
The commonality would then allow any changes or enhancements to apply to both (or any other similar mode that may come along), and anyone else wishing to setup a similar service would have a consistent starting point.
Incidentally - one thing I have wondered with both the WSPR and FT8 monitoring and reporting.
I often use WSPR to test antennas and rely upon comparing spots of transmissions from reporting stations in the same time slot.
Do the time stamps get corrected to allow for any processing delay, so that the reports correctly match the actual time of transmission ?
Regards,
Martin - G8JNJ
Happy New Year 2019!
Yuri
sudo nano /etc/systemd/system/pskreporter.service
[Unit]
Description=PSK Reporter
Wants=network-online.target
After=network-online.target
Wants=ntp.service
After=ntp.service
[Service]
ExecStartPre=/bin/sleep 10
ExecStart=/home/{user_name}/ft8/pskr.pl CALLSIGN LOCATOR
WorkingDirectory=/home/{user_name}/ft8
StandardOutput=inherit
StandardError=inherit
Restart=always
[Install]
WantedBy=multi-user.target
where {user_name} - your user name on Linux, CALLSIGN - your callsign, LOCATOR - your QTH locator.
Then enable it:
systemctl enable pskreporter.service
For more stable start pskr.pl I added checking for exist decode-ft8.log file, because after reboot VM pskr.pl can start before it's create:
...
# datagram size in bytes
my $ds;
$| = 1;
# wait the LOG file
while( not(-f '/dev/shm/decode-ft8.log')) {
sleep 3;
}
# setup tail to watch FT8 decoder log file and pipe for reading
...
I hope my info was helpful for something else who want to create external FT8 reporter with KiwiSDR.
Perhaps someone with an installation on another Linux platform ran into similar difficulty has some pointers on how to to proceed.
I do like the fact that it takes up only a single Kiwi channel and using FT8 gives a really quick glance at the band openings something that using WSPR does take a lot more time. Never mind that some folks seem to use Megawatts to run FT8....
Regards, Ben
Viewing the results on PSK reporter is quite interesting. Thanks Andyz.
Ben
https://pskreporter.info/pskmap?callsign=ai2cw&search=Find
Thanks for such great instructions!
73
-kevin n3ka (ex n3lga)