Second build sold out. Message will appear here when store is ready for third build ordering.

Suggestion - FT8 decoder?

Hi,

This suggestion/request stems off of work that's already been done by Pavel for the Redpitaya (FPGA+dual core ARM) to create a multi-band FT8 decoder. I (Andy/K1RA) posed the question/suggestion and started an on going discussion here

https://github.com/pavel-demin/red-pitaya-notes/issues/689

A more detailed description of Pavel's running end product is here

http://pavel-demin.github.io/red-pitaya-notes/sdr-transceiver-ft8/

In short his FPGA code creates .c2 complex data in a memory map for each band. The ARM processor reads the memory map and create files, which in turn are handed off to the ft8d command line decoder that produces decoded packets written to an text log file. From that log file there are a few options to upload to PSKreporter.info , ReverseBeaconNetwork, send LAN WSJT-X UDP broadcast packets, as well as a local telnet DXcluster capability.

Since Pavel has already modified the WSJT-X FT8 command line decoder to compile standalone, located here

https://github.com/pavel-demin/ft8d

I was wondering what it would take to get this to work in conjunction with one or more of the KiwiSDR channels. The wish is to run self contained on the Beagle/KiwiSDR and not through the browser.

Doable on the KiwiSDR? Thoughts, comments, suggestions?

Thanks & 73

andyz - K1RA

https://k1ra.us/
http://kiwisdr.k1ra.us:8073/

Comments

  • edited November 2018
    you can use kiwirecorder.py to capture an audio file and then process it with a standalone decoder. No browser needed.
  • OK - I'll give that a try first.

    Thanks!

    andyz
  • I just ran last night, streaming audio from my remote Kiwi into WSJT-X and decoding FT8 on 40m. I had no idea there was so much activity of FT8, over 2000 different transmitters in 60 countries! 30-40 spots every 15 seconds, can the BBG even keep up with it?
    See the image from PSKReporter below.

  • Glenn, you'd have to run kiwirecorder on another computer like Rob does with kiwiwspr
  • I think this is correct, that's how I generated these spots, by running on a moderately capable desktop. Judging from the relative performance of the KiwiSDR WSPR extension and other systems, I think asking an FT8 extension on the KiwiSDR to keep up with a band like 40m is lately is probably not realistic. I posted this in response the "Doable on the Kiwi?" question at the top of this thread. I suspect it really isn't....
    Glenn n6gn
    WA2ZKD
  • Here's a follow up report on my desire to run the WSJT-X FT8 decoder and propagation reporter all standalone on my KiwiSDR.

    For my earlier work with Pavel on the Redpitaya, where I compiled WSJT-X for that ARM processor, I found I was able to take the binary executable apps from that platform and run them directly on the KiwiSDR. That saved me a ton of time as I didn't have to go through the time intensive process of recompiling all of WSJT-X, which took >1 hour on the Redpitaya. I need to investigate the use of cross compiling tools for ARM on my Intel i7, since I'm currently using an older 1.8.0 release of WSJT-X on the Kiwi since I compiled it some time ago. I would like to update to 1.9.0 and eventually 2.0.0. Nevertheless I used the 'jt9' WSJT-X command line app with the '--ft8' command line option, along with kiwirecorder.py and initially proved I could capture a 15 second WAV file from one of the FT8 calling frequencies and decode the WAV file packets right on the KiwiSDR. I ran a UNIX 'time' test on the decoder and it takes ~4 seconds to decode the 15 second WAV window on the BeagleBone ARM. By comparison it takes ~1 second on the Redpitaya ARM processor.

    After several trials of trying to run 2 kiwirecorders & 2 FT8 decoder apps on different bands, I found the ARM processor bogged down too much to keep up. Maybe I could kill GPS or something else to help free up some CPU cycles? Anyhow, to accomplish my goal of monitoring and reporting across all 10 FT8 bands (160-10m) I hacked together a shell script and a crontab and used my PSKreporter uploader written in PERL for the Redpitaya and have been successfully uploading reports to the map server for the last hour or two under my club call KW4VA, since my Redpitaya is uploading under K1RA. Here are my spots on the map

    https://pskreporter.info/pskmap.html?preset&callsign=kw4va&txrx=rx&mode=FT8&timerange=86400&hideunrec=1




    Since FT8 decoding is very time sensitive I needed to create an app to wait an exact number of seconds to hit the 0, 15, 30 & 45 second start time windows, since crontab can only be scheduled to a single minute resolution. I found I had to kick off the kiwirecorder at 12, 27, 42 & 57 seconds after the minute to hit the timing correctly. I decided to use crontab to rotate through monitoring each of the 10 bands. I monitor a band for one full minute every 10 minutes, which is four 15 second sequences. That would allow me to see odd and even or 0/2, 1/3 pairs to catch most user's transmissions. I will eventually change this as 160-60m has little to no activity during the day and 15m and above none at nite. The shell script kicked off by crontab is here

    https://www.dropbox.com/s/8in6gfgykvyomp8/ft8.sh?dl=0

    and you can see my grossly verbose crontab used to accomplish the band rotation and sequence timing here

    https://www.dropbox.com/s/cd838fpd4aql0fn/crontab.txt?dl=0

    So, the basic process is run kiwirecorder in USB mode for 14 seconds with a 0-3000 Hz bandwidth and save to a unique kHz WAV filename. That filename is passed to 'jt9 --ft8' and the output is piped through a few 'awk' scripts to take the original decoder output that looks like this

    000000 -10 -0.6 273 ~ KA2F KE0AOO RRR

    and since GMT time is always 0000000 I replace with local HHMMSS and there is no band info so I replace ~ with kHz which get me a log line that looks

    190330 -10 -0.6 273 7074 KA2F KE0AOO RRR

    I save and append all that decoder output text to /dev/shm/decode-ft8.log and along with a line break I insert makes it look like this

    Decoding
    190415 11 -1.2 756 7074 ND8G N4TKW -24
    190415 -14 -0.6 1215 7074 K4VBM WA3YZD RR73
    190415 8 -0.5 1650 7074 CQ VA3HP EN92
    190415 -8 -0.4 1828 7074 WD5ACR K9AZH EN62
    190415 10 -0.5 2436 7074 CQ VE3MZD EN82
    190415 15 -0.6 2825 7074 CQ K4WW EM78
    Decoding
    190430 -15 -0.4 1498 10136 CQ SV1LHZ KM18
    190430 -15 -0.7 684 10136 VE3LTG/W4 73
    190430 2 -1.9 853 10136 CQ VO1SA GN37
    190430 -2 -0.5 954 10136 KJ4EG AA5AU R-01
    190430 1 -0.6 1174 10136 N4TB W1TMG -02

    I modified my PERL PSKReporter uploader for the Redpitaya to handle the KiwiSDR format and it now watches the KiwiSDR log file and uploads the latest decodes to PSKreporter. The uploader can be seen here

    https://www.dropbox.com/s/sdqpwm507f5wv0i/pskr.pl?dl=0

    The 'sleepx' executable app I use for timing can also be found here

    https://www.dropbox.com/sh/stoi61cxpoyjzuh/AAAzNIGeUKMU-ts2TFmpAMlFa?dl=0

    Also there is a usr-local-lib.tgz file there that needs to be untarred into /usr/local/lib to support the execution of the 'jt9' app.

    So if anyone else wants to play with this feel free. If you have any comments, questions, suggestions, updates, etc. please let me know. I'll most likely clean this up and make it more efficient over time, but the proof of concept seems to work.

    73

    andyz - K1RA
  • Hi Andyz,

    You might want to try the squelch options of kiwirecorder.py. I do not know down to which S/N FT8 Mode can go. But you can see what happens if you use -T 10 (10dB over running average) --squelch-tail 0.2.

    And if the CPU power of the beaglebone is not sufficient you can do the processing and recording on another computer, e.g. a RPI, like some people do for WSPR.

    Christoph
    WA2ZKDKA7U
  • Christoph,

    Thanks for the suggestions. FT8 can detect signals down to an SNR of -24.

    I do understand I can run the recorder off box, but my goal is to run everything on the KiwiSDR/BeagleBone self-contained like I do with the Redpitaya. Maybe converting kiwirecorder.py to C is an option to speed things up too.

    All,

    Please consult my _README-KiwiSDR-FT8.txt file for step by step instructions for running the FT8 scheduler, decoder and PSKreporter uploader directly from a KiwiSDR/BeagleBone. See my Dropbox here:

    https://www.dropbox.com/sh/stoi61cxpoyjzuh/AAAzNIGeUKMU-ts2TFmpAMlFa?dl=0

    I've cleaned up and improved my crontab and ft8.sh shell script for scheduled band decoding, as well as added a scheduling example to account for N.A. east coast band conditions. All that is in my Dropbox above. There are comments in the .sh, .pl and cron files to help those understand what's happening. Finally I put a copy of the WSJT-X (jt9) command line decoder used by the ft8.sh script in the Dropbox so everything is available for others to try on their KiwiSDR's.

    Please let me know if you attempt this and if you have any successes or failures.

    Thanks & 73

    andyz - K1RA/KW4VA

    KW4VA KiwiSDR FT8 24 hr spots here:

    https://pskreporter.info/pskmap.html?preset&callsign=kw4va&txrx=rx&mode=FT8&timerange=86400&hideunrec=1
  • Hey Andyz, any thoughts about the 2.0 upgrade and this one? I suspect the decoder would need some changes?

    73!
    - Kevin, N3LGA
  • I think you'll run out of mips on the BB
  • Kevin,
    Yes I've compiled and been running the new FT8 v2.0 decoder on the KiwiSDR now for over a week using that and my on board PSKreporter upload script. With that I have been placing in the Top 20 world spotters on the stats page here

    https://pskreporter.info/cgi-bin/pskstats.pl

    Look for my club call KW4VA. That is running one instance of kiwirecorder and FT8 decoder on the KiwiSDR in a band hop arrangement, where I watch a different band every minute. I'm also posting under my K1RA call, but those spots are from the new decoder running on my Redpitaya which supports watching, decoding and posting 8 bands simultaneously.

    If you wish, you can grab the latest jt9 binary/executable for the KiwiSDR below. You can just copy that over the old one on your KiwiSDR

    https://www.dropbox.com/sh/stoi61cxpoyjzuh/AAAzNIGeUKMU-ts2TFmpAMlFa?dl=0

    I've also made an update to the README. Thanks to Yuri RZ3DVP, he pointed out I failed to add instructions for acquiring and installing kiwiclient software onto the KiwiSDR which is required for this all to work.

    Please let me know if anyone else has success or issues with these instructions.

    73 & Merry Christmas / Happy Holidays

    andyz - K1RA
    https://k1ra.us/
  • Only 1 instance of FT8 is running at a time, correct?
  • edited December 2018
    Correct - I use kiwirecorder to pull in a four sequential, 15 second WAV files, which equates to four FT8 transmission windows per minute. This is on a single band and produces up to 25+ FT8 call signs decodes per quarter or >100 per minute. I then switch bands and repeat.

    I use cron to schedule which bands I concentrate on, mainly 160-17m at nite and 40-10m during the day. I can usually revisit a band once every 5 minutes and that has produced daily copies over the last few weeks that look like this

    https://pskreporter.info/pskmap.html?preset&callsign=kw4va&txrx=rx&mode=FT8&timerange=86400&hideunrec=1



    I've recently started to experiment with 630m, but down there most people are running the WSJT-X JT9 mode. Fortunately I can use the same technique as above, though instead I use kiwirecorder to capture a full minute of WAV audio, or 1 JT9 transmission sequence and then handing that off to the WSJT-X "jt9" command line decoder I compiled earlier, just with a different command line option (-9 instead of -8) to decode that mode. I had to tweak my PSKreporter upload script slightly to handle this, but the result is good and can be seen

    https://pskreporter.info/pskmap.html?preset&callsign=k1ra&txrx=rx&mode=JT9&band=200000-600000&timerange=10800&hideunrec=1



    Its nice to have everything self contained on the KiwiSDR/BeagleBone, but I will experiment with running all this on a multi-core RPi to see how many simultaneous bands I can run using the KiwiSDR that way.

    73

    andyz - K1RA
    benson
  • if you haven't bought the RPi yet, consider an Odroid XU4 for even more horsepower
  • Hi All,

    It strikes me that using the same methodology for FT that has already been applied to WSPR monitoring would be the most sensible solution i.e. "no point in reinventing the wheel"

    The commonality would then allow any changes or enhancements to apply to both (or any other similar mode that may come along), and anyone else wishing to setup a similar service would have a consistent starting point.

    Incidentally - one thing I have wondered with both the WSPR and FT8 monitoring and reporting.

    I often use WSPR to test antennas and rely upon comparing spots of transmissions from reporting stations in the same time slot.

    Do the time stamps get corrected to allow for any processing delay, so that the reports correctly match the actual time of transmission ?

    Regards,

    Martin - G8JNJ
  • Martin, in Rob's wspr scheme the processing delays are no different than in wsjt-x as it uses wsprd. The use of a 2nd device, RPi or whatever assures that there is adequate computing power to assure timeliness of everything. The inability of a kiwi alone to process multiple channels of wspr was the reason Rob developed kiwiwspr.sh which has now evolved into wsprdaemon.sh which is currently in beta. KD2OM is testing the latter and typically runs #1 or 2 for unique wspr spots by a North American station. KD2OM and others use multiple kiwi to feed one decoding and processing device, in his case an RPi3 and can exceed 1000 spots/hour
  • Thanks Andyz for help to start this script on Ubuntu 18 LTS 64 bit (x86) virtual server. My antenna for this KiwiSDR - FLAG and statistics on https://pskreporter.info/pskmap?callsign=rz3dvp
    Happy New Year 2019!

    Yuri
  • edited January 2019
    Hello everyone! I installed Andyz script on 64bit Ubuntu VM and add simply service for auto start it after reboot.

    sudo nano /etc/systemd/system/pskreporter.service

    [Unit]
    Description=PSK Reporter
    Wants=network-online.target
    After=network-online.target
    Wants=ntp.service
    After=ntp.service
    [Service]
    ExecStartPre=/bin/sleep 10
    ExecStart=/home/{user_name}/ft8/pskr.pl CALLSIGN LOCATOR
    WorkingDirectory=/home/{user_name}/ft8
    StandardOutput=inherit
    StandardError=inherit
    Restart=always
    [Install]
    WantedBy=multi-user.target

    where {user_name} - your user name on Linux, CALLSIGN - your callsign, LOCATOR - your QTH locator.

    Then enable it:
    systemctl enable pskreporter.service

    For more stable start pskr.pl I added checking for exist decode-ft8.log file, because after reboot VM pskr.pl can start before it's create:
    ...
    # datagram size in bytes
    my $ds;

    $| = 1;

    # wait the LOG file
    while( not(-f '/dev/shm/decode-ft8.log')) {
    sleep 3;
    }


    # setup tail to watch FT8 decoder log file and pipe for reading
    ...


    I hope my info was helpful for something else who want to create external FT8 reporter with KiwiSDR. :)
  • Tried to install the FT8 v2.0 decoder on a RP3 and the ft.sh scripts runs well with decodes shown. Originally Andy installed this in root on a Kiwi SDR and so I tried that as well. Trouble is I can not get the crontab running which is a crucial part of the operation. Tried user and root crontab versions without success.
    Perhaps someone with an installation on another Linux platform ran into similar difficulty has some pointers on how to to proceed.

    I do like the fact that it takes up only a single Kiwi channel and using FT8 gives a really quick glance at the band openings something that using WSPR does take a lot more time. Never mind that some folks seem to use Megawatts to run FT8....

    Regards, Ben
  • It turned out that following the steps posted in dropbox is all that is needed, however when creating a crontab as root I had to remove root as user on all the lines to get the cron job running on my RP3.
    Viewing the results on PSK reporter is quite interesting. Thanks Andyz.

    Ben
  • edited April 2020
    Know its an old thread, but completely worked on a RPi 3 today with my n3ka kiwisdr.

    https://pskreporter.info/pskmap?callsign=ai2cw&search=Find

    Thanks for such great instructions!
    73
    -kevin n3ka (ex n3lga)
Sign In or Register to comment.