8 channel Kiwi WSPR decoding script 'kiwiwspr.sh' using Raspberry Pi or other external server

rrobinetrrobinet
edited September 2018 in WSPR, wsprdaemon, kiwirecorder
The Kiwi's WSPR extension is a marvel and provides a much better view of the WSPR band operation than does WSJT-x.
However the Kiwi is not quite as good as WSJT-x at decoding spots on crowded and/or noisy bands with many low level signals.
The attached script addresses many of the Kiwi's limitations as a WSPR decoder by moving the WSPR signal processing to an external server.
For example, at KPH (http://kphsdr.com/) this script has been simultaneously and reliably decoding 14 WSPR bands from two Kiwis on one Raspberry Pi 3b (not even a 3b+) for several months.
Once installed on the Pi (or other Linux computer) the script runs automatically in the background after power up or reboot.

The benefits over using the Kiwi's internal autowspr decoder extension are:

1) You can decode 8 WSPR bands at once on one Kiwi. The Kiwi can be configured to autowspr 8 bands but its CPU is severely overtaxed and thus very few spots get decoded in that mode.
2) On busy bands even in 4 channel mode the Kiwi CPU runs out of time, in which case kiwiwspr.sh will decode 1.5-3x the number of spots
3) I believe that the Kiwi's autowspr decoder is derived from an older version of 'wsprd', so even on quiet bands kiwiwspr will extract 20% more spots than autowspr
4) My script decodes using the recent 'wsprd' decoder included in the WSJT-x 1.9.1 package and can easily be upgraded to any newer version of 'wsprd'
5) kiwiwspr.sh supports scheduled band changes, so you can spot on 630 and 160M at night and switch those channels to 15M and 10M during the day
Times can be specified as HH:MM or 'sunrise/sunset +- HH::MM'

I have tried to make installation self documenting after you copy it into a directory on your Pi, make it executable (chmod +x kiwiwspr.sh), and execute './kiwiwspr.sh'.
Of course I will be happy to help with installation and debug and am soliciting more beta testers who will inevitably uncover bugs and feature improvements.

Since installing this script KPH has become one of the top spotters in the world and second only to K9AN (with his more favorable central USA QTH) in North America. My hope is that more Kiwi owners will join KPH at the top of the lists.

Many thanks go to John Seamons who provided the extensions to his kiwirecorder.py script upon which kiwiwspr.sh is built.

Attachments:
https://forum.kiwisdr.com/uploads/Uploader/0f/0cbc66c8809bd99cc2ed706074585e.sh
WA2ZKD

Comments

  • WA2ZKDWA2ZKD
    I have been running the variants on this for months and now have the latest version online. Very pleased! Thanks Rob
  • rrobinetrrobinet
    I have made major fixes and improvements to the scheduling features of the attached Version 1.1 of kiwiwspr.sh which should be compatible with v 0.7 and later config files.
    You should also install the latest version of kiwiclient from https://github.com/jks-prv/kiwiclient/tree/jks-v0.1 which dramatically improves the reliability of communications with the Kiwi
    One Raspberry Pi 3B can easily capture and decode 16 WSPR bands simultaneously from 2 or more Kiwis.

    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/70/48db02fa71fe71ca8a3760b004e5d7.sh
  • rrobinetrrobinet
    Attached is version 1-1a which fixes a schedule error when there is no job specified for time 00:00

    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/29/c043c13a46ad9b6425695370c28aa3.sh
  • VK2YCJVK2YCJ
    Hello Rob,

    I have a dedicated KiwiSDR running 6 x WSPR auto-run decoding tasks and I'd like to use your scheme to push that up to 8 sessions with better performance and greater reliability. The bands I would like to cover are 160m to 10m - 9 bands in all - so maybe the scheduled band change feature could help gain the extra band.

    I can order the RPi-3B+ tomorrow and start on the conversion as soon as it arrives.

    What other material do I need ?

    I'd be delighted to be a beta tester.


    Thanks, Jamie VK2YCJ
  • rrobinetrrobinet
    You just need a Raspberry Pi 3B or newer or other Debian server. However the Pi is a nice choice and I have a Pi 3B spotting 19 channels with plenty of idle CPU You should find you spot more signals on your current bands.
  • VK2YCJVK2YCJ
    Hello Rob,

    Thanks for the reply. I’ll order the Rpi and a spare tomorrow. I have 3 x KiwiSDRs that are going together in a box at a very low noise rural location with a shared antenna and front end protection. One will be the dedicated WSPR unit and the other two general coverage receivers. I have a QS1R SDR with Intel NUC running CW Skimmer Server on 6 bands and that is going in the box too.

    The broadband antenna with ground radials will be physically challenging to construct but once done properly it should last a very long time. Your TCI-530 at your station is an amazing gadget to have. I have seen one of those here in real life at a ship to shore HF station and it looked impressive. Do you have any idea on how much the TCI-530 costs ?

    Thanks, Jamie VK2YCJ
  • jksjks
    Do you have any idea on how much the TCI-530 costs?
    B)
  • VK2YCJVK2YCJ
    John, a TCI-530 is a commercial broadband HF antenna made by TCI in California. Pretty serious stuff.

    http://websdr1.utahsdr.org/info/TCI530.pdf
  • WA2ZKDWA2ZKD
    KD2OM is now #2 in the US, second to K9AN. Signficantly, he is doing it with only 6 kiwi channels, a 1M active loop, and Rob's script.
  • rrobinetrrobinet
    edited October 2018
    Our MRHS volunteer group inherited the < 500 Km TCI-530 and the rest of the facility from RCA, and I believe it costs $100Ks. We also have a >1000 Km range TCI-540, but it or the transmission lines to it need repairs since even on distant stations it delivers 6 dB worse SNRs than the 530.
    The KPH site is exceptionally RF quiet with a very large ground plane installed by Dr Beveridge in the 1920s when he selected the site for RCA. Of course we don't know how much of that ground plane remains functional, but the site is a large marshy seaside peninsula.

    In addition to refining kiwiwspr.sh, I am working to remove all of the spurious signals and lower the background noise at KPH and hopefully the the TCI-540 working well.

    I have attached kiwiwspr.sh version 1.1c which includes a couple of scheduler bug fixes and enhanced validation of the kiwiwspr.conf file. It is reliably running 19 WSPR captures on a Pi 3B at KPH.

    You should also download a current copy of Kiwiclient, as Christoph has fixed a number of bugs which lead to recorder crashes.

    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/95/347ea80dc186294c46ddc492d3623c.sh
  • rrobinetrrobinet
    I have attached version 1-1e in which the watchdog daemon purges zombie wav files. I found that on a number of beta test systems the /tmp directory used to store the 2 minute audio sample files was completely filling the partition when the Pi lost contact with the Internet for minutes or hours. So this version fixes that problem and is in all other ways completely backwards compatible.
    To upgrade: For safety save away your current kiwiwspr.sh, then *Replace* your current .../kiwiwspr.sh with the attached file.

    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/9e/d205faa0fc797299ef80b4977f2695.sh
    WA2ZKDKK6PR
  • rrobinetrrobinet
    Version 1.1e has been in reliable service at a number of sites for a week. To summarize a clean installation:

    1) create the directory /home/pi/ham/kiwiwspr

    2) copy kiwiwspr-1-1e.sh to /home/pi/ham/kiwiwspr/kiwiwspr.sh

    3) run "chmod +x /home/pi/ham/kiwiwspr/kiwiwspr.sh"

    4) run: "/home/pi/ham/kiwiwspr/kiwiwspr.sh -h" until you are no longer prompted to install any of the following programs:
    The program will prompt you to install kiwirecorder.py. Download it from https://github.com/jks-prv/kiwiclient/tree/jks-v0.1 to /home/pi/ham/kiwiwspr/kiwiclient-jks-v0.1
    The program will prompt you in install WSJT-x from http://www.physics.princeton.edu/pulsar/K1JT/wsjtx.html
    The program will prompt you to install 'bc' by running: 'sudo apt-get install bc'

    5) Edit the prototype configuration file '/home/pi/ham/kiwiwspr/kiwiwspr.conf' created during step #4 to reflect:
    a) The name for your Kiwi(s), their IP addresses and Grid locator
    b) The schedule of WSPR bands for it to spot. Configure only hour '00:00' if you want the same configuration 24/7

    6) run "./kiwiwspr.sh -j a" to start the capture jobs defined in your kiwiwspr.conf

    7) print the status of your jobs by running "./kiwiwspr.sh -j s"

    Once "./kiwiwspr.sh -j s" lists capture and decode daemons for each of your configured bands:

    8) run "./kiwiwspr.sh -w a" to start the watchdog daemon which will every 2 minutes verify that your jobs are still running and execute the configuration changes defined in your kiwiwspr.conf
    "./kiwiwspr.sh -w a" will also configure your Pi so the watchdog daemon runs every time your Pi is powered up or rebooted.

    Finally, each job writes 200 Kbps of wav data to files in /tmp/kiwi-captures/... If that file tree is at its Pi default location on the microSD, this writing will significantly shorten the lifetime of the microSD. To avoid that wear, configure your Pi's /etc/fstab to make /tmp/kiwi-captures/ a ramdisk file system:

    pi@PiKPH:~/ham/kiwiwspr $ tail -2 /etc/fstab
    ### For kwiwwspr.sh capture files
    tmpfs /tmp/kiwi-captures tmpfs defaults,noatime,nosuid,size=100m 0 0
    pi@PiKPH:~/ham/kiwiwspr
    n6gnPowernumpty
  • SWLJO43SWLJO43
    Hello,

    in my case i use an older Computer and i installed
    - debian-9.5.0-amd64-netinst
    - dpkg -i wsjtx_1.9.1_amd64.deb
    - apt-get install bc
    - apt-get install ntp
    - apt-get install python-numpy
    - apt-get install curl
    and your kiwiwspr.sh to /home/pi/kiwiwspr
    and the kiwirecorder.py to /home/pi/kiwiwspr/kiwiclient-jks-v0.1


    To configure in my case: nano /home/pi/kiwiwspr/kiwiwspr.conf

    Now i use 6 Channels on my "http://jnlb0dg4phlqpnun.myfritz.net:8074" without problems.

    Thanks Dirk (SWLJO43)
    HuubWA2ZKDPowernumpty
  • VK2YCJVK2YCJ
    The progress you guys are making with these WSPR mods is very exciting. To be able to have a multiband HF antenna and a dedicated WSPR KiwiSDR to decode 6-8 bands at the same time 24*365 is nothing short of amazing. Keep up the good work guys.

    73, Jamie - VK2YCJ
  • rrobinetrrobinet
    Thanks. It is gratifying to be part of such an active and dedicated group.
    Everyone using the Kiwi, especially for WSPR, should upgrade to v 1.242. It corrects a bug which introduced -40dB +/- 23 Hz sidebands to spots of > +10 dB SNR. Even more important, it appears to have improved SNRs by about 2 dB which means the Kiwi will detect low SNR spots which it missed before.
    n6gnG0LUJSWLJO43GeneLX1DQ
  • GeneGene
    It is difficult to quantify results without a direct comparison to a benchmark, however I believe I am seeing an overall increase in WSPR spots after loading v 1.242. Many thanks to all the members of the team that worked to improve kiwiwspr.

    Gene W3PM
  • rrobinetrrobinet
    I have attached version kiwiwspr.sh 1.1g which includes a number of significant enhancements. The most important in my judgement is that it now tests the version of wsprd and if wsprd 2.0 is installed it is run with command line flags which will result in 10% more spots from the same captures.
    You can obtain that new wsprd by downloading to your Pi the WSJT-x 2.0 release candidate from the WSJT-x web site. I am told we can expect no further ?hanges in the wsprd included in the current RC3 package.
    To install this version, download this file to your Pi and rename it to kiwiwsr.sh and then replace your current kiwiwspr.sh with the renamed version.
    Verify the version you are running with "./kiwiwspr.sh -V'

    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/24/69e0f37e0a99a477ac3861ea99f04b.sh
    KK6PR
  • WA2ZKDWA2ZKD
    Picture of my system 12 channels of wspr using Rob's script
Sign In or Register to comment.