wsprdaemon - A Raspberry Pi WSPR decoding service

rrobinetrrobinet
edited December 2019 in WSPR, wsprdaemon, kiwirecorder
This is a major upgrade in functionality, reliability and installability from previous versions of wsprdaemon and its predecessor kiwiwspr.sh
Configured to connect to two Kiwis configured in 8 channel mode, it easily simultaneously decodes all 14 WSPR LF/MF/HF WSPR bands during every two minute cycle and reliably uploads spots to wsprnet.org

To install it:

1) login to the Pi as user 'pi' (or other non-root user) and create the directory '~/wsprdaemon'.
2) copy the attached file to that directory, rename it to 'wsprdaemon.sh', and make it executable with 'chmod +x '
3) cd to that directory and execute it: './wsprdaemon.sh'. It should walk you through the installation of all of the utility programs it needs and create a prototype 'wsprdaemon.conf' file
4) edit the wsprdaemon.conf file following the directions in it
5) to start the daemon: './wsprdaemon.sh -a'
6) to see the daemon status: './wsprdaemon.sh -s'
7) to stop the daemon: './wsprdaemon.sh -z'
8) for help about the daemon: './wsprdaemon.sh -h'

During installation it creates a tmpfs files system '/tmp/wspr-captures/...' where all of the recordings and logs are kept, so 24/7 recording of 14 bands => 3 Mbps of wav files doesn't wear out your microSD in a few months
It also modifies the systemclt files so the daemon will autostart during a powerup or reboot.

I have tested installation and operation on a clean 'Stretch' OS. The script can run in other Debian servers, but installation may require some custom tweaks.

Attachments:
https://forum.kiwisdr.com/uploads/Uploader/e1/1c8ad4c70a3021e25b6bf2ca55aae2.txt
HB9TMCPowernumptyrz3dvp
«13456

Comments

  • WA2ZKDWA2ZKD
    It also runs on Odroid with Ubuntu
  • SWLJO43SWLJO43
    It also runs on an PC with Debian

    cu Dirk
  • PowernumptyPowernumpty
    Virtual PC with Ubuntu too (64 bit).

    Had to upgrade (install) wsjtx on the VM
    wget https://physics.princeton.edu/pulsar/K1JT/wsjtx_2.0.1_amd64.deb --no-check-certificate
    sudo dpkg -i wsjtx_2.0.1_amd64.deb

    Also get the kiwiclient (after installing git)
    git clone git://github.com/jks-prv/kiwiclient

    Only posting that to remind myself, for some reason I always seem to make hard work of this.
  • rrobinetrrobinet
    In V2.3 I test for those applications and try to download them if they are missing. Did I fail to try to find they were missing or did my downloads fail?
  • PowernumptyPowernumpty
    The VM in question is more likely the issue, it had WSJTX 1.9 something so that was installed and I guess it would need confirmation to overwrite the old one?
    It did identify the installed version report that I should run with the correct setting for the version installed.

    It reported also the Kiwiclient as being required and where to get it. Probably wouldn't hurt to have a post linked to show that the easiest way to make it available in the ~/wspdaemon directory via git or elsewhere and added to the path.
    Coming from new user, not on the intended platform (64B) - the script requires some "things" and they are not installed the same way, one -WSJTX is an install (or can be) and one is a simple download that needs to be accessible to the user/script.
    Users of the Kiwiclient and probably wsjtx are so used to having them already and where they should be, that it is a non issue for most.
    As I said for some reason I got frustrated in the past having broken a working install (SDRPlay Pi Image) during an upgrade I am the very worst person to test something like this.

    This a 99.98% fine on an Ubuntu 64B VM, I expect 100% if used correctly as described in the thread title (Pi).

    Thanks for your fine work.
  • oe9ghvoe9ghv
    edited May 2019
    Hello,

    thanks for the great work! So far i got it running with a kiwi configured to 8 slots. But, the spottet frequency/band on wsprnet.org is not correct. Frequency is shown as 0.001470 or 0.001424 on wsprnet. the call for spots is oe9ghv-1 at the moment.
    Where i made a mistake?

    vy 73 de Holger, OE9GHV
  • oe9ghvoe9ghv
    Hello,
    i found the failiure in my installation. I needed to change the locale from de_AT to en_GB on my raspberry. So it is a point-comma
    bug. Works now perfectly. For comparison there is a secound kiwi with a splitter at the same antenna witch is decoding on 20m with the call OE9PTI. WSPRDAEMON is spotting with OE9GHV on 8 bands. The antenna is a about 80m long vertical loop.
    The kiwi is accessible via kiwi.oe9.at:8073 . If the radio station is in use, the kiwis have no antenna connected (via a Coaxrelay)!

    vy 73 de Holger, OE9GHV
    WA2ZKD
  • n6gnn6gn
    Holger,
    Thanks for catching the localization issue. We here on the 'left side' didn't notice it before now!

    I don't see any spots from OE9GHV-1 at the moment but if/when both wsprdaemon and kiwi-extension WSPRs are running, on a busy band I think you'll find that wsprdaemon does much better. That is my experience. The cpu in the BeagleBones is just too busy to get around to all the necessary work within 2 minutes! OTOH, even a Raspberry PI can handle more than a dozen simultaneous WSPR bands running wsprdaemon. Since it uses the WSJT-X wsprd decoder, it gets the same (good) results.

    What a pretty spot you have on "Radio Hill", shown on your Kiwi's splash screen! It also is showing a nice low and flat noise floor 0-30 MHz. I think it may do well on WSPR. There's a bit of thunderstorm activity right at the moment so I can't get a really good number for noise floor but it appears to be below -140 dBm/1-Hz on 80m. Presuming your antenna efficiency to be OK, that it is well matched, this would appear to be well on the way to a "quiet site" on the ITU-R P.372-8 noise scale.

    Best,
    Glenn n6gn

    [also posted to wsprnet.org]
  • rrobinetrrobinet
    Hi Holger,

    Thanks for reporting that localization issue. I expect I can add a line to wsprdaemon V2.4 which defines the number format so you don't need to make any environment changes.
    Your site is excellent and I expect you may get 2 to 3 times the spots from WD than from the autowspr on busy bands.
    I am starting beta testing of V2.4 which adds noise level logging and reporting to WD. See http://kphsdr.com:20080/ for an example of the output.
    Email me If you are interested in participating.

    73,
    Rob
  • oe9ghvoe9ghv
    Hello,

    Glenn, the call used by the wsprdeamon now is OE9GHV. The kiwi uses OE9PTI for decoding. I compared it a few times searching the database for the last 60 minutes. The wsprdaemon heard 50 to 100% more! I am vy happy with this solution!

    This summer we will an an other loop, 90degree offset from this one. There will be also 2 kiwis listening. One exclusivly for wspr.

    Thanks for your great work!

    vy 73 de Holger, OE9GHV
  • WA2ZKDWA2ZKD
    Holger,

    I note that today you are one of the Top 5 wspr-reporters in the World.
  • oe9ghvoe9ghv
    Hello,

    Glenn, the call used by the wsprdeamon now is OE9GHV. The kiwi uses OE9PTI for decoding. I compared it a few times searching the database for the last 60 minutes. The wsprdaemon heard 50 to 100% more! I am vy happy with this solution!

    This summer we will an an other loop, 90degree offset from this one. There will be also 2 kiwis listening. One exclusivly for wspr.

    Thanks for your great work!

    vy 73 de Holger, OE9GHV
  • n6gnn6gn
    Holger,
    FB. As Jim mentioned you are doing well with the Kiwi from that site. This morning (US time) you are number 3 in Europe and the world and #1 on 30m.

    Rank Reporter spots uniqs 2200m 630m 160m 80ja 80m 60eu 60m 40m 30m 20m 17m 15m 12m 10m V/U
    1 DK6UG 15493 765 0 4 18 52 32 9 19 217 122 186 41 35 7 23 0
    2 EA8BFK 9134 759 0 0 0 49 0 6 0 214 164 240 51 21 0 14 0
    3 OE9GHV 11207 737 0 3 19 47 0 0 0 216 170 187 41 31 5 18 0

    (Sorry, I can't easily make the columns line up)
    Glenn n6gn
  • rrobinetrrobinet
    Hi Holger,

    Add spotting on 80M and the two 60M bands and you will be #1 worldwide! How many Kiwis do you have and are they fed by the same antenna? I want one for my Maui WSPR site ;=)

    Rob
  • WA2ZKDWA2ZKD
  • oe9ghvoe9ghv
    WOW!!!

    Wsprdaemon is working really well! Yesterday evening i changed the band config a bit. Now it looks like i am on place 2. GREAT!

    Rob, there are running 2 kiwis fed by the same antenna via a splitter. One kiwi is public access and spotting on 20m
    wspr with the call OE9PTI. The second is configured to 8 slots and is only for wspr.

    This is an old picture showing the kiwis, splitter and the coaxrelay. The wspr-pi is not on the picture.

    vy 73 de Holger, OE9GHV

  • WA2ZKDWA2ZKD
    edited May 2019
    Holger,

    You should consider running the kiw spotting on 20m wspr with the call OE9PTI on wsprdaemon also and assign it to 3.59, th other portion of 80M
  • jimcnyjimcny
    Up and running on Atomic Pi. First thing after loggin run;

    1) sudo systemctl disable lightdm.service
    will now boot to prompt and services will start

    2) when armhf config fails uses this link
    http://physics.princeton.edu/pulsar/K1JT/wsjtx_2.0.1_amd64.deb
    sudo dpkg -i wsjtx_2.0.1_amd64.deb

    3) edit your .conf

    over double performance of pi with
    rtc
    16GB eMMC
    2GB DDR3L-1600
  • rrobinetrrobinet
    I have attached WD V2.5a to this post. It includes a large number of installation and run time fixes, most importantly a 'monitor for and kill zombie recording sessions on the Kiwi(s)'.

    There is also major new feature: It can be configured to generate graphs of background noise level measurements at the same time as WSPR decodes which can be viewed by the Apache web service running on the WD server.
    Those graphs can give you great insights into the quality of your receive system and site. However calculating the noise levels triples the CPU requirements on the server, so a Raspberry Pi which can support 20 no-noise bands can only support 8 bands with the noise feature enabled. So if you have more than one Kiwi and want noise graphs, you should wun WD on an Odroid, Atomic Pi, or x86 server running Ubuntu 18.04.

    This code should be completely compatible with existing 2.3 installations, so save your WD2.3, stop your current WD with '-z', copy this code over the existing WD.sh and start it with '-a'
    As before, '-s' will print the status of your system.

    Because of all the fixes, I recommend that all sites upgrade to this code. Please report your experience and if you have problems you can easily revert to your WD 2.3 code.

    Attachments:
    https://forum.kiwisdr.com/uploads/Uploader/0e/ca90dc60bc6d1902ca12fa594d8b0d.txt
    PowernumptyHB9TMCrz3dvp
  • PowernumptyPowernumpty
    Have moved across will report back.
    Thanks again for the software, (and Kiwi) never thought I'd be graphing band reception so clearly here and comparing with solar wind values, at times getting a feel for a more energetic atmosphere before it shows on solarham.

    Other change I have made is today building a shed within six inches of my ground mounted antenna, (space is tight) if there is any negative result for my WSPR reports it will most likely be that.
  • rrobinetrrobinet
    edited June 2019
    If you don't see the diurnal background noise level changes like those at KPH : http://graphs.wsprdaemon.org/, then your antenna and/or location need improvement or you are in a part of the world where there are few thunderstorms.
    It appears that at quiet sites like KPH the Kiwi's noise figure limits it WSPR decode sensitivity at 30M and above. We are working on a preamp system to address that problem.
  • PowernumptyPowernumpty
    edited June 2019
    I can't face looking at the noise, I know I could do with improving antenna and location already. It's taken me so long to get where I am by incremental improvement if I change anything, even slightly, it gets worse now (E-field horror site, small houses, very close together). Will look at the apache stuff for the practice and maybe marvel at other's noise figures while rocking back and forth mumbling...
  • PowernumptyPowernumpty
    As usual I will be the fool who asks:

    I assume the process is
    As above, replace the wsdaemon.sh then run it.
    >that generates /tmp/wsprdaemon.conf that we edit to reflect our local receiver list
    We then copy that back to the wpsrdaemon.sh root folder, replacing the previous version to allow us to take advantage of the new options.

    I managed to get an unworkable receiver array by moving the signal overrides on KIWI_1 to KIWI_2 without properly closing _1 or adding an extra quotation on 2, my error but not sure the "### Each element of RECEIVER_LIST is a string with 5 space-separated fields:" is true now when there may be more?
  • rrobinetrrobinet
    edited June 2019
    You can use your old wsprdaemon.conf and un-remark one or more of these lines to add noise level monitoring features. I chose not to run the local Apache server since I can see the KPH graphics page at http://graphs.wsprdaemon.org/KPH.

    #SIGNAL_LEVEL_STATS=yes ### Generates a signals.log file in each rx directory and publishes it in this server's home web page
    #SIGNAL_LEVEL_UPLOAD_ID="KPH" ### The name put in the title bar of the graph
    #SIGNAL_LEVEL_UPLOAD="yes" ### If this variable is defined AND SIGNAL_LEVEL_UPLOAD_ID is defined, then upload signal levels to the wsprdaemon cloud database
    #SIGNAL_LEVEL_UPLOAD_URL="us-central1-iot-data-storage.cloudfunctions.net" ## use this until we get 'logs.wsprdaemon.org' working as a URL.
    #SIGNAL_LEVEL_LOCAL_GRAPHS="no" ### If this variable is defined AND SIGNAL_LEVEL_UPLOAD_ID is defined, then put a graph in the local web server page
    #SIGNAL_LEVEL_UPLOAD_GRAPHS="yes" ### If this variable is defined AND SIGNAL_LEVEL_UPLOAD_ID is defined, then upload the graphs to logs.wsprdaemon.org
  • PowernumptyPowernumpty
    edited June 2019
    I did consider just taking across the lines but figured I'd then have to reference two files for experimentation so I replaced the previous config for the comments and example entries.

    On my Ubuntu VM enabling the stats (after installing Apache2, no reboots or permissions changes) stopped it uploading spots as it was throwing sudo password requests, I assume I just need to sort the group membership for my user writing to www.
    It was late so I just turned it back off to revisit when I next have time but was a little surprised as just before bed I checked the last ten minute's RX (for my call) on http://wsprnet.org/drupal/wsprnet/map and got "nothing".

    --later that day--

    "sudo chown -R myusername:myusername /var/www"

    Got me going, I think, just did that on my lunch and more software was installed and files have been created, will check after work.
  • rrobinetrrobinet
    WD creates a whole directory tree under /tmp/wspr-captures and of course is subject to Linux user permissions restrictions.
    So don't install as one user and try to run as another user.
    If you want to run as a different user, you can clean out that file system with: 'sudo rm -rf /tmp/noise_graph.png /tmp/noise_plot.py /tmp/wsprdaemon.conf /tmp/wspr-captures/*'
    When SIGNAL_LEVEL_STATS=yes, the number of sudos and programs installed is daunting and I only install programs required by the configuration, but they are all from the repository.
  • PowernumptyPowernumpty
    I think it needed to check write access to /var/www/ and by default in Ubuntu the user does not have that write permission.
    There is only one user on the machine, with sudo of course.

    The issue was that the "password for user" prompt didn't seem to be actually asking for the password as when I started typing it was not obscured and was taken as a new command.

    I think the script running as the normal user needs write access to /var/www/ to properly execute, once that was writeable I had one sudo password request and few "OK to install" questions which went well and I'm now geting local graphs.
  • rrobinetrrobinet
    each two minutes WD executes a 'sudo cp /tmp/noise_graph.png /var/www/html' which will prompt for the user's password if auto sudo isn't configured.. I will get rid of the need for that sudo in 2.5b.
  • PowernumptyPowernumpty
    OK that makes sense, at the moment it's working, no prompts unless they are hidden.
    For me, as a single user, it was easy to just own the /var/www/ directory, not sure if it should work like that but seems to.

    Not sure how to check the noise info is being uploaded though.
  • rrobinetrrobinet
    I have just enhanced graphs.wsprdaemon.org so that if you configure your wsprdaemon to upload graphs, you can view those graphs at http://graphs.wsprdaemon.org/SIGNAL_LEVEL_UPLOAD_ID/
    It can take up to 10 minutes between setting up your configuration and the appearance of that link.
    Sites with low noise levels (i.e. < -150 dBm) or 'top spotters' may appear on http://graphs.wsprdaemon.org/ unless you don't want your graphs there.
    G0LUJ
Sign In or Register to comment.