What is the status of the WSPR background and autostart features? [added in v1.181]



  • I have encountered two problems:
    1) The script seems to be sensitive to packet read errors. After running for several hours on a Mac connected to the same LAN as the KIwi, I got this error:

    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwiworker.py", line 39, in run
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwiclient.py", line 355, in run
    received = self._stream.receive_message()
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 757, in receive_message
    frame = self._receive_frame_as_frame_object()
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 462, in _receive_frame_as_frame_object
    opcode, unmasked_bytes, fin, rsv1, rsv2, rsv3 = self._receive_frame()
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 459, in _receive_frame
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 194, in parse_frame
    received = receive_bytes(2)
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 454, in _receive_bytes
    return self.receive_bytes(length)
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_base.py", line 159, in receive_bytes
    new_read_bytes = self._read(length)
    File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_base.py", line 128, in _read
    (length, e))
    ConnectionTerminatedException: Receiving 2 byte failed. socket.error (timed out) occurred
    Median: -101 Thr: -208 S

    I am trying to work around that problem by putting kiwirecorder.py in a bash forever loop.

    2) When I add the --ncomp flag the capture file grows from 1 MB to 10 MB, which seems strange to me. I would expect the number and size of the samples in a 2 minute window should remain the same, only the amount of data transferred across the LAN should grow. Along with the larger file size, wsprd no longer finds spots in the capture file. It just silently reports no spots. This is a problem I don't know how to work around.
  • 1) Not sure what the read timeout is about.

    2) Yep, there was a bug. Please update from jks-v0.1 and try again. Files should be same size.
  • edited June 2018
    Thank you. The fix works and I am running a test of uncompressed audio from the browser feeding WSJT-x spotting as KPH2 against kiwirecorder uncompressed wav files processed by 'wsprd -d' spotting as KPH. Initial results show that in some captures the 'wspr -d' extracts 10% more spots. I am enhancing my script to fully document the differences. 5 instances of kiwrecorder consume only 12% of the i5 cpu of my 2001 MacAir, so I should be able to run the 12 instances for 2200-10M on the Mac and perhaps even on a Pi.
    Would it b possible to suppress the 'in progress Block NNNN, RSSR: NNN' printouts? They fill my log files with lines which are distracting even when viewed with 'less -S'
  • Update and use new arg -q / --quiet to silence the "block" messages.
  • edited July 2018
    The -q really quiets it down - I get no output ;=)
    My only current (minor) problem is that in quiet mode nothing is printed until there is an error (e.g. kick the session off of the kiwi), at which time there is the error printout you saw above followed by a burst of lines which must be un-flushed stdout. e.g.:
    Started a new file: 20180701T001600Z_474200_usb.wav

    Started a new file: 20180701T001800Z_474200_usb.wav

    Started a new file: 20180701T002000Z_474200_usb.wav
  • I don't know how the python runtime works exactly w.r.t output flushing. But I'd expect it to be similar to libc. Most of the messages are done via print() calls which I would expect to flush due to the implied end-of-line \n. The "block" messages, with their single-line overwriting behavior, are done via sys.stdout.write() with a leading \r and an embedded sys.stdout.flush(). The -q option is effective bypassing the calls to sys.stdout.flush() but this shouldn't effect the implied flush from print() (if there is one). Are you running the "python kiwirecorder ..." in some sort of shell pipeline command?
  • jksjks
    edited July 2018
    And some quick Googling shown python print() output is, in fact, buffered. See if you can do "python -u kiwirecorder.py ..." to un-buffer the I/O.
  • Yes, running python -u produces the quiet mode output in my log files, so I now have all of the WSPR runctionality I want from the Kiwis.
    I am now running six instances of my kiwiwspr.sh script on a Pi 3b+ and using only 20% of its CPU except when 'nice wsprd -d' runs for 15 seconds at 100% of one of the four CPU cores.
    top says each kiwirecorder.py consumes about 6% of one of the four Pi CPU cores, which I think is an impressively low burden.
    I won't know for sure if one Pi can process wspr for all 12 bands until the Massdrop Kiwis arrive by their very slow truck shipper, but there is every indication that this single Pi will work.
    Thanks so much for your support through this project. I see many users of the Maui Kiwi you gave me and when those Kiwis arrive we will make your Kiwi at KPH publically available as well.
    I can upload my kiwiwspr.sh companion script if you or others would like to use it.
  • Yes, please upload is and also a description of your block diagram. Thanks!
  • I plan to try this on an Odroid XU4 which is even faster than any Pi
  • edited July 2018
    I am currently running 6 instances of this command line script on a Raspberry Pi 3b+ at KPH, and it should also run on Max OSX.

    To use it:

    1) create a directory for it (e.g. /home/pi/kiwiwspr) and copy the attached kiwiwspr.sh into that directory
    2) execute: "chmod +x kiwiwspr.sh"
    3) execute ./kiwiwspr.sh -h

    The program will guide you through the setup of the environment. It depends upon:

    1) the 'bc' (Binary Calculator)
    2) the kiwirecorder.py program
    3) and WSJT-x. WSJT-x doesn't need to run, only the 'wsprd' command line application is used here.
    4) Then you will need to edit the template configuration file 'kiwiwspr.conf' file created by kiwiwspr.sh to include the IP address(es) of your Kiwi(s)
    5) The simplest way to run the program is to execute './kiwiwspr.sh -J a' (start all jobs)
    ./kiwiwspr.sh -j will display the status of all the jobs defined in kiwiwspr.conf
    ./kiwiwspr.sh -h attempts to describe all of the command line options, although I hope you will need to use only "-J a" an "-J z"

    There are log files for each job in the /tmp/kiwi-captures/... directory tree if you want to look 'under the hood'

    I expect that you will experience problems installing and running ?this problem, so don't hesitate to ask for help.

  • my kiwi.conf look like

    declare -r KIWI_LIST=(
    ### Format of each element:
    ### OurID(no spaces) IP:PORT MyCall MyGrid KiwPassword (NULL => none required)
    "MY_KIWI_0 WA2ZKD FN13ed password"

    ### List of WSPR jobs to be run or killed by -j/-J
    declare -r CAPTURE_JOBS=(
    "MY_KIWI_0 80"

    when I run kiwi.sh I get

    root@odroid:~/kiwi# ./kiwi.sh
    ERROR: bad or no args were supplied to command ''
  • edited July 2018
    ./kiwi.sh -h => prints help menu
    ./kiwi.sh -J a => "CAP J" space "a": starts the job(s) defined in your kiwiwspr.conf file It spawns two processes for each job:
    1) Captures 2 minute blocks of the kiwi output to a series of time-named .wav files in /tmp/kiwi-captures/...
    2) Runs wsprd on those wav file from capture and uploads to wsprnet.org
    ./kiwi.sh -J z => "CAP J" space "z": stops/kills those jobs
    ./kiwi.sh -j => "lower case j": prints the status of those jobs
  • an issue with captures as it needs numpy... need to track that down
  • edited July 2018
    now working!!
  • tons of decode on 40, will need to compare after a while
  • it definitely catches stuff that the kiwi wspr would miss. I need to now consider how to use my 5 available channels of kiwi with this scheme
  • I am glad the script is working for you. I developed it because at KPH the are so many signals on 40/30/20M during the day that the Kiwi autowspr would run out of time before finding all the signals in a busy band. However Glenn N6GN has found today that his Apache => WSJT-x does report slightly better SNR than the Kiwi and my script. His bench measurements
  • what about 60M... I can add it but curious why left out
  • an option on your script that would be handy would be a -L or ?, load so that all existing daemons are killed and the one in name.conf is loaded kiwiwspr.sh -L name.conf
    That would work great for using cron in Linux to shift bands based upon TOD
  • I didn't know there was any activity on 60M, but of course I'll add it.
    I too want to mmimic the "band Hopping' feature of WSJT-x. I won't have time to work on it for a few days.
    In the mean time, you could create a set of kiwiwspr.conf files and have the cron job run "-J z; mv replace kiwiwspr.conf; -J a"
Sign In or Register to comment.