I have encountered two problems: 1) The script seems to be sensitive to packet read errors. After running for several hours on a Mac connected to the same LAN as the KIwi, I got this error:
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwiworker.py", line 39, in run self._recorder.run() File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwiclient.py", line 355, in run received = self._stream.receive_message() File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 757, in receive_message frame = self._receive_frame_as_frame_object() File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 462, in _receive_frame_as_frame_object opcode, unmasked_bytes, fin, rsv1, rsv2, rsv3 = self._receive_frame() File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 459, in _receive_frame unmask_receive=self._options.unmask_receive) File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 194, in parse_frame received = receive_bytes(2) File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 454, in _receive_bytes return self.receive_bytes(length) File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_base.py", line 159, in receive_bytes new_read_bytes = self._read(length) File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_base.py", line 128, in _read (length, e)) ConnectionTerminatedException: Receiving 2 byte failed. socket.error (timed out) occurred Median: -101 Thr: -208 S
I am trying to work around that problem by putting kiwirecorder.py in a bash forever loop.
2) When I add the --ncomp flag the capture file grows from 1 MB to 10 MB, which seems strange to me. I would expect the number and size of the samples in a 2 minute window should remain the same, only the amount of data transferred across the LAN should grow. Along with the larger file size, wsprd no longer finds spots in the capture file. It just silently reports no spots. This is a problem I don't know how to work around.
Thank you. The fix works and I am running a test of uncompressed audio from the browser feeding WSJT-x spotting as KPH2 against kiwirecorder uncompressed wav files processed by 'wsprd -d' spotting as KPH. Initial results show that in some captures the 'wspr -d' extracts 10% more spots. I am enhancing my script to fully document the differences. 5 instances of kiwrecorder consume only 12% of the i5 cpu of my 2001 MacAir, so I should be able to run the 12 instances for 2200-10M on the Mac and perhaps even on a Pi. Would it b possible to suppress the 'in progress Block NNNN, RSSR: NNN' printouts? They fill my log files with lines which are distracting even when viewed with 'less -S'
The -q really quiets it down - I get no output ;=) My only current (minor) problem is that in quiet mode nothing is printed until there is an error (e.g. kick the session off of the kiwi), at which time there is the error printout you saw above followed by a burst of lines which must be un-flushed stdout. e.g.: .... Started a new file: 20180701T001600Z_474200_usb.wav
Started a new file: 20180701T001800Z_474200_usb.wav
Started a new file: 20180701T002000Z_474200_usb.wav ....
I don't know how the python runtime works exactly w.r.t output flushing. But I'd expect it to be similar to libc. Most of the messages are done via print() calls which I would expect to flush due to the implied end-of-line \n. The "block" messages, with their single-line overwriting behavior, are done via sys.stdout.write() with a leading \r and an embedded sys.stdout.flush(). The -q option is effective bypassing the calls to sys.stdout.flush() but this shouldn't effect the implied flush from print() (if there is one). Are you running the "python kiwirecorder ..." in some sort of shell pipeline command?
Yes, running python -u produces the quiet mode output in my log files, so I now have all of the WSPR runctionality I want from the Kiwis. I am now running six instances of my kiwiwspr.sh script on a Pi 3b+ and using only 20% of its CPU except when 'nice wsprd -d' runs for 15 seconds at 100% of one of the four CPU cores. top says each kiwirecorder.py consumes about 6% of one of the four Pi CPU cores, which I think is an impressively low burden. I won't know for sure if one Pi can process wspr for all 12 bands until the Massdrop Kiwis arrive by their very slow truck shipper, but there is every indication that this single Pi will work. Thanks so much for your support through this project. I see many users of the Maui Kiwi you gave me and when those Kiwis arrive we will make your Kiwi at KPH publically available as well. I can upload my kiwiwspr.sh companion script if you or others would like to use it.
I am currently running 6 instances of this command line script on a Raspberry Pi 3b+ at KPH, and it should also run on Max OSX.
To use it:
1) create a directory for it (e.g. /home/pi/kiwiwspr) and copy the attached kiwiwspr.sh into that directory 2) execute: "chmod +x kiwiwspr.sh" 3) execute ./kiwiwspr.sh -h
The program will guide you through the setup of the environment. It depends upon:
1) the 'bc' (Binary Calculator) 2) the kiwirecorder.py program 3) and WSJT-x. WSJT-x doesn't need to run, only the 'wsprd' command line application is used here. 4) Then you will need to edit the template configuration file 'kiwiwspr.conf' file created by kiwiwspr.sh to include the IP address(es) of your Kiwi(s) 5) The simplest way to run the program is to execute './kiwiwspr.sh -J a' (start all jobs) ./kiwiwspr.sh -j will display the status of all the jobs defined in kiwiwspr.conf ./kiwiwspr.sh -h attempts to describe all of the command line options, although I hope you will need to use only "-J a" an "-J z"
There are log files for each job in the /tmp/kiwi-captures/... directory tree if you want to look 'under the hood'
I expect that you will experience problems installing and running ?this problem, so don't hesitate to ask for help.
./kiwi.sh -h => prints help menu ./kiwi.sh -J a => "CAP J" space "a": starts the job(s) defined in your kiwiwspr.conf file It spawns two processes for each job: 1) Captures 2 minute blocks of the kiwi output to a series of time-named .wav files in /tmp/kiwi-captures/... 2) Runs wsprd on those wav file from capture and uploads to wsprnet.org ./kiwi.sh -J z => "CAP J" space "z": stops/kills those jobs ./kiwi.sh -j => "lower case j": prints the status of those jobs
I am glad the script is working for you. I developed it because at KPH the are so many signals on 40/30/20M during the day that the Kiwi autowspr would run out of time before finding all the signals in a busy band. However Glenn N6GN has found today that his Apache => WSJT-x does report slightly better SNR than the Kiwi and my script. His bench measurements
an option on your script that would be handy would be a -L or ?, load so that all existing daemons are killed and the one in name.conf is loaded kiwiwspr.sh -L name.conf That would work great for using cron in Linux to shift bands based upon TOD
I didn't know there was any activity on 60M, but of course I'll add it. I too want to mmimic the "band Hopping' feature of WSJT-x. I won't have time to work on it for a few days. In the mean time, you could create a set of kiwiwspr.conf files and have the cron job run "-J z; mv replace kiwiwspr.conf; -J a"
Comments
1) The script seems to be sensitive to packet read errors. After running for several hours on a Mac connected to the same LAN as the KIwi, I got this error:
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwiworker.py", line 39, in run
self._recorder.run()
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwiclient.py", line 355, in run
received = self._stream.receive_message()
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 757, in receive_message
frame = self._receive_frame_as_frame_object()
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 462, in _receive_frame_as_frame_object
opcode, unmasked_bytes, fin, rsv1, rsv2, rsv3 = self._receive_frame()
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 459, in _receive_frame
unmask_receive=self._options.unmask_receive)
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 194, in parse_frame
received = receive_bytes(2)
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_hybi.py", line 454, in _receive_bytes
return self.receive_bytes(length)
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_base.py", line 159, in receive_bytes
new_read_bytes = self._read(length)
File "/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/mod_pywebsocket/_stream_base.py", line 128, in _read
(length, e))
ConnectionTerminatedException: Receiving 2 byte failed. socket.error (timed out) occurred
Median: -101 Thr: -208 S
I am trying to work around that problem by putting kiwirecorder.py in a bash forever loop.
2) When I add the --ncomp flag the capture file grows from 1 MB to 10 MB, which seems strange to me. I would expect the number and size of the samples in a 2 minute window should remain the same, only the amount of data transferred across the LAN should grow. Along with the larger file size, wsprd no longer finds spots in the capture file. It just silently reports no spots. This is a problem I don't know how to work around.
2) Yep, there was a bug. Please update from jks-v0.1 and try again. Files should be same size.
Would it b possible to suppress the 'in progress Block NNNN, RSSR: NNN' printouts? They fill my log files with lines which are distracting even when viewed with 'less -S'
My only current (minor) problem is that in quiet mode nothing is printed until there is an error (e.g. kick the session off of the kiwi), at which time there is the error printout you saw above followed by a burst of lines which must be un-flushed stdout. e.g.:
....
Started a new file: 20180701T001600Z_474200_usb.wav
Started a new file: 20180701T001800Z_474200_usb.wav
Started a new file: 20180701T002000Z_474200_usb.wav
....
I am now running six instances of my kiwiwspr.sh script on a Pi 3b+ and using only 20% of its CPU except when 'nice wsprd -d' runs for 15 seconds at 100% of one of the four CPU cores.
top says each kiwirecorder.py consumes about 6% of one of the four Pi CPU cores, which I think is an impressively low burden.
I won't know for sure if one Pi can process wspr for all 12 bands until the Massdrop Kiwis arrive by their very slow truck shipper, but there is every indication that this single Pi will work.
Thanks so much for your support through this project. I see many users of the Maui Kiwi you gave me and when those Kiwis arrive we will make your Kiwi at KPH publically available as well.
I can upload my kiwiwspr.sh companion script if you or others would like to use it.
To use it:
1) create a directory for it (e.g. /home/pi/kiwiwspr) and copy the attached kiwiwspr.sh into that directory
2) execute: "chmod +x kiwiwspr.sh"
3) execute ./kiwiwspr.sh -h
The program will guide you through the setup of the environment. It depends upon:
1) the 'bc' (Binary Calculator)
2) the kiwirecorder.py program
3) and WSJT-x. WSJT-x doesn't need to run, only the 'wsprd' command line application is used here.
4) Then you will need to edit the template configuration file 'kiwiwspr.conf' file created by kiwiwspr.sh to include the IP address(es) of your Kiwi(s)
5) The simplest way to run the program is to execute './kiwiwspr.sh -J a' (start all jobs)
./kiwiwspr.sh -j will display the status of all the jobs defined in kiwiwspr.conf
./kiwiwspr.sh -h attempts to describe all of the command line options, although I hope you will need to use only "-J a" an "-J z"
There are log files for each job in the /tmp/kiwi-captures/... directory tree if you want to look 'under the hood'
I expect that you will experience problems installing and running ?this problem, so don't hesitate to ask for help.
Attachments:
https://forum.kiwisdr.com/uploads/Uploader/4a/d65acd06a7d6c589e79bb65f985cd6.sh
declare -r KIWI_LIST=(
### Format of each element:
### OurID(no spaces) IP:PORT MyCall MyGrid KiwPassword (NULL => none required)
"MY_KIWI_0 192.168.1.5:8073 WA2ZKD FN13ed password"
)
### List of WSPR jobs to be run or killed by -j/-J
declare -r CAPTURE_JOBS=(
"MY_KIWI_0 80"
)
when I run kiwi.sh I get
root@odroid:~/kiwi# ./kiwi.sh
ERROR: bad or no args were supplied to command ''
./kiwi.sh -J a => "CAP J" space "a": starts the job(s) defined in your kiwiwspr.conf file It spawns two processes for each job:
1) Captures 2 minute blocks of the kiwi output to a series of time-named .wav files in /tmp/kiwi-captures/...
2) Runs wsprd on those wav file from capture and uploads to wsprnet.org
./kiwi.sh -J z => "CAP J" space "z": stops/kills those jobs
./kiwi.sh -j => "lower case j": prints the status of those jobs
That would work great for using cron in Linux to shift bands based upon TOD
I too want to mmimic the "band Hopping' feature of WSJT-x. I won't have time to work on it for a few days.
In the mean time, you could create a set of kiwiwspr.conf files and have the cron job run "-J z; mv replace kiwiwspr.conf; -J a"