As an alternative, I am studying writing a Kiwi extension modeled on the auto-wspr extension which simply writes the 2 minutes of IQ samples to a file on the Kiwi and then each 2 minutes it transfers that file to a PC/Mac/Linux server for processing by the WSJT-x sw on that machine. Such an approach would disconnect the Kiwi-PC connection from time-critical functions and also free the Kiwi from a need future changes/enhancements to the WSPR processing code base.
The problem with this is that you're now waking up Linux to write the files and that will likely destroy the realtime environment. Especially with the WSPR extension which is running the cpu flat-out with all those FFTs and even more so with all 4 channels active. The fax extension seems to get away with it with its file recorder. But the processing requirements there are much lower.
I didn't realize that time critical operations on the Kiwi run in user space code. My intention is to minimize the signal processing performed on the Kiwi by transferring the IQ or audio data produced (I assume) by the FPGA to a server with a much more powerful CPU, rather than have the Kiwi CPU process it.
As for file I/O: In my products I have avoided uncontrolled CPU stalls in Disk IO operations by writing files to RAM disk partitions. Alternatively, I would send the IQ/audio data over the LAN; it seems that the current auto-wspr code is able to do such transfers to wsprnet.org without disrupting Kiwi timing. What would you suggest?
I didn't perform an experiment with only one auto-wspr running. I was running three auto-wsprs on 40/30/20M and then ran a browser session from a Mac on the fourth rx which was tuned to 20M. I had WSJT-x on the Mac log spots I compared the autowspr spots against the wsjt-x spots and found 2x the number of spots from wsjt-x 1.8.0. I will try a 1 autowspr versus wsjt-x to see if the results are still 2x for wsjt-x.
Yes, I do everything myself in user space. At various times during development I tried C-threads, Linux realtime scheduling etc. and I had nothing but trouble.
I would suggest just sticking with processing the existing audio IQ stream you can currently get over the net. You have the Python kiwiclient example showing how to do it. Many other languages have web socket libraries if you don't want to deal with Python.
As for why the Kiwi timing isn't disrupted by uploads to wsprnet.org: it's because the upload happens from javascript in the user's browser, not from C-code running on the Kiwi side. This is an example of the advantage of being able to split your extension code into C and javascript parts.
OK. Thanks for the direction. I will work on the python script.
I thought I saw last fall that the RDF work was enabled by a 20 uSec accurate timestamp prepended to the IQ samples. If that is true, then how/where are those highly time accurate samples and where are stored?
These timestamps are stored in a custom wav header. For this you need to use the command line parameter '-w'. An example for decoding this custom wav header can be found in the c++ code for the read_kiwi_iq_wav .oct file. The precision of these time stamps is better than 1us (more like <~100ns)
To elaborate on Christoph's answer: For each SND (sound) type packet received by kiwiclient.py::_process_aud() there is a header containing a sequence number and S-meter value (also containing some flags). If, and only if, IQ mode has been selected there is also the GPS timestamp. If you're running kiwirecorder.py then there is the "-w" option (aka "--kiwi-wav") as Christoph mentions to create the .wav file with the timestamp in a dedicated header. The presence of this header in the file does not effect it being played as a regular .wav file but allows later post-processing code to recover the timestamp in its proper position in the stream.
But the .wav file stuff doesn't matter much as what you need to do is stream IQ data to the WSPR app in as much realtime as possible. I don't know anything about how WSPR accepts data in this case (and more importantly at what data rate). K9AN's WSPR code in the Kiwi WSPR extension needs the data at a sample rate of 375 Hz. So the 12 kHz Kiwi IQ audio is decimated by 32. But because the Kiwi sample rate is not exactly 12 kHz you may need to fill/remove samples at the two minute epoch using the timestamp to keep time-alignment. Or you may be able to restart the entire decode process at the two minute boundary which is what the Kiwi code does. But this would only work if the WSPR app doesn't require continuous samples to be supplied from one epoch to the next.
Thanks for that clarification. I hope to be able to invest time in this soon. My cursory examination of the WSJT-x code on my PC uncovered a stand-alone wspr decode application that is delivered as part of the WSJT-x installation. Since I've had WSJT-x hang after running for a day or two, it would seem best to avoid running the WSJT-x GUI and just feed the wav file to that wsprd app. I would much prefer to run the whole process from a Python or bash script so it can recover from timeouts and other error conditions. So I thought I would modify kiwirecorder to wait for the start of the 2 minute cycle and have it write the 110 seconds of wav data to a file. then run that file thorugh wsprd, then write a bash/python/.. script parse its output and post the results to wsprnet.org
There would be no change or new burden to the Kiwi, I would exploit all of the considerable work in the kiwirecorder.py, and exploit the current and future versions of wsprd. The result would be portable to any platform for which there is a WSJT-x distribution.
Does that seem a reasonable approach?
============
rob@Robs-MBP:/Users/rob> /Applications/wsjtx.app/Contents/MacOS/wsprd ; exit; Usage: wsprd [options...] infile infile must have suffix .wav or .c2
Options: -a path to writeable data files, default="." -B disable block demodulation - use single-symbol noncoherent demod -c write .c2 file at the end of the first pass -C maximum number of decoder cycles per bit, default 10000 -d deeper search. Slower, a few more decodes -e x (x is transceiver dial frequency error in Hz) -f x (x is transceiver dial frequency in MHz) -H do not use (or update) the hash table -J use the stack decoder instead of Fano decoder -m decode wspr-15 .wav file -q quick mode - doesn't dig deep for weak signals -s single pass mode, no subtraction (same as original wsprd) -v verbose mode (shows dupes) -w wideband mode - decode signals within +/- 150 Hz of center -z x (x is fano metric table bias, default is 0.45) logout Saving session... ...copying shared history... ...saving history...truncating history files... ...completed. Deleting expired sessions...5 completed.
Seems reasonable. It would be interesting to know if wsprd is as up-to-date as the algorithms in WSJT-x. Another thing to try before investing too much effort is to collect some files manually while doing VAC to WSJT-x and then compare the decode "efficiency" against wsprd using the file. An A/B comparison in other words.
I hope to find that wsjt-x is already set up to use wsprd. Hopefully the source code isn't impenetrable or there are wsjt-x developers as helpful as you and Christoph
It appears that WSJT-x executes 'wsprd' from the GUI interface when it has a capture file to process, so I am pursuing using wsprd. However I've run into some bumps in the road to using kiwirecorder.py:
1) It doesn't know how to supply a password and can't even access the one 'no password needed' channel on my Kiwis. When I have more Kiwi's later this month I can work around this, but it would be nice to support Kiwi passwords.
2) To record the WSPR packets I am waiting until the next 2 minute start and then spawning kwirecorder.py, waiting for 110 seconds and killing the process. this seems ugly, so it would be nice if there were a 'record for an optional X seconds and terminate' flag in kiwirecorder. I have yet to examine the output wav file, but hopefully my pkill of kiwirecord will leave behind a file with a fairly predictable start time and length.
3) In addition, kiwirecorder seems to record to a filename it autocreates. Those names are unique and I think I can extract them from the recorder's output, but it would be nice if there were a flag to pass the record filename. I can work around that one too, but the names suggest that the recording starts some seconds after kiwirecorder is started. Fixing that so the record starts at 2 minutes is of course essential for this scheme to work
wsprd accepts a wav file as its input, but I am next going to see if (probably how much) the recorder's output file needs to be massaged in order to be accepted by wsprd
Thanks for the enhancements. I found that if I didn't pass a fully qualified filepath to '--fn=' it would corrupt the kiwirecorder.py environment. With a full path it works fine. wsprd doesn't decode a usb or iq format capture, so I can now proceed to the next task of figuring out what wav format it wants. I'll report back tomorrow.
"Corrupt the environment"? You mean like it erased other files or the current directory? I had no problems using plain file names with --fn.
Also, from your earlier message, I tried the "no password needed" mode and it worked fine for me. If I had a password-protected Kiwi with one password-free channel allowed then I could make one connection as expected but not a subsequent one.
The k9an-wsprd code I used for the Kiwi wanted a mono .wav file having a sample rate of 12 kHz and a BFO of 1500 Hz. The routine wsprd.c::readwavfile() does an FFT and down-samples by 32 to get the required 375 Hz sample rate.
starting kiwirecorder.py by hand or by a script every two minutes is certainly possible, but it might be better to let kiwirecorder.py chop the wav files into two minute long files. I have just committed a small change to kiwirecorder.py for this: when the option --dt-sec=DT is used, a new file is started every DT seconds whenever sec_of_day%DT == 0.
Thanks @Christoph; your enhancement is certainly superior.
Do get a capture file from the Kiwi I am executing: /Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwirecorder.py -s 10.11.11.72 -p 8073 -f 14.0956 -m usb -L 300 -H 2600 --pw=password --tlimit=110 --fn=/tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav
I am first trying to see how that wav file created by kwirecorder.py can be processed by wsprd. It outputs spots from the wspr_example.wav file supplied in the wsjt-x source, but wsprd immediately and silently returns when supplied the kwiwrecorder capture file. ffprobe shows the only difference in the wav file is the (I think) sample rate of the good wav file is 12000 Hz while kiwirecorders's wav file reports 12001 Hz sample rate. Before I start adding diagnostics to wsprd or blindly processing the kwirecorder wav file, could the 12001 Hz be a rounding error in kiwirecorder? The similar wav file sizes suggest that sample rates are pretty close.
rob@Robs-MBP:/Users/rob/ham/kiwi> ls -l /tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav wspr_example.wav -rw-r--r-- 1 rob wheel 2646060 Jun 8 07:28 /tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav -rw-r--r-- 1 rob staff 2880044 Jun 7 21:58 wspr_example.wav
kiwirecorder adds the .wav extension to the filename given by --fn / --filename. So in your example command you should end up with a file name ending with .wav.wav This behavior is perhaps wrong.
The 12001 business is indeed round-off error in the sample rate recorded in the /wav header. Remember that the Kiwi does not have a sampling rate of exactly 12 kHz due to the ADC clock frequency and correction by the GPS. But my 2015 version of wsprd ignores the .wav file header. You should check your wsprd and see if it rejects the file if the header sample rate is not exactly 12000. The file sizes will differ mostly because of small differences in the exact recording time. Even at only 12 kHz the files grow quite quickly.
Also note the -f parameter is in kHz. So you want "-f 14095.6" instead of what you have.
After you have it working you might want to try an A/B comparison between your wide 300 - 2600 Hz passband and narrowing it down to just the required 300 Hz (i.e. 1500 +/- 150). My version of wsprd uses an FFT to do the 32:1 decimation, so the filtering is essentially built-in. But I'm not sure what you've got.
The same spots, except OZ8WSP, were shown in the KiwiSDR window in the web browser.
I am not an expert in WSPR receiving but it seems you have to set the rx on the frequencies listed here http://www.g4ilo.com/wspr.html and as John has mentioned the signals are expected to be between 1400 and 1600 Hz in your recorded files.
In wsprd there is the option '-s single pass mode, no subtraction (same as original wsprd)' With this option I get the same decodes as from the KiwiSDR. So one possibility is that the differences between wsprd and KiwiSDR are not due to limited computing power on the KiwiSDR but due to the fact that wsprd is using improved decoding algorithms.
Thanks @Christoph, it seems that your enhancement is just what I need. Do I find your changes in the kiwirecorder main trunk? @jks password enhancement is useful to me.
there is a frequency argument to wsprd which should add that value to the offsets seen in your report.
My primary motivation for this is that I have found that the wspr decoder included in the kiwi cannot decode all of the signals recorded on HF at KPH here in California. During the day on 20meters, the antenna systems there are so good that the Kiwi runs out of time before finishing its wspr decode.
I'll report back once I script the decoding and posting the results to wsprnet.org
Yes the option to start a new wav file every dt seconds (--dt-sec) is in the jks-v0.1 branch.
> My primary motivation for this is that I have found that the wspr decoder included in the kiwi cannot decode all of the signals recorded on HF at KPH here in California. During the day on 20meters, the antenna systems there are so good that the Kiwi runs out of time before finishing its wspr decode.
Do you still see differences when you use the option "-s" with wsprd?
One more thing: you should use the option "-T -100" with kiwirecorder.py. Otherwise the squelch can interfere with data taking and start a new wav file inside a 2min window.
Yes, it is working perfectly for me. I will use it to run a bank of Kiwis at KPH, one for each WSPR band 24/7/365. Look for many more KPH spots later this month once the Kiwis I ordered from Massdrop are delivered.
In addition to the problem decoding all of the spots on a busy band, the decoder in the Kiwi is somewhat old and the wsjt-x 19.1 release notes says the wspr decoding performance has been improved. For example even in a band with only 2 signals, in my command line test just now, wsprd picked out the -18 SNR signal of WA4SZE on 630M which was missed by the Kiwi auto-wspr:
wsprd handles the problem of strong signals "covering up" weak ones in an interesting way. After a first pass of decoding signals they are coherently subtracted from the spectrum and the entire decoding pass is run a second time to, hopefully, pickup the remaining weaker signals.
The Kiwi version doesn't do the subtraction and second pass because of how it implements the wsprd algorithm. To more efficiently use the limited processing power of the Beagle, and to help meet the realtime constraints, the per-symbol FFTs are spread out over the entire sampling time. As opposed to wsprd which does everything at once (at the end) when presented with all the samples together in a file. So when the two minutes of sampling are done the Kiwi only has to run the decoding process on the previously computed FFTs of the symbols during the subsequent two minute interval (i.e. overlapped with the next sampling period). But to do the subtraction and a second pass would require all that processing to be done again during the two minute window and there just wouldn't be time for it in most cases.
We have to remember what the Kiwi extensions are. They are meant to be a very convenient, browser-based, built-in subset of fancier decoders that normally require separate programs with their associated hassles of virtual-audio-cables etc. They will never match the features or performance of existing decoders that have had much more effort put into them (and are closed-source and fee-based in many cases).
My Kiwi might have missed a few signals due to a busy CPU, but it was one of the few european WSPR receiving stations that heard your signal from Kaui tonight @rrobinet. Running 4 Kiwis 24/7 with 4 channels each in autoWSPR mode. Very stable and I am very pleased with it. Thanks for your input on the subject.
Thanks for the report. Yesterday was the first time in months that I had tried to tx WSPR from Maui and I thought no one was hearing me until your post. In addition to the ICOM-7300 sending WSPR, I have a Kiwi in Maui which autoposts on three rx channels and is open to the public on this fourth channel at http://kiwisdr.robinett.us:8073/. John's UI is vastly better than WSJT-x and autowspr is an amazing enhancement. I have only asked for the kiwirecorder.py as an alternative to auto-wspr because at another site ('KPH' here in California) I receive more signals on 20M than the Kiwi can process in the 2 minute cycle time. During the day on 20M the Kiwi feeding WSJT-x 1.9.1 would sometime post 30 signals when auto-wspr posted only 15 signals and I could watch the decoder run out of time while processing a -11 SNR signal. When we repair KPH's TCI-540 long distance antenna I expect we will be seeing even more signals. I have four Kiwis on order and when they are installed later this month you will see KPH posts 24/7 on every band 2200M...10M. Auto-wspr is probably fine for many of those bands, but 40M and 20M definitely need more CPU power than the Kiwi can supply
I have my daemon working except for posting to wsprnet and frequently get more sports than the Kiwi's auto-wspr. I have been unable to find a clear specification of the ALL_MEPT.TXT file and it no longer exists in WSJT-x 1.9.1 where it appears to have been replaced by ALL_WSPR.TXT I have attempted to duplicate the format of that file., but when I execute the curl command I get a "Tried to find "" and failed!" message:
+ curl -F allmept=/tmp/ALL_MEPT.TXT -F call=KPH2 -F grid=CM88mc http://wsprnet.org/meptspots.php .... Processing Log Upload Tried to find "" and failed!
=============
I found my bug: a missing '@' in allmept=@/tmp/...
My daemon is running and extracting a few more spots than auto-wspr even on a quiet band. However it appears that the audio stream recorded by kiwirecorder.py is compressed. In previous experiments I found that WSJT-x would extract even more spots when the Kiwi audio compression is turned off. So would it be possible to add a flag to record the uncompressed audio stream?
The latest jks-v0.1 branch has --ncomp (or --no_compression) to disable audio compression. Also a new optional --dir=path option to specify the directory to write the files.
Comments
The problem with this is that you're now waking up Linux to write the files and that will likely destroy the realtime environment. Especially with the WSPR extension which is running the cpu flat-out with all those FFTs and even more so with all 4 channels active. The fax extension seems to get away with it with its file recorder. But the processing requirements there are much lower.
My intention is to minimize the signal processing performed on the Kiwi by transferring the IQ or audio data produced (I assume) by the FPGA to a server with a much more powerful CPU, rather than have the Kiwi CPU process it.
As for file I/O:
In my products I have avoided uncontrolled CPU stalls in Disk IO operations by writing files to RAM disk partitions.
Alternatively, I would send the IQ/audio data over the LAN; it seems that the current auto-wspr code is able to do such transfers to wsprnet.org without disrupting Kiwi timing.
What would you suggest?
I would suggest just sticking with processing the existing audio IQ stream you can currently get over the net. You have the Python kiwiclient example showing how to do it. Many other languages have web socket libraries if you don't want to deal with Python.
As for why the Kiwi timing isn't disrupted by uploads to wsprnet.org: it's because the upload happens from javascript in the user's browser, not from C-code running on the Kiwi side. This is an example of the advantage of being able to split your extension code into C and javascript parts.
I thought I saw last fall that the RDF work was enabled by a 20 uSec accurate timestamp prepended to the IQ samples. If that is true, then how/where are those highly time accurate samples and where are stored?
But the .wav file stuff doesn't matter much as what you need to do is stream IQ data to the WSPR app in as much realtime as possible. I don't know anything about how WSPR accepts data in this case (and more importantly at what data rate). K9AN's WSPR code in the Kiwi WSPR extension needs the data at a sample rate of 375 Hz. So the 12 kHz Kiwi IQ audio is decimated by 32. But because the Kiwi sample rate is not exactly 12 kHz you may need to fill/remove samples at the two minute epoch using the timestamp to keep time-alignment. Or you may be able to restart the entire decode process at the two minute boundary which is what the Kiwi code does. But this would only work if the WSPR app doesn't require continuous samples to be supplied from one epoch to the next.
My cursory examination of the WSJT-x code on my PC uncovered a stand-alone wspr decode application that is delivered as part of the WSJT-x installation.
Since I've had WSJT-x hang after running for a day or two, it would seem best to avoid running the WSJT-x GUI and just feed the wav file to that wsprd app.
I would much prefer to run the whole process from a Python or bash script so it can recover from timeouts and other error conditions.
So I thought I would modify kiwirecorder to wait for the start of the 2 minute cycle and have it write the 110 seconds of wav data to a file. then run that file thorugh wsprd, then write a bash/python/.. script parse its output and post the results to wsprnet.org
There would be no change or new burden to the Kiwi, I would exploit all of the considerable work in the kiwirecorder.py, and exploit the current and future versions of wsprd. The result would be portable to any platform for which there is a WSJT-x distribution.
Does that seem a reasonable approach?
============
rob@Robs-MBP:/Users/rob> /Applications/wsjtx.app/Contents/MacOS/wsprd ; exit;
Usage: wsprd [options...] infile
infile must have suffix .wav or .c2
Options:
-a path to writeable data files, default="."
-B disable block demodulation - use single-symbol noncoherent demod
-c write .c2 file at the end of the first pass
-C maximum number of decoder cycles per bit, default 10000
-d deeper search. Slower, a few more decodes
-e x (x is transceiver dial frequency error in Hz)
-f x (x is transceiver dial frequency in MHz)
-H do not use (or update) the hash table
-J use the stack decoder instead of Fano decoder
-m decode wspr-15 .wav file
-q quick mode - doesn't dig deep for weak signals
-s single pass mode, no subtraction (same as original wsprd)
-v verbose mode (shows dupes)
-w wideband mode - decode signals within +/- 150 Hz of center
-z x (x is fano metric table bias, default is 0.45)
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
Deleting expired sessions...5 completed.
However I've run into some bumps in the road to using kiwirecorder.py:
1) It doesn't know how to supply a password and can't even access the one 'no password needed' channel on my Kiwis. When I have more Kiwi's later this month I can work around this, but it would be nice to support Kiwi passwords.
2) To record the WSPR packets I am waiting until the next 2 minute start and then spawning kwirecorder.py, waiting for 110 seconds and killing the process. this seems ugly, so it would be nice if there were a 'record for an optional X seconds and terminate' flag in kiwirecorder. I have yet to examine the output wav file, but hopefully my pkill of kiwirecord will leave behind a file with a fairly predictable start time and length.
3) In addition, kiwirecorder seems to record to a filename it autocreates. Those names are unique and I think I can extract them from the recorder's output, but it would be nice if there were a flag to pass the record filename. I can work around that one too, but the names suggest that the recording starts some seconds after kiwirecorder is started. Fixing that so the record starts at 2 minutes is of course essential for this scheme to work
wsprd accepts a wav file as its input, but I am next going to see if (probably how much) the recorder's output file needs to be massaged in order to be accepted by wsprd
All the kiwirecorder problems can be fixed. I'll look into it.
Also, from your earlier message, I tried the "no password needed" mode and it worked fine for me. If I had a password-protected Kiwi with one password-free channel allowed then I could make one connection as expected but not a subsequent one.
The k9an-wsprd code I used for the Kiwi wanted a mono .wav file having a sample rate of 12 kHz and a BFO of 1500 Hz. The routine wsprd.c::readwavfile() does an FFT and down-samples by 32 to get the required 375 Hz sample rate.
starting kiwirecorder.py by hand or by a script every two minutes is certainly possible, but it might be better to let kiwirecorder.py chop the wav files into two minute long files. I have just committed a small change to kiwirecorder.py for this: when the option --dt-sec=DT is used, a new file is started every DT seconds whenever sec_of_day%DT == 0.
Do get a capture file from the Kiwi I am executing:
/Users/rob/ham/kiwi/kiwiclient-jks-v0.1/kiwirecorder.py -s 10.11.11.72 -p 8073 -f 14.0956 -m usb -L 300 -H 2600 --pw=password --tlimit=110 --fn=/tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav
I am first trying to see how that wav file created by kwirecorder.py can be processed by wsprd. It outputs spots from the wspr_example.wav file supplied in the wsjt-x source, but wsprd immediately and silently returns when supplied the kwiwrecorder capture file. ffprobe shows the only difference in the wav file is the (I think) sample rate of the good wav file is 12000 Hz while kiwirecorders's wav file reports 12001 Hz sample rate. Before I start adding diagnostics to wsprd or blindly processing the kwirecorder wav file, could the 12001 Hz be a rounding error in kiwirecorder? The similar wav file sizes suggest that sample rates are pretty close.
rob@Robs-MBP:/Users/rob/ham/kiwi> ls -l /tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav wspr_example.wav
-rw-r--r-- 1 rob wheel 2646060 Jun 8 07:28 /tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav
-rw-r--r-- 1 rob staff 2880044 Jun 7 21:58 wspr_example.wav
rob@Robs-MBP:/Users/rob/ham/kiwi> ./wsprd /tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav
rob@Robs-MBP:/Users/rob/ham/kiwi> ./wsprd wspr_example.wav
mple -9 1.1 0.001446 0 ND6P DM04 30
mple -15 0.1 0.001460 0 W5BIT EL09 17
mple -24 2.2 0.001465 0 G8VDQ IO91 37
mple -6 0.7 0.001489 0 WD4LHT EL89 30
mple -1 -0.8 0.001503 0 NM7J DM26 30
mple -21 0.5 0.001517 0 KI7CI DM09 37
mple -18 -2.0 0.001530 0 DJ6OL JO52 37
mple -11 0.8 0.001587 0 W3HH EL89 30
mple -25 0.7 0.001594 0 W3BI FN20 30
rob@Robs-MBP:/Users/rob/ham/kiwi> ffprobe wspr_example.wav
ffprobe version 3.3.4 Copyright (c) 2007-2017 the FFmpeg developers
Input #0, wav, from 'wspr_example.wav':
Duration: 00:02:00.00, bitrate: 192 kb/s
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 12000 Hz, 1 channels, s16, 192 kb/s
rob@Robs-MBP:/Users/rob/ham/kiwi> ffprobe /tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav
ffprobe version 3.3.4 Copyright (c) 2007-2017 the FFmpeg developers
built with Apple LLVM version 8.1.0 (clang-802.0.42)
Input #0, wav, from '/tmp/kiwi_capture_10_11_11_72_20_8073_14_0956_2018_06_08_07_26.wav':
Duration: 00:01:50.24, bitrate: 192 kb/s
Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 12001 Hz, 1 channels, s16, 192 kb/s
rob@Robs-MBP:/Users/rob/ham/kiwi>
The 12001 business is indeed round-off error in the sample rate recorded in the /wav header. Remember that the Kiwi does not have a sampling rate of exactly 12 kHz due to the ADC clock frequency and correction by the GPS. But my 2015 version of wsprd ignores the .wav file header. You should check your wsprd and see if it rejects the file if the header sample rate is not exactly 12000. The file sizes will differ mostly because of small differences in the exact recording time. Even at only 12 kHz the files grow quite quickly.
After you have it working you might want to try an A/B comparison between your wide 300 - 2600 Hz passband and narrowing it down to just the required 300 Hz (i.e. 1500 +/- 150). My version of wsprd uses an FFT to do the 32:1 decimation, so the filtering is essentially built-in. But I'm not sure what you've got.
> ../kiwiclient/kiwirecorder.py -s sielsdr.ddns.net -p 8073 -m usb -f 3592.6 -L 1400 -H 1600 --station=DF0KL --dt-sec 120
/Applications/wsjtx.app/Contents/MacOS/wsprd 20180609T044800Z_3592600_DF0KL_usb.wav
_usb 1 0.3 0.001401 0 DL0PBS JO33 17
_usb -1 0.4 0.001514 0 G0FTD JO01 30
_usb -8 0.4 0.001517 0 OZ8WSP JO57 27
The same spots, except OZ8WSP, were shown in the KiwiSDR window in the web browser.
I am not an expert in WSPR receiving but it seems you have to set the rx on the frequencies listed here
http://www.g4ilo.com/wspr.html and as John has mentioned the signals are expected to be between 1400 and 1600 Hz in your recorded files.
In wsprd there is the option
'-s single pass mode, no subtraction (same as original wsprd)'
With this option I get the same decodes as from the KiwiSDR. So one possibility is that the differences between wsprd and KiwiSDR are not due to limited computing power on the KiwiSDR but due to the fact that wsprd is using improved decoding algorithms.
there is a frequency argument to wsprd which should add that value to the offsets seen in your report.
My primary motivation for this is that I have found that the wspr decoder included in the kiwi cannot decode all of the signals recorded on HF at KPH here in California. During the day on 20meters, the antenna systems there are so good that the Kiwi runs out of time before finishing its wspr decode.
I'll report back once I script the decoding and posting the results to wsprnet.org
> My primary motivation for this is that I have found that the wspr decoder included in the kiwi cannot decode all of the signals recorded on HF at KPH here in California. During the day on 20meters, the antenna systems there are so good that the Kiwi runs out of time before finishing its wspr decode.
Do you still see differences when you use the option "-s" with wsprd?
One more thing: you should use the option "-T -100" with kiwirecorder.py. Otherwise the squelch can interfere with data taking and start a new wav file inside a 2min window.
In addition to the problem decoding all of the spots on a busy band, the decoder in the Kiwi is somewhat old and the wsjt-x 19.1 release notes says the wspr decoding performance has been improved. For example even in a band with only 2 signals, in my command line test just now, wsprd picked out the -18 SNR signal of WA4SZE on 630M which was missed by the Kiwi auto-wspr:
rob@Robs-MacBook-Air-2:/Users/rob/ham/kiwi> /Applications/wsjtx.app/Contents/MacOS/wsprd 20180609T063400Z_474200_AI6VN_usb.wav
_usb -18 0.4 0.001452 0 WA4SZE EM65 23
_usb -6 1.0 0.001564 0 W0YSE CN85 30
rob@Robs-MacBook-Air-2:/Users/rob/ham/kiwi>
The Kiwi version doesn't do the subtraction and second pass because of how it implements the wsprd algorithm. To more efficiently use the limited processing power of the Beagle, and to help meet the realtime constraints, the per-symbol FFTs are spread out over the entire sampling time. As opposed to wsprd which does everything at once (at the end) when presented with all the samples together in a file. So when the two minutes of sampling are done the Kiwi only has to run the decoding process on the previously computed FFTs of the symbols during the subsequent two minute interval (i.e. overlapped with the next sampling period). But to do the subtraction and a second pass would require all that processing to be done again during the two minute window and there just wouldn't be time for it in most cases.
We have to remember what the Kiwi extensions are. They are meant to be a very convenient, browser-based, built-in subset of fancier decoders that normally require separate programs with their associated hassles of virtual-audio-cables etc. They will never match the features or performance of existing decoders that have had much more effort put into them (and are closed-source and fee-based in many cases).
I have only asked for the kiwirecorder.py as an alternative to auto-wspr because at another site ('KPH' here in California) I receive more signals on 20M than the Kiwi can process in the 2 minute cycle time. During the day on 20M the Kiwi feeding WSJT-x 1.9.1 would sometime post 30 signals when auto-wspr posted only 15 signals and I could watch the decoder run out of time while processing a -11 SNR signal. When we repair KPH's TCI-540 long distance antenna I expect we will be seeing even more signals. I have four Kiwis on order and when they are installed later this month you will see KPH posts 24/7 on every band 2200M...10M. Auto-wspr is probably fine for many of those bands, but 40M and 20M definitely need more CPU power than the Kiwi can supply
I have been unable to find a clear specification of the ALL_MEPT.TXT file and it no longer exists in WSJT-x 1.9.1 where it appears to have been replaced by ALL_WSPR.TXT
I have attempted to duplicate the format of that file., but when I execute the curl command I get a "Tried to find "" and failed!" message:
+ curl -F allmept=/tmp/ALL_MEPT.TXT -F call=KPH2 -F grid=CM88mc http://wsprnet.org/meptspots.php
....
Processing Log Upload
Tried to find "" and failed!
=============
I found my bug: a missing '@' in allmept=@/tmp/...
However it appears that the audio stream recorded by kiwirecorder.py is compressed. In previous experiments I found that WSJT-x would extract even more spots when the Kiwi audio compression is turned off. So would it be possible to add a flag to record the uncompressed audio stream?