Direct access to S Meter and Spectrum data
I am intrigued by the Kiwi SDRs and considering buying one.
What I'd really like to do is analyze the S-meter plot as a digital time series, i.e., I want to convert the plot into a sequence of numbers (I estimate there are 42 per second for some idiosyncratic reason). Likewise with the spectrum plot: I'd like to get access to the entire set of FFT levels as a vector time series. In each case, I would want to convert the graphical displays into TXT files (or similar like PRN) and export them to my laptop computer for analysis.
Is there a way to do this in a plain vanilla Kiwi, just as a built-in feature? I have no equipment to modify the unit, so I'm hoping it's just a matter of "Oh, plug into the jack on the back and ...".
Cheers,
TRW
Comments
The Python program kiwirecorder (and kiwiwfrecorder) from the repo https://github.com/jks-prv/kiwiclient is a good starting point. It can connect to a Kiwi and sample waterfall and S-meter data.
You'll probably have to modify the Python code slightly to process the data exactly how you want.
jks, thank you for the pointer.
TRW
I have been enjoying my Kiwi2 for some months, though I still have problems with an antenna that will fit in my apartment. But listening to all the others has been great and has helped my research along. To advance the research I need to get data files out of the Kiwi. You previously suggested a way to do this, but I personally find your answer to my question on February 5 above difficult to operationalize as I am not much of a coder and do not know Python. But I'm trying to learn about data extraction in particular but also how the Kiwi processes the HF signals in general. I have assembled a list of questions to ask you (below). Perhaps you could answer with simple "true" or "false" and for more-open ended ones give a short response. Regards, Tom (ex K1VBJ).
1. The S-Meter records the sum of the discrete RF spectrum values over the passband.
2. The passband width depends on the chosen modulation mode, e.g., CW vs CWN vs AMW, etc.
3. The time between the plotted points in the S-Meter display varies. (I manually counted a range of about 42 per second to 47 per second.)
4. Why is the time between plotted points allowed to vary?
5. The waterfall plot is a more direct way to plot the RF spectrum.
6. The bandwidth of the spectral waterfall plot is governed by the chosen size (i.e., the zoom level) of the display window.
7. What is the time between successive “lines” of the waterfall plot?
8. Is this timing constant?
9. It is possible to use the CURL command to manually sample the S-Meter and spectrum by specifying a time in the former case and a time and a frequency in the latter case.
10. What would induce you to make it possible to directly access the time series of S-Meter and Waterfall Plot values as, say, CSV files, rather than struggling with recorder and Python?
Well.. There needs to be a more detailed discussion here as there may be some misconceptions about what the KiwiSDR is and what its capabilities are.
The timing repeatability of the measurement you want are a little difficult to achieve given the "best effort" nature of some of the Kiwi functions. This is due to the Kiwi design. Remember that the Kiwi goal is to provide a total solution in a single box, including what is usually the "PC" side of most SDRs that consist of an SDR "IQ-generator" device plus PC with SDR software running on it. And the PC has vastly superior processing power to the Kiwi's single board computer (BeagleBone Green). BUT at the dollar cost of the PC (which most people incorrectly discount) and the often terrible burden of software installation problems (you see this even now in the support forums of the IQ-generator products -- very sad).
So with its design the Kiwi has to give highest priority to processing the audio and guarantee no audio drops (except when the network connection is poor -- something the Kiwi cannot control). Everything else has lower priority, especially waterfall processing. And there are lots of other things going on simultaneously: GPS, user interface, possibly processing to support an open extension etc.
Consider the case of the wsprdaemon group (https://groups.io/g/wsprdaemon/topics). For a long time they asked for custom modification to the Kiwi. And I provided them, best I could. And they bought a number of Kiwi. Fair enough. But in the end their ever-growing requirements forced them to switch to a more traditional IQ-based SDR with a standard PC for the increased processing power.
Some technical points we should discuss. The audio and waterfall output each use two completely independent, tunable, digital downconverters (DDCs). The S-meter values are derived from the audio channel and influenced by the passband. It is not derived from passband-limited waterfall bins. Not many people understand or use this feature but it is entirely possible to tune the waterfall far away from where the audio channel is listening. The audio could be listening to 7020 cw and enter "#28200" in the frequency field to check for 10m beacon activity -- without disturbing the 80m audio (enter "#" to re-align the waterfall to the audio).
Now to your questions:
1. The S-Meter records the sum of the discrete RF spectrum values over the passband.
No. See above. It's just a
2. The passband width depends on the chosen modulation mode, e.g., CW vs CWN vs AMW, etc.
No, the passband is not fixed based on mode. The initial passband values are set based on mode. But you can change them to be whatever you want afterwards. And those new values will persist when you return to that mode (I'm talking about the Kiwi user interface here).
3. The time between the plotted points in the S-Meter display varies. (I manually counted a range of about 42 per second to 47 per second.)
Are you talking about the S-meter extension? (graph at the top of the display when active). You must be. Did you really count display pixels between the vertical marker lines to determine this?
4. Why is the time between plotted points allowed to vary?
I can't remember. That extension was written many years ago. Probably because there was no buffering of the S-meter values. Remember that the S-meter extension runs on a best-effort basis. And also involves a Javascript component that runs in the browser. So yet another source of timing uncertainty. The actual S-meter values are prepended to each audio packet sent from the Kiwi. So occur at a regular rate. But there can be many reasons why they are not processed at a regular rate.
5. The waterfall plot is a more direct way to plot the RF spectrum.
Yes.
6. The bandwidth of the spectral waterfall plot is governed by the chosen size (i.e., the zoom level) of the display window.
Yes. 30 MHz divided by power of two: 15, 7.5 MHz etc. The user control panel "stat" tab shows the spectrum span value.
7. What is the time between successive “lines” of the waterfall plot?
This varies considerably due to the low priority of the waterfall task. And also the "acquisition time" determined on the zoom factor. And there is a "WF Rate" control on the WF tab that sets relative update time as well. Just like a real spectrum analyzer, high zoom factors take considerable time for the DDC to accumulate all the necessary samples. At z0 it happens in a flash. At z14 it takes forever.
8. Is this timing constant?
No. See above.
9. It is possible to use the CURL command to manually sample the S-Meter and spectrum by specifying a time in the former case and a time and a frequency in the latter case.
Yes for the S-meter, no for the waterfall (but it would be relatively easy to add and is maybe the easiest solution to your measurement problem). The various URL-based queries that can also be used via curl are listed here: http://kiwisdr.com/info/#id-urls
10. What would induce you to make it possible to directly access the time series of S-Meter and Waterfall Plot values as, say, CSV files, rather than struggling with recorder and Python?
See #9 answer. I'm more likely to add another query URL to return a sample of waterfall data. Most of the code to do so has already been written for the SNR measurement function since it is based on averaged waterfall bin data.
I'm in the middle of working on new products and upgrades and don't have much time for anything else. I'm having trouble keeping up with my emails as it is..
Dear JKS:
Thanks for taking the time to answer perhaps the longest question in forum history. I understand and appreciate your answers. I had hoped to rely completely on the Kiwi for what is essentially a science project involving the monitoring of lighting around the globe. As is, the Kiwi does part of the job, but your answers suggest it won't offer sufficiently precise timing estimates for all my needs. Regardless of my project's issues, I applaud your design and the job you are doing to support the system.
TRW
Well, now hold on. You should have mentioned your exact application at the beginning.
Something the Kiwi does do is allow precise GPS timestamping of the audio stream. This is how the TDoA direction finding application is able to work. Now post-processing the audio, extracting the timestamps and synchronizing the audio samples from geographically dispersed Kiwi is not a trivial task. But it is essentially what TDoA does.
So if your goal is to measure time-of-flight of LF/VLF static crashes (a la Blitzortung.org) then there might be some hope. But it will take some moderately complex programming.