How to use kiwirecorder to record waterfall history
I would like to study the performance of the antenna at KPH and to do so I would like to record the 0-30 MHz spectrum over 1+ hours while changes are made to the antenna and RF transmission system.
It seems that kiwirecorder.py -wf -z0
might do that, but I get no console or file output when I run that command.
Can you suggest how I could achieve my goal with the Kiwi + kiwirecorder?
"kiwirecorder -wf" only reports max/min bin values per 1 Hz waterfall update. So it's more for reporting statistics.
"kiwiwfrecorder" creates a file of actual waterfall data. But it's in .npy (numpy, binary) format that needs to be post-processed.
The "kiwifax" code is able to create arbitrarily large .png image files. So with some work creating .png files could presumably be added to kiwirecorder and/or kiwiwfrecorder.
I just pushed some changes to https://github.com/jks-prv/kiwiclient that makes "kiwirecorder --wf" work a lot better. Particularly for the non-zero zoom case. See the Makefile for usage examples.
There has been an open issue for almost a year about the lack of image file generation: https://github.com/jks-prv/kiwiclient/issues/77 Also necessary would be timestamps on the image so you could correlate it to antenna/propagation changes etc.
I've just started to look at how the existing Kiwi right-button menu item "save waterfall as JPG" works (contributed feature). It saves more waterfall history than I thought including stuff that has past beyond the bottom of the scrollable screen area.
If you set the waterfall update rate to 1 Hz this could be a very substantial period of sampling time. It's also trivial to add timestamps (the IBP scanner already has code to annotate the waterfall).
One missing piece is a bug that has been pointed out before: If the waterfall is set to update at 1 Hz then there should be an option for it to be averaging at full rate during that time so what's displayed every 1 Hz is not just a drop sample.
Okay, I have a partial fix for this. Go to the admin page update and click the "force update" button. Then you can monitor the build progress by going to the console tab, clicking the connect button and typing the "m blog" command. Takes about 15 minutes to rebuild.
After the restart you can use the waterfall timestamp "wfts" URL parameter. So "my_kiwi:8073/?wf=1&wfts=60" will cause the wf to update at 1 Hz (wf=1) and timestamp every minute (wfts=60). The range of wfts is 2 seconds to one hour (2..3600).
Then use the right-button menu "save wf as jpg" to save. Despite the shorter scrollable waterfall on the screen that saved wf image is about 4600 lines. At 1 Hz that's about 76 minutes. Obviously less if you have the waterfall updating at e.g. "slow".
Remember that if you change anything about the waterfall (zoom, pan, etc.) the history and timestamps may/will be erased. So set everything up, including the colormap and max/min dB you need, and test the jpg save process. With Firefox I could set the response to the jpg save to either save as a file or open an image viewing program.
Medium speed waterfall with 10 second timestamps:
How does latency affect the timestamp?
Hi @jks! If possible add frequencies line to jpg file with waterfall on the top and bottom.
Wow John, that is all I wanted in this feature.
I don't appear to be getting 'new post' emails from the forum, so I just noticed your response and efforts.
Jim: I hadn't considered that. Was just trying to get something done fast for Rob.
Yuri: The right-click "save waterfall as JPG" adds full date/time, frequency and hostname at the top of the image file. Although the frequency used seems a little off. I will look at this.
Rob: For my own account I had recently re-enabled sending forum email notifications. And it worked. So maybe try disabling and re-enabling and see what happens. This forum code has always been a little wonky unfortunately.
adding a way to add offset at the client end would be fully adequate I think. I measured it here and in my local system, it is 2.16 secs.
@jks sorry, I wrote about frequency bar:
Yuri: Oh, I see. Yes, that is a very good idea. I will try and do that.
It might also be nice, if the spectrum is open, to capture it at the top of the file. Even though this would only represent the top line/sample of the waterfall. The spectrum scale shows dBm values which would help correlate colors in the waterfall to actual signal strength.
In my view, the spectrum is much more valuable than the wf for a lot of measurements.
Not trying to add work but if I could, many times I'd turn off WF and SWL labels and have only spectrum. There's so much information there that gets lost in color representation.
This is not a bug report or a criticism, but an observation which may be relevant to this thread.
Be aware that if you are using the spectrum or waterfall detail, it will only provide limited amplitude resolution with dips in the response, which I assume is due to the sample rate and the size of the FFT bins.
For a fixed KiWi zoom setting the 'dips' remain at constant frequencies, and can be observed when almost fully zoomed in, but the 'dips' are then only a few Hz apart.
This does not seem to be a problem when observing modulated signals, but it is noticeable if you are performing CW RF sweeps to measure antennas and distribution systems, which how I got caught out by it.
Here's one example of a fully zoomed out image of the KiWi showing the spectrum and waterfall, when fed with a 60 second duration 1-30MHz stepped sweep at -40dBm.
Note that the dips and peaks of the waterfall and spectrum displays differ from each other.
Changing the sweep time does not alter the position of the 'dips', but altering any KiWi display settings will do. You can see the start of a second sweep, which is identical to the first, even though the KiWi and sweep source are not synchronised (just a very slow sweep).
Using a signal generator and manually stepping the frequency in small increments can also be seen to produce dips of up to 30dB, which remain when the frequency is held constant.
The S-Meter and dB scale read correctly at all times, as previously stated.
Hi Martin. Good to hear from you.
It could be that for testing with very narrowband carriers like this the pre-FFT windowing needs to be turned off (see: https://en.wikipedia.org/wiki/Spectral_leakage). The next release includes a new waterfall extension with space where some window function choices could be included. There could also be a URL parameter for this. Let me try and get these added for v1.462
There were a couple of recent, related threads here:
Okay, so I have a partial solution for this. It has nothing to do with FFT windowing, but rather with this FFT bin interpolation business we had trouble with a few months back.
If you're sweeping with narrowband carriers add the URL parameter "wfi=0", e.g. "my_kiwi:8073/?wfi=0"
Below are some spectrums made using the Kiwi's signal generator extension. The first without any "wfi" parameter, sweeping 10k - 30M, stepping every 100 kHz. It looks terrible, just like what you're seeing Martin. The second made with "wfi=0". Much better. I know this doesn't address all of your observations, but it's a start.
If you're using the sig gen extension note that it only works when you're connected on rx channel 0. And that the SFDR is only about 90 dB. So there are spurs when the output level is low (attenuation high).
Thanks for the information and links.
I suspected something along the lines you mentioned, but I'm not overly familiar with FFT's, or the correct terminology for what I'd been observing.
Reading the previous threads, it all now makes sense.
It's not a problem, just something to be wary of in certain situations, and definitely very much an edge case.
Just tried wfi=0
Big improvement, which is especially noticeable on the 'Data over Powerline' interference bursts around 20, 23 & 26MHz on my home KiWi.
The spectral components I thought I'd been observing in the interference, were in fact artefacts of the Spectral Leakage.
All averaging turned off in the following plots. Peak hold on the spectrum display was active for about 30 seconds before I took the screen grabs.
Although the resolution of the waterfall image is very slightly 'blurred' when wfi=0 is used, the overall effect, especially when fully zoomed out is impressive.
Here's two more grabs, this time using EMA avg =4 (my favourite setting).
For wfi=0 the wf min setting is 5dB higher so that the noise floor colours are about the same as each other.
Note how the data burst as around 3.5 Mhz is more apparent, also the OTHR systems at 16 & 18MHzand the twin ionospheric sounders sweeping around 22 to 23MHz. The switched mode interference around 20MHz is also more evident.
I think this improves upon the waterfall filtering issue that was discussed way back when in this thread.