v1.443: automatic SNR measurement, queue/camp panel reload button, misc fixes
From the CHANGE_LOG file:
v1.443 March 22, 2021
Added automatic SNR measurement:
Every 6 hours (4 times daily) a Kiwi will connect to a free channel (if available)
and measure its own SNR. These results are available in JSON format from the
URL: my_kiwi:8073/snr (adjust Kiwi name and port number as required).
This URL is publicly accesible. Disable on the admin page, control tab if you
don't want to share this information.
The measurement details can be easily changed. Suggestions welcome.
SNR is currently defined in the same way as used by the sites snr.kiwisdr.com
and snr2.kiwisdr.com, i.e. waterfall dBm values, sorted, noise = 50th percentile
value (median), signal = 95th percentile value, SNR = signal - noise.
Two measurement bands are recorded: 0-30 MHz and 1.8-30 MHz in an attempt to
filter out the AM BCB. The JSON data also contains min/max dBm values and a
timestamp (local time unless the "UTC" value is set).
Added "reload page" button to queueing panel ("camp" parameter is removed from URL)
Use to quickly access any Kiwi redirection that might be available.
Autoscale button and aperture auto mode now disregard masked areas from calculations.
Admin page UI changes:
GPS RSSI now red color for sats not actually acquired (i.e. no subframe decoding)
Registration now requires a valid location field (for benefit of map.kiwisdr.com)
Comments
Got the following from one of my kiwis;
[{"ts":"Mon Mar 22 04:48:31 2021","utc":0,"ihr":6,"d":[{"lo":0,"hi":30000,"min":-97,"max":-45,"p50":-84,"p95":-72,"snr":12},{"lo":1800,"hi":30000,"min":-97,"max":-66,"p50":-84,"p95":-77,"snr":7}]}]I think the setup is fine for a general idea of snr and cutoff at 1800 is logical, even here in LA where there are no MW transmitters, they influence some db on the data. (PS the above dataset is on a very noisy active loop placed indoor, picks up all my house rfi).
I found it useful to see a timegraph of noise, but more detailed analysis would probably be better done with dedicated software and not within the kiwi.
Will only the last measurement be shown or will the kiwi store the values and return the timehistory?
73 de Olaf - LA3RK
Sorry, I neglected to mention that. A day's worth of rolling data is kept. So after 24 hrs there should be 4 entries spaced at roughly 6 hour intervals (the "ihr" value).
The intent is eventually as a data source to drive the SNR values shown on the linkfanel.net sites.
Now make compare easy could add many bands like http://sibamanna.duckdns.org/sdr_map.html and day night beacon map http://www.dxatlas.com/Faros/ keep making good happy
This website will load the URL and display it in an easy to read format. https://jsonbeautifier.org/ . It looks like the time zone is the KiwiSDR local time zone? If that is the case, then the user of this data will need to consider the time difference from the user location and the KiwiSDR location, to know the optimum performance times to use a given KiwiSDR. Or something like that.. If this data rotates every 4 six hour reporting cycles, then a casual user won't be able to understand the most effective time of day to use the KiwiSDR as the data will be past and the present data is not yet available and yesterday's data is not available to interpolate the future effectiveness that day. I suppose an intrepid user could setup a cron job to do something like:
curl "ka7u.no-ip.org:8073/snr" >> 8073get.json
and gather the information, but then how long will the information last after the 4th report of the reporting period? My report #3 "ts": "Mon Mar 22 15:57:36 2021", indicates the 4th report will be at 21:57:37? So is it safe to assume a 6 hour window to download the 4 six hour reports before they are overwritten?
Just curious.
Ron -- KA7U
By looking at the full spectrum as the code does, it will not show the typical difference between day and night. The snr for the low frequencies seem to dominate in my setup. At my location the snr below 10-14 MHz is completely out of phase with the snr for the spectrum above when these two parts are separated. There are also some interesting snr peaks around sun up / sun down. But to see such effects, measurements needs to be taken more frequently and also split in sections of the spectrum.
Olaf - LA3RK
About changing the SNR measurement details and referring to LA3RK's post about diurnal effects.
Is there a place where the administrator can change "utc":0,"ihr":6 ?
For longer time monitoring hourly SNR snapshots would be useful, if the rolling data history retained on the Kiwi nodes can be expanded to say cover one week of data. This was also the time frame Marco -IS0KYB previously used for his quite well working Dynamic SNR map.
Thanks, Ben
This new SNR feature really does not give me any real value using my Kiwis but a SNR value of the signal I am listening would really help. For example when comparing antennas. Is that very hard to code? But all in all very nice updates just keep coming 😊. Thank you for that.
Best 73´s
Pekka
You should be using the spectrum display for something like that. Code can't reliably determine the noise floor outside of the passband to give you a narrowband SNR figure.
These changes were prompted by many people asking that rx.linkfanel.net marker SNR colors and SNR rankings at rx.linkfanel.net/snr.html be updated more frequently by shifting the measurement work to the Kiwis themselves. The long-running discussion of the issue is here: https://github.com/priyom/dyatlov/issues/1
No signal 2130 2544 11180 23000 much noise bad many test choose good best way?
I've updated http://rx.linkfanel.net/snr.html to display the new SNR scores self-reported by KiwiSDR receivers. For now I've also kept the classic scoring. This way we can start seeing what we're talking about. It looks to me like nighttime makes a huge difference, it's a bit surprisin. Comments welcome.
v1.449 April 3, 2021
SNR measurement adjustments on admin page control tab:
Adjust recording interval: 1, 4, 6 or 24 hours per measurement.
One week's worth of data is retained in the rolling buffer.
New JSON field "seq" (incrementing) to help identify most recent entry.
In addition to existing frequency measurement intervals 0-1800 and 1800-max kHz
added 0-1800, 1800-10000 10000-20000 and 20000-max
These are similar to what Marco uses (sibamanna.duckdns.org/sdr_map).
Selectable local/UTC timestamp.
Most excellent work how show graph see changes
Retaining a week's worth of SNR data is a great addition, since each individual reading fluctuates so much, thanks!
I wonder if there is merit in perhaps including the average of all of those readings on the status page, rather than just the most recent reading, since the average may provide a better overall SNR value for a particular receiver?
@ChrisSmolinski I think for a user, the most current SNR would be helpful in choosing a receiver for a particular region, especially if it showed up in the "mouse hover info window" on the http://rx.linkfanel.net/ map. The immediate value would be the most useful in selecting what to listen to at that hourly moment in time. Of course updating the map for all KiwiSDR receivers on an hourly basis might be a bit much for the site manager to take on.
__________________________________________________________________________...
Getting a status report from (kiwisdr...:xxxx/status) is probably the best solution and with a little more programming, the average and the last hour could be displayed, I suppose. I'm not real sure what to do with this data. My radio time is mostly fixed in the morning and the evening, a few times a week.
So what is the vision for this data? To rate the reception effectiveness of receivers? Determine frequency reception over time? To compare antenna systems at a given location or geographic region? Or whatever a user might imagine, I suppose. But specifically, is there an organized effort underway to analyze the KiwiSDRs as a group or as a region?
Not to complain, but it is true that different receivers have issues that negate what appears to be a good SNR report. So unless a user actually uses that receiver, the effectiveness of the receiver is not really known. Issues that plague receivers are not always constant, but intermittent and random radio environment events.
Another issue is the overuse of a given receiver, which I think is somewhat the result of ratings on http://rx.linkfanel.net/snr.html , The higher up the list, the more traffic that arrives. I have experienced a decline in connections as the rating of my receivers has dropped, with the result of more return users that are mostly monitoring local and regional nets. When I think about it, this is probably the best and highest use of the receivers, given my Ham radio enthusiasm. 😎
Sorry about the ramble.. Just had to chime in.
Ron - KA7U
Some best SNR have much overload signal with level in one band need less strong signals by filter maybe tell with numbers add message
To @linkfanel 's comment:
After an areas local sunset, many shortwave stations will move to lower frequencies where propagation support is available to their target area, and so you end up with a lot of very strong shortwave stations which are often fairly close (relatively) to the listener. This, combined with the D-layer dissipating, means the overall signal levels observed by a receiver (in the area of those stations) rises.
In my stations case, right after sunset I begin to see the observed SNR on my KiwiSDRs (VK5ARG #1 and #2) rise significantly, and often the overall signal levels received by the KiwiSDR get so high that I get close to overload (hence the other thread and issue raised about getting access to the A/D overhead level, to investigate this further). The signals that are causing this are not MW broadcast stations (As I have a high-pass filter), but shortwave stations between 4 MHz and ~12 MHz from the Asian region to my north.
For a station that is not being limited by local noise, I would expect the SNR as calculated by the metrics used now to swing considerably between day and night-time, which is exactly what we see.
I've been taking 5-minute measurements of SNR on my stations for a few weeks now, and the repeated patterns of low-SNR in daytime, with SNR rising after local sunset (~8-9Z in these plots) is very obvious. In these plots the spikes in the median level observed at the same time as the SNR starts to rise is likely from the KiwiSDR overloading when there is enhanced propagation during greyline. After greyline it settles down again. Being able to look at the overall A/D overhead level (a metric of total signal entering the A/D, and how close the receiver is getting to overload) would be useful in confirming this.
73 Mark VK5QI
Just as a bit more information on this one, here's a spectrogram plot from one of the KiwiSDRs over a few days, and also the estimated total RX power (just from summing all the bins). The scales are not quite matching, still some work to do there.
I'm not sure what the clipping point of the KiwiSDR is, or how accurate the RX Power estimate I'm doing is.
Anyway, the day/night cycles are clearly visible, and you can see those large red bands of shortwave stations that are causing me problems at some times!
EDIT: The code to capture the required data and generate the above plots is now on github here: https://github.com/darksidelemm/kiwisnr-rrd
73 Mark VK5QI
Like spectrogram great could study if add new snr to url for see quickly best when change antenna
About the purpose of the scores, and what score to use, I think there are several use cases, or maybe several schools of thought.
Of course, the SNR listing, the map markers and so on are different tools with different purposes, they don't have to use the same approach if several scores are available; this isn't one size fits all. I guess there's no problem displaying all available score options in the listing; they just have to be available. For the map, I think I have to choose what I do, but I'm listening.
The immediate value would be the most useful in selecting what to listen to at that hourly moment in time. Of course updating the map for all KiwiSDR receivers on an hourly basis might be a bit much for the site manager to take on.
That's no technical issue for me, it really makes no difference with the current state in that regard.
KiWi problem press manual autoscale and few strong signals stay above top of scale not lower down as should be maybe is not right
It looks as though whenever a KiwiSDR gets updated, the SNR history is cleared. Any way to preserve this information across updates? It might be helpful for viewing long term trends/patterns. Thanks!
I have just switched the markers on the map to use the new self-reported internal SNR scores. The logic is as follows: the HF-only self-reported score is used if available, if not the full-spectrum self-reported score is used instead, and if no self-reported score is available, the classic external score is used as fallback; this last case is mostly about receivers running an older version of the software.
Users choosing a receiver will benefit from up-to-date and instantly relevant scores, as by default self-reported scores are calculated every hour. As such you may notice how marker coloration now brightens up when nighttime comes and covers their area. HF-only scores are favored as it was pointed out that very strong MW bands tend to skew the scores. I don't especially like excluding signals below 1800 kHz, but a choice has to be made since it makes a significant difference in usefulness; and this is a shortwave receiver map, not just a map of remotes for commercial MW band broadcasts, so this is our editorial line.
Thanks to everyone involved for their contributions to the discussion, scoring code or monitoring framework. In the future I'm fine accepting as input whatever KiwiSDR receivers calculate and report as SNR score, so don't let the fact that the map uses those scores hinder further change or refinement about them :)
Question
Should local SNR testing be classified as a non Kiwi connection?
I had a 8 channel set to 7 for non-Kiwi FT8 while last channel empty the SNR test failed due to no free channel.
Just an observation, not sure if it should or not, or even if I was mistaken.
VK5QI created some good looking rrd plots as shown in his posts above. Also linkfanel's page with signal-to-noise ratio based scores is useful, but in order to visualize how these SNR values change over time and location I have been working on creating SNR plots in Python for single and multiple Kiwi stations. Now using jks recent addition of up to a week of hourly SNR data this has become a lot easier and quicker than previously where waterfall data had to be retrieved and processed for each Kiwi node.
The multi station plots are for comparison of hourly data. Kiwi nodes have been selected for 4 longitude bands trying to get some reasonable geographic coverage and the list has been limited to 20 of the subjectively "better" stations.
Please note that sometimes the SNR json files cannot be retrieved due internet connectivity time-outs and at times will possibly have missing hourly data points when all channels are in use. In addition the individual Kiwis can be offline for quite long periods as well.These plots are generated with 'bokeh', a versatile Python library for interactive visualizations that produces javascript for use in html web pages .
The plots spanning a whole week do look crowded, but by clicking on the legends any or all bands can be turned off. On the top right corner you will find some tools including (wheel) zoom which make it easy to navigate the plots.
Currently running as a trial on a RP3, you can check out the alpha version on http://sparks.epizy.com/SNR/MultiSNR_plots.html
A pdf screenshot is attached.
Note : on many browser F5 refreshes the browser cache.
The diurnal changes in SNR scores are interesting to observe, below is the KiwiSDR with a 400 ft south directed Beverage antenna. Somewhat different results on the other KiwiSDRs with different antennas, as expected the 11/10m dipole is pretty dismal by comparison 😀
20000 to 30000 good sign of HF working condition better past days
Question for friends is possible to output string full spectrum FFT bins for zoom with URL command when asked Help make external picture
It looks like the 14 channel radios don’t report SNR. I don’t see a mention in the release notes.
Steve KD2OM
nothing in the json file buffer either