wsprdaemon noise graphs
This discussion was created from comments split from: wsprdaemon version 2.3 (latest 2.5a) - A Raspberry Pi WSPR decoding service.
It looks like you're new here. If you want to get involved, click one of these buttons!
Comments
Hi Rob, many thanks for the great work and support for the kiwi-hardware!
I am trying to get the WD 2.5a (latest) work with my single kiwi ... on my windows 10 , I installed VMware Workstation player with fresh installed Ubuntu 18.04lts and all other necessary Linux stuff...
started the watchdog as documented in the helpfile (-h) with extension -a
However: Cannot get the png-files get written...
What the kiwi-rx receives got decoded and posted correctly
Signal levels log files and csv's are written
NO PNGs -
the wsprdaemon.2.5a.conf file is configured to upload the png and also to copy png to local webserver... however there are no .png's !
The only single png on the local Apache webserver is empty (0kb) (/var/www/html/...)
the original (apache-)index file is renamed and replaced by a new one, which calls the (empty) png (noise_graph.png)
Even after 1hour of running the kiwi, with lots of spots to wsprnet.org, there are no PNG's...
And hence there is nothing in graphs.wsprdaemon.org/ON5KQ/
I would like to have such bandnoise overview, because I have periodically very strong noise sources from neighbors - perhaps such pictures can show some detailed time patterns, so I might think what causes the noise and find the (broadband) sources...
Is there any secret, I need to know, to get this running ;-)
best regards,
Ulli, ON5KQ
--------------------------- start
Sun Sep 29 09:13:44 PDT 2019: watchdog_daemon() starting as pid 2816
Traceback (most recent call last):
File "/tmp/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
Traceback (most recent call last):
File "/tmp/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
sudo: no tty present and no askpass program specified
Traceback (most recent call last):
File "/tmp/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
sudo: no tty present and no askpass program specified
Traceback (most recent call last):
File "/tmp/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
sudo: no tty present and no askpass program specified
Traceback (most recent call last):
...
...
... {many lines with the same}
-----------------------------end
Any Idea of the experts, what is actually wrong ?
How to repair...?
Ulli, ON5KQ
On a VM
I was not getting the images written until "sudo chown -R myusername:myusername /var/www"
I'm not saying that is the correct way just what worked for my internal server.
On an Apache server with other services running taking ownership of the directory could cause issues.
Cheers
Stu
Powernumpty is correct that your problem derives from the ownership and permissions of the /var/www/html/noise_graph.png file.
I think that if you 'sudo chmod 777 /var/www/html' then your local graphs will start to appear.
This is one of a number of file permissions issues which I will address as soon as I can free myself from other obligations and resume work on WD W 2.6
In anticipation of that work, I have published the current version WD 2.5a to github: https://github.com/rrobinett/wsprdaemon
Rob
I changed /var/www/html permissions (own it with 777 chmod rights) however still no succes.
Watchdog log still shows:
------------alert in watchdog log....
Traceback (most recent call last):
File "/tmp/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
cp: cannot stat '/temp/noise_graph.png': No such file or directory
...
...
--------------
What is the problem with the signal-levels.csv file, which has been written....
I can open the csv as well as the txt-file (signal-levels.log) in a text editor - both files look ok ?!
What is the real problem with noise_plot.py ?
Ulli, ON5KQ
I restart everything:
- the VMware player with Ubuntu to reboot
- Switched off kiwi and disconnect 5V and restarted it (reconnect 5V DC)
- Changed wd-conf file from simple schedule to complex (last row in wsprdaemon.conf file, as I only have a single kiwi and want to try low bands at night and high bands at local daytime hours...
- deleted the KIWI_0 directory in the signal_levels directory (getting rid of old signal level data from yesterday)
- deleted the watchdog.log file
Then started again:
./wsprdaemon.2.5a.sh -a
It seems to work now!
Also 'graphs.wsprdaemon.org/on5kq/' has been written and filled with first data on the web...
Great application and service!
Many thanks....
Will now check, what I can do with it in the future.... no decent antennas yet...
Ulli, ON5KQ
- Kiwi S-meter calibrated with my R&S lab-equipment on 20m for -73dbm signal input at the SMA of the kiwi
- check whether there is a difference of S-meter reading with same generator input level, when changing bands from 2200,630,160,80 ....,10m. Noted indeed about 2db peak on 15m and 12m
- in the valid wsprdaemon.conf file, I accounted for the difference of S-meter reading in my kiwi:
so the correct lookup table for my particular kiwi here is adjusted for best calibration...
Let it read the data for now and lets see.
The antenna is about 60m of wire through a tree feeding at the bottom on one side of the tree and going down to ground level (open end) at the other side of the tree.
Nothing special - rather sensitive, but noisy also...
just a starting point...
Ulli, ON5KQ
http://www.wa2zkd.net:8088/today_int.html
Ask WA2ZKD to let you have a progress page then you can plot curves to compare days.
However at the moment noise source chasing here...
Need some time for real improvements...
Ulli, ON5KQ
Even antennas need some time and good wx - chasing noise sources hopefully will pay off as well...
Many thanks - Ulli, ON5KQ
A question: Is the graph colour (red/blue) the same as mentioned in this article:
https://www.researchgate.net/publication/334612025_Estimating_LF-HF_band_noise_while_acquiring_WSPR_spots
Is there anything different in the version WD 2.5a ?
I still wonder, what increased scattering in the graphs mean:
- good condx with lots of propagated noise (good)
- noisy antenna (bad)
- both... ;-))
Ulli, ON5KQ
We expected the blue ( FFT) to show lower variability than the red (RMS) technique, but that doesn't seem to be the case. Understanding those differences is a subject of great interest to us, and we would welcome any insights you might have.
Topband is very quit at kph, not much thunderstorms - we can see, that scatter even now at night is low, although good propagation (ZK1A is solid S7 with Noise about S3 on S-meter in CW)
How is the TCI antenna connected to the Kiwi ?!
I am interested in the signal distribution - because you need a lot of attenuation to get noise to S3 on that antenna...
Ulli, ON5KQ
The KPH site is indeed quiet, there's some evidence already that it can detect galactic noise and a galactic noise increase as Sagittarius transits the southern sky. The input preamp there should about a 4 dB NF across HF and there is sufficient gain for system noise to be pretty low. All to say an attempt has been made to have a quiet and calibrated system at KPH but 160 may fall outside that calibration. We'll see what Rob says...
TCI 530 -> 2-way splitter (4 dB loss) -> very sharp 1.8 Mhz HPF ( 1 dB loss?) -> DXE/Clifton (23 dB gain, 4 dB NF) -> 30 Mhz LPF ( 1 dB loss ) -> 8 way splitter (9 dB loss) -> Kiwi inputs.
As I understand it, the 5 dB of passband loss before the Clifton increases the system NF from 4 dB to 9 dB. The gain of the Clifton means its NF largely determines the NF of the system rather than the poor NF of the Kiwi.
The exact gain/loss values have not been measured since the new 1.8Mhz HPF was installed. Making those measurements will be a primary goal of my next KPH visit.
At kphsdr.com:8072 there is a Kiwi fed by a 200 ft Marconi-T which one would think would be more sensitive on 160M than the 3-30Mhz TCI-530. However the TCI-530 gets more 160M WSPR spots than the Marconi-T.
I think it is due to common-mode problems together with many networkrouters and -cables radiating, etc...
The site is too valuable not to check the cause of that...
A dead band must be very smooth dark colour in the spectrum with NO signal at all visible. Of coarse with very sensitive antennas it is difficult to achieve completely at almost all but a few locations. However I believe you belong to the lucky ones with such a location - so there is still headroom for improvement....
I often log-in at your local nighttime (our noon)
Keep on working --- the location is a dream!
Ulli, ON5KQ
Attachments:
https://forum.kiwisdr.com/uploads/Uploader/c3/196e3d2e0463afa5797560aeac0cdd.png
Check, if any pre-amp is the reason, because at the kiwi itself, signals are not too strong - it happens in earlier stages perhaps...
The TCI530 kiwi is not clipped - but also consider, that TCI antenna is no real DX antenna! I lived in Anaheim, CA years ago and noticed, that you need really some decent DX-antennas for all the DX-stations (JA, far east Asia) over the pacific on topband...
Ulli, ON5KQ
If you are referring to residual coherent ingress on KPH's noise floor, here's a current spectrograph
yes, there is still some very low level ingress but do recognize that this is a calibrated system and those levels are quite low. It is likely that these are either LAN related CM currents, coupled through imperfect symmetry onto the 'ground' of the kiwi and exiting out the antenna line, even though it too has some measures taken to reduce CM ==> TEM conversion or they are artifacts due to rolling over the top ADC bit (OV). As the above measurement was made, there was also a very strong ~4 MHz signal pushing the system in and out of OV condition. When this happens lines pop up quickly and badly! Here's another graph a few minutes later at an instant when OV was not lit
At levels down in the -160 dBm/1-Hz area, it takes only 40 pico-amperes into 50 ohms to produce a response. Compare this with possible microamperes of CM current on these lines. It takes a lot of symmetry & isolation to get that kind of rejection.
This same system easily sees propagated noise at 20 MHz and probably galactic noise as well (in between any coherent ingress).
You are right though, there's seems to always be room for improvement.
I appreciate your insights. Apart from the occasional OV on the Marconi, there seem to be at least these sources of crud on both the LF/MF and HF Kiwis.
1) The 10 kHz and other narrowly spaced lines you see above 1.6 Mhz appear to be due to IMD generated by a legacy LNA fed by the Marconi-T and (perhaps) rusted fence wires in the antenna field. I have proposed removing or replacing that LNA, but KPH is an historic 'living history' museum so such changes are made only after much consideration.
2) I suspect there are 64 KHz lines from the wired LAN connections, but I don't see those so much at KPH. I have seen those mostly above 15 MHz at other Kiwi sites (e.g. http://kiwisdr.robinett.us:8074/?mute&sp&f=10000/1000z0AMN&keys=yyy&wfm=-120,-50). Those lines can be removed by replacing the wired LAN connections with USB wifi dongles, but I have had stability problems with the Wifi connections, so until I am confident in the dongles I can't deploy them at a remote site like KPH.
3) There are probably 1 Mhz lines due to radiation from the SWPSU feeding the Kiwis. The KPH site is run off of a linear 14 VDC PSU feeding a 12VDC battery backup system, so when the AC power goes out this winter we should learn how quiet the site really is. The individual 12V to 5V Kiwi SWPSU would benefit from some CM filtering from torriods on the input and output lines of the SWPSU, but I'm not sure that this is a serious noise source and it will take hours to fix.
4) IT equipment in the rack near the Kiwis. I have been asked to confine all of this 21st Century equipment to a single small rack, so the modem, router, enet switches and Pis are all probably sources of RFI which is almost impossible to isolate and identify.
Access to KPH is a major challenge to addressing these issues. The site is open regularly only on Saturday afternoons, during which time the fluorescent lights are turned on and generate crud all over the bands above 10 Mhz. I try to stage changes, but the debug cycle is at least one week and I really need to get back to fixes and enhancements to WD after a four month break.
I have had much help from Glenn and others, but the limited access time and the distance out to KPH (2 hours each way for me) impede our progress.
many thanks of taking my comments constructive - and fully understand, that the character of a museum is most important!
And understood, that driving time and limited access time is a problem...
You already have identified a lot of things - so it is a matter of time and the personal efforts of the local enthousiasts -- congratulation for so much personal efforts the team is spending.
Ulli, ON5KQ
As the vertical is only 6m of wire attached to an old fishing pole it is not very effective radiator on 160m - now I know, that the huge signals on medium wave causing the pre-amp overloading in the evening, were simply the signals from the "Beverage on ground" (=the feedline to the house!) NOT the signal from the vertical.
I also noticed some directions being rather week, before I installed the common-mode chokes. I guess it was interaction with the cable which causes the nulls of radiation before - now I can't see any directivity any more - the vertical seems to work as a good omnidirectional antenna.
Today I found some interesting noise plot on 60m with my kiwi:
From 11h utc the developing thunderstorms coming from France along the coastline towards Belgium became more and more audible on 60m band.
While the noise increases we can see the blue (FFT noise) becomes almost a line, but without scattering as we usually can see. The difference to the RMS-noise measurement increases...
I think this is typical for such a severe QRN build-up with upcoming thunderstorms getting closer and closer...
Here is the actual weather condx with the thunderstorm front over France
(my position is in jo10os, for reference)
And that is the SDR-screen - look at the waterfall - full of very strong static crashes - in between deep noise holes many s-units lower than the peak of the static crashes... an other word of noise plot scattering (the picture above?)...
Don't get confused from the S3 on the s-meter . I was just using a piece of wire to capture the foto for this post - no resonant antenna on 60m - I think the static crashes are almost S9+20db on a 60m dipole...
regards,
Ulli, ON5KQ
I had carefully calibrated the S-meter with a signal generator in the lab.
Now - how should I specify the Default noise levels in each band in de conf file of wsprdaemon? What does for example mean "Default:-10,80" ?
How do I correctly use the Default: level, band parameter ?
At this moment there seem to be something wrong to me with my kiwi:
I just compared the Noise figures of the website http://graphs.wsprdaemon.org/ON5KQ/ with the S-meter readings of my Kiwi. In 1kHz bandwidth on the currently dead 10m band the kiwi S-meter shows -120dbm noiselevel, which is about 7db above the level with a terminated input of 50Ohm (pure receiver noise)
That means that in 1Hz bandwidth the noise is 30db lower so arround -150dbm/Hz
However: the noiseplot shows almost 10 db lower values on 10m just above -160dbm - why?
Did I wrongly configure the 10m DEFAULT parameter - (it is now configured as "Default:-10,10")
Why the default should be -10db for all bands as mentioned in the conf. template?
Can someone please explain the Default parameter for noise plots in wsprdaemon 2.5a to me please ?
I once used a calibrated antenna system at my location to estimate my real noise field strength. I was thinking this "default"- parameter can be used like a look-up table when the antenna factor is known - so then we can compare the values directly with current ITU recommendation...
Many thanks,
Ulli, ON5KQ
Thank you for posting the 60m noise graph before and during the thunderstrorms, and for the map - the line north/south over France was immediately to the south of my own QTH here in Southampton, UK.
I worked with Rob, AI6VN, on the noise analysis software, and I think I have a plausible explanation for why the RMS and FFT algorithms result in the pattern that you see.
For the RMS algorithm (red) the noise measurement is from the quietest fifty milliseconds during either the 0.5 second before WSPR transmissions or during 5 seconds after transmissions. That means the algorithm has 110 fifty millisecond "windows" to choose the quietest - so there is a good chance of finding one of the 'deep noise holes' you mention. But not always! So there are two red dots near the blue dots, near 1300 and just after, where there possibly wasn't a 'hole' during those 5.5 seconds.
The FFT algorithm (blue) estimates the noise from the lowest 30% of Fourier coefficients calculated throughout the two minute period. Normally, this percentage is low enough to exclude the WSPR signals themselves and interference - after all 70% is excluded. But, I think that in the case you show, there's just so much thunderstorm activity and coming towards you, that noise from the lightning creeps more and more into the 30% measurement interval and produces a higher noise level estimate.
I've written at length as we are collecting nice observations such as yours of noise measurements with wsprdaemon and the possible causes for the patterns seen. I have downloaded your 60m data for this period from the Grafana site. What was the source for your thunderstorms map, so I can provide a credit?
73 and thanks for the interesting observations
Gwyn G3ZIL
The noise measurement at the KiwiSDR SMA connector is generally quite close to correct when "-16" is entered into the calibration value on the admin page. This seems to be true for both noise and coherent signals, at least within a dB or so.
To get calibrated to external noise, we have to decide on a reference. The ITU values presume a well-matched monopole over a 'good' ground. There are not assumed to be large ground losses which would increase the antenna factor of that reference vertical monopole. Simply moving to a horizontal dipole complicates all this both due to take-off angle and ground loss, which is often on the order of 6 dB. A vertical monopole in this configuration has roughly 2 dB of pattern gain toward the horizon and another 3 dB or "ground gain" - it's only looking at one hemisphere, since currents in the image element (within the grounding system) adds to the 'visible' hemisphere.
Thus we need to modify the gain settings based on how different the actual antenna is from this reference monopole, with any gain or loss between that antenna and the SMA connector accounted for. This isn't always easy. For anyone with a monopole mounted over seawater and lossless feedline with no preamplification and no mismatch loss, no correction may be necessary. For everyone else some value is probably indicated. The reality is that many will not know nor even be able to accurately estimate the antenna system antenna factor, so we probably only do the best we can.
The easiest part is to account for any known feedline mismatch and loss and any preamplifier, filter gain or loss. Probably the best way to do this is to measure with a VNA, perhaps something like the nanoVNA can do a fine job in a 50 ohm system. As a minimum a correction (vs. frequency) of this value should be included in the WD correction to move the reference point from the kiwi SMA to the antenna connection.
To remove the remaining piece, the nature of the actual antenna, that factor also needs to be known. Getting to the the antenna factor of an arbitrary antenna, particularly one that isn't as 'simple' as a monopole or dipole and is mounted over unknown "ground" material, is probably harder. NEC2 (or its derivatives) may be the best hope to even get close to determining it.
The goal of all this is to have useful absolute values which may be compared to the ITU excess noise numbers as well as to allow comparison among multiple kiwi/WD installations. We're probably not going to get there perfectly. In the end this is rather like a radio astronomy problem, at least if systems are good and noise floor is determined by distant noise sources rather than local ingress or system noise figure. Low antenna factors help this as do low system noise figures and good IMD, image and other unwanted signal rejection.
A stock, terminated kiwi often shows about a -156 dBm/1-Hz (-122 displayed in a SSB bandwidth). Simply looking to see how much the noise comes up (if it does) when the antenna system, including all the elements above, is connected then gives an idea of quality of the operation when it is compared to ITU numbers. For many of us that seems to be a starting point in improving our systems.
The usefulness of the noise plots is increased by knowing the absolute levels and circumstances but even without proper calibration, just relative noise and noise from multiple measurement algorithms is useful and interesting, as you have described. Diurnal variation and the variation between algorithms alone has proven to be interesting.