Many thanks for the information - so I really need to change my settings for the two receivers in the .conf-file
To WA2ZKD: Please forgive me not to understand coding in Linux.. What do I need to do, if I want to use your methode by comparing performance of the merged receivers?
What to do to install the bash-file and where can I read the results afterwords - I guess the program (bash -file) generates a sort of list ?!
Sorry for my stupid question - no computer expert unfortunately...
Ulli, ON5KQ
P.S.: I have many broadband local noise sources recently. The purpose is to compare different locations of the same antenna systems. So I build two identical antenna's and want to compare them. One antenna is in my garden only 40m from the house. The other antenna I will place in the middle of the field about 300m far away from the house. Want to see - first - if there is a major difference in the noise graphs and second, what the difference in S/N ratio spotting stations is... Hopefully placing the antenna far away from the houses helps ...
use an editor to to put that short bash script into a file in /usr/local/bin and name it post. then run this command chmod 555 /usr/local/bin/post you can run that command by typing post and it will produce a listing of received spots and show which was posted
If WD is started with verbosity level 1 then simply typing cat ME*/*/posting.log will show a comparison among all merged receivers or something like cat ME*/20/posting.log to show recent results from a single band (20 meters in this case) Jim's method is more flexible but I tend to lose these unless I build an alias for them.
"Installation on a system with an existing copy of wsprdaemon not installed using 'git clone'" These instructions are for a system WITH wsprdaemon installed, surely?
I have an earlier version of wsprdaemon running OK, and have not had to ssh into the Pi for ages, now I try to ssh in with Putty, or look in with winSCP neither see the Pi, although it's there with a static IP address on my network and I can ping it, and it's running several receivers and uploading spots. Any idea why i can no longer access it? At the moment it's headless, I guess I will have to put a monitor and keyboard on it as RealVNC can't access it either Any ideas please? I am using the correct IP address as shown on my router access pages.
Thanks for the updates Rob, great addition to a Kiwi. Regards 2E0ILY
I have tried to clarify that title by changing it to: "Installation on a system running wsprdaemon that was not installed using 'git clone'"
Perhaps WD has overflowed the Pi's file system. While working on V2.6* I have added some checks to limit the size of log files, so later versions 2.6e+ should address some or all of those problems.
So I would suggest you power cycle your Pi and then 'git pull' and get v2.6f and restart WD.
OK, thanks, I see I misunderstood the title in either case, my fault! I did re cycle the Pi and still can't access it remotely, I'll pull it out of its hiding place and put a monitor and keyboard on it. Should re cycling it have purged the logs if they are full?
Yes, a reboot or power cycle rebuilds the /tmp/wsprdaemon tmpfs under which all of WD's temporary files are kept. There are a few small status and log files kept in ~/wsprdaemon, but only 'wsprdaemon.log' could grow large, and that only when WD is run at verbosity level 1 or higher (-va or -vva, ...)
Thanks, for some reason ssh and VNC were no longer set active, so I fixed that and hopefully it I have now updated wsprdaemon, but it no longer seems to auto start the wsprdaemon on reboot. I have to manually start it in the terminal
Sorry to be a pain, but A: Is there a command to check and see the version that's now running and B: Is there a way to see if auto start of the daemon is set, and how to set ti to auto start on reboot if it's not set already? Many thanks!
WD -V (capital V) will print the version number of yoru WDS "sudo systemctl status wsprdaemon" will show info about status "sudo systemctl enable wsprdaemon" will ensure that WD is started at server boot time. I will add a command flag to v2.7x check/enable/disable startup
Rob, some problem shows up regularly - it shows up in the wsprdaemon.log file I do not understand what the code means, but at least I understand it is an error. Do you know, what the problem is and what I need to do ?
Currently I do not use Surise/Sunset parameters in .conf file - so therefore WD seam to work, but it would be nice to configure with sunrise/sunset features...
Here is the error code (copy/paste from Linux): ------------- Tue Jan 28 10:38:10 CET 2020: watchdog_daemon() starting as pid 128533 Traceback (most recent call last): File "/tmp/wsprdaemon/suntimes.py", line 8, in l.sun() File "/usr/lib/python2.7/dist-packages/astral.py", line 735, in sun sun = self.astral.sun_utc(date, self.latitude, self.longitude) File "/usr/lib/python2.7/dist-packages/astral.py", line 1569, in sun_utc dawn = self.dawn_utc(date, latitude, longitude) File "/usr/lib/python2.7/dist-packages/astral.py", line 1607, in dawn_utc 'at this location.') % (depression - 90)) astral.AstralError: Sun never reaches 6 degrees below the horizon, at this location. ./wsprdaemon.sh: line 601: astral_times[0]: unbound variable -------- I use Ubuntu 18.04LTS on virtual machine with WD 2.6f
Ulli, ON5KQ
P.S.: yesterday afternoon my 20...10m longwire broke - now the wire is only 1m "high" pointing to west (170m end-fed longwire) ... I need it back on 15m over the ground at least - but need better wx-condx. Forecast is very bad with higher winds and rain this week. Fingers crossed for succesfull antenna work perhaps next week...
The local sunrise and sunset times are calculated by WD translating the Maidenhead grid you specify for your receiver in the conf file to a lat/long. Those lat/long values are passed to the python program /tmp/wsprdaemon/suntimes.py which uses the python Astral library to calculate the sunrise and sunset times. Your error messages are from that python program and suggest that the lat/long values are not valid. So do you specify the full six character Maidenhead? If you share your conf file I can reproduce the error.
Hi, Rob - thanks for the hint. I checked my configuration of Maidenhead locator. I noticed that I configured 'jo10os' (all small letters) and changed it in JO10os (two first letters in cap-letters) - so far, I did not get error message again in the wsprdaemon log file (starting WD with ./wsprdaemon.sh -va)
Ulli, ON5KQ
P.S: Found nice foto's on google street map from kph location - the view from the main street to the antenna fields is terrific (receive site)... so I enjoyed my virtual visit...hi
I have just published v2.7a to github. It adds persistent storage of spots and noise data through reboots and power cycles and uses very little extra CPU cycles to calculate noise. A Pi 4 should easily support plotting noise on 30+ bands.
Rob - great, I can see the Grafana FFT curve again after upgrading...
However WSPRDAEMON.log throws some errors regularly: do I need to 'repair' something ?
----- ... Sat Feb 8 07:29:06 CET 2020: let_kiwi_get_gps_lock() kiwi 'NW_170mLW' reports '2' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi Traceback (most recent call last): File "/tmp/wsprdaemon/noise_plot.py", line 69, in noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:] IndexError: too many indices for array mv: cannot stat '/tmp/wsprdaemon/wd_tmp.png': No such file or directory Sat Feb 8 07:31:10 CET 2020: let_kiwi_get_gps_lock() kiwi 'E_170mBev' reports '0' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi Sat Feb 8 07:31:11 CET 2020: let_kiwi_get_gps_lock() kiwi 'NW_170mLW' reports '2' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi Traceback (most recent call last): File "/tmp/wsprdaemon/noise_plot.py", line 69, in noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:] IndexError: too many indices for array mv: cannot stat '/tmp/wsprdaemon/wd_tmp.png': No such file or directory Sat Feb 8 07:33:10 CET 2020: let_kiwi_get_gps_lock() kiwi 'E_170mBev' reports '0' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi Sat Feb 8 07:33:10 CET 2020: let_kiwi_get_gps_lock() kiwi 'NW_170mLW' reports '2' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi ... ----------------- Ulli
I'm also getting one of the same errors as Ulli above: Traceback (most recent call last):
File "/tmp/wsprdaemon/noise_plot.py", line 72, in
noise_vals=noise_vals.reshape(n_recs,14) # reshape to 2D array with n_recs rows and 13 columns
ValueError: cannot reshape array of size 78 into shape (5,14)
mv: cannot stat '/tmp/wsprdaemon/wd_tmp.png': No such file or directory
Grafana data is updating fine. But, the local .png noise graph is not updating post-update, and I think that is leading to the graph on graphs.wsprdaemon.org also not being updated.
Quick additional note from the update instructions. Not all of us have the alias "wd" as a shortcut to the wsprdaemon script, and the instructions if followed exactly as stated will fail for people. Hopefully most people will be able to figure out that one should specify the script, but you might get more questions.
I've seen a significant drop in CPU usage on both of my PIs with ~20% less overall CPU used, and loads dropping from ~3.1 to ~2.1 - a not-insignificant reduction in processing usage.
I restarted Ubuntu (so complete reboot) and the error messages have not popped up again... may be some old data (after updating wsprdaemon) caused the error messages...
I should better look also for a raspberry PI mini-computer. Still running the App on my main PC in virtual machine with Linux..
Thanks for the feedback. I will clarify the use of 'wd' and alias and add further cleanup to the installation instructions.
As for the plotting problems, here is more information than may interest you on the cause.
v2.7 adds c2_fft data to each line of the signal-levels.log file which is used to create the noise graphs.
In 2.6 the 13th column was the source of fft data which was created by running the sox fft command
In 2.7 there are 14 column in each line and by default sox fft is not run, so that column is set to -999.9. The the newly implemented c2_fft value is always calculated and put in to the newly populated column 14 (the last one of each line).
In v2.7 the python plotting program /tmp/wsprdaemon/noise_plot.py has been modified to use the values in column 14. I think I regenerate that program when the plotting is done each odd 2 minutes, but if a 2.6 noise_plot.py is run it will plot column 13 which is now -999.9 and thus off the bottom of the graph.
In addition, legacy lines in /home/pi/wsprdaemon/signal_levels/*/*/signal-levels.log files might not contain 14 columns and crash the new noise_plot.py
So the safest way to upgrade probably involves: 1) flushing your old signal-level.logs files with: 'rm -rf /home/pi/wsprdaemon/signal_levels/*' then 2) rebooting your server which will recreate /tmp/wsprdaemon/....
Of course it will then be several WSPR cycles before there are enough measurements for the noise graphs to show visible points.
but there is a problem, that spots from WD Vers 2.7 are not accepted from WSPRnet.org anymore. What is going on - it seems that WSPRNET.org doesn't like Wsprdaemon script and blocks the Curl-uploads...
No spots from OE9GHV either (check the VK7JJ site for example), although his Kiwis are running in WSPR mode...
Now it works! I deleted all the old spots WD decoded, but could not upload because of the server problems. Now as the server works again WD wanted to upload hundreds of spots from WD.... that was the reason WSPRnet.org, didn't liked me...hi. The server simply refused to take hundreds of spots from the past... Now, while I deleted the old spots, obviously there is no problem with the 20-30spots per 2min in realtime (not from the past)
In 2.7, WD caches spots in ~/wsprdaemon/uploads.d/... and every odd 2 minutes it tries to upload a block of no more than 200 spots starting with the oldest spots in the cache. I set these limits so that WD clients don't swamp wsprnet.org after an extended outage.
When wsprnet.org is down for many hours like it was last night, top spotting sites like you and Ulli will have cached 1000's if not 10000's of spots. So it can take hours for WD to flush its cache, but all spots will eventually be uploaded. I will add a 'cache status' message in v2.8 to give users some feedback about the upload status.
I knew about the basics and never touched by WD box. Unfortunately some reset theirs I think. I shut my scraper down during this morning for 8 hours so that it wasn't "nagging" wsprnet. Today's data on it will not be complete. Who knows what wsprnet's archive will end up with.
Many thanks for info.... next time, I will keep it running, in such " server down' event...
Some interesting information: I am testing a new antenna, comparing it with the beverages. It works very well so far! I really recommend it to build, especially for broadband wspr and general SWLing...
Comments
To WA2ZKD:
Please forgive me not to understand coding in Linux..
What do I need to do, if I want to use your methode by comparing performance of the merged receivers?
What to do to install the bash-file and where can I read the results afterwords - I guess the program (bash -file) generates a sort of list ?!
Sorry for my stupid question - no computer expert unfortunately...
Ulli, ON5KQ
P.S.: I have many broadband local noise sources recently. The purpose is to compare different locations of the same antenna systems. So I build two identical antenna's and want to compare them. One antenna is in my garden only 40m from the house. The other antenna I will place in the middle of the field about 300m far away from the house. Want to see - first - if there is a major difference in the noise graphs and second, what the difference in S/N ratio spotting stations is...
Hopefully placing the antenna far away from the houses helps ...
use an editor to to put that short bash script into a file in /usr/local/bin and name it post.
then run this command chmod 555 /usr/local/bin/post
you can run that command by typing post and it will produce a listing of received spots and show which was posted
I did that and it worked...
Interesting new features, I learned. So now it needs some time to find out best antennas and locations...hi
Ulli, ON5KQ
cat ME*/*/posting.log
will show a comparison among all merged receivers or something like
cat ME*/20/posting.log
to show recent results from a single band (20 meters in this case)
Jim's method is more flexible but I tend to lose these unless I build an alias for them.
"Installation on a system with an existing copy of wsprdaemon not installed using 'git clone'" These instructions are for a system WITH wsprdaemon installed, surely?
I have an earlier version of wsprdaemon running OK, and have not had to ssh into the Pi for ages, now I try to ssh in with Putty, or look in with winSCP neither see the Pi, although it's there with a static IP address on my network and I can ping it, and it's running several receivers and uploading spots. Any idea why i can no longer access it? At the moment it's headless, I guess I will have to put a monitor and keyboard on it as RealVNC can't access it either
Thanks for the updates Rob, great addition to a Kiwi. Regards 2E0ILY
I have tried to clarify that title by changing it to: "Installation on a system running wsprdaemon that was not installed using 'git clone'"
Perhaps WD has overflowed the Pi's file system. While working on V2.6* I have added some checks to limit the size of log files, so later versions 2.6e+ should address some or all of those problems.
So I would suggest you power cycle your Pi and then 'git pull' and get v2.6f and restart WD.
Rob
Sorry to be a pain, but A: Is there a command to check and see the version that's now running and B: Is there a way to see if auto start of the daemon is set, and how to set ti to auto start on reboot if it's not set already? Many thanks!
"sudo systemctl status wsprdaemon" will show info about status
"sudo systemctl enable wsprdaemon" will ensure that WD is started at server boot time.
I will add a command flag to v2.7x check/enable/disable startup
some problem shows up regularly - it shows up in the wsprdaemon.log file
I do not understand what the code means, but at least I understand it is an error.
Do you know, what the problem is and what I need to do ?
Currently I do not use Surise/Sunset parameters in .conf file - so therefore WD seam to work, but it would be nice to configure with sunrise/sunset features...
Here is the error code (copy/paste from Linux):
-------------
Tue Jan 28 10:38:10 CET 2020: watchdog_daemon() starting as pid 128533
Traceback (most recent call last):
File "/tmp/wsprdaemon/suntimes.py", line 8, in
l.sun()
File "/usr/lib/python2.7/dist-packages/astral.py", line 735, in sun
sun = self.astral.sun_utc(date, self.latitude, self.longitude)
File "/usr/lib/python2.7/dist-packages/astral.py", line 1569, in sun_utc
dawn = self.dawn_utc(date, latitude, longitude)
File "/usr/lib/python2.7/dist-packages/astral.py", line 1607, in dawn_utc
'at this location.') % (depression - 90))
astral.AstralError: Sun never reaches 6 degrees below the horizon, at this location.
./wsprdaemon.sh: line 601: astral_times[0]: unbound variable
--------
I use Ubuntu 18.04LTS on virtual machine with WD 2.6f
Ulli, ON5KQ
P.S.: yesterday afternoon my 20...10m longwire broke - now the wire is only 1m "high" pointing to west (170m end-fed longwire) ... I need it back on 15m over the ground at least - but need better wx-condx. Forecast is very bad with higher winds and rain this week. Fingers crossed for succesfull antenna work perhaps next week...
Those lat/long values are passed to the python program /tmp/wsprdaemon/suntimes.py which uses the python Astral library to calculate the sunrise and sunset times.
Your error messages are from that python program and suggest that the lat/long values are not valid. So do you specify the full six character Maidenhead?
If you share your conf file I can reproduce the error.
I checked my configuration of Maidenhead locator.
I noticed that I configured 'jo10os' (all small letters) and changed it in JO10os (two first letters in cap-letters) - so far, I did not get error message again in the wsprdaemon log file (starting WD with ./wsprdaemon.sh -va)
Ulli, ON5KQ
P.S: Found nice foto's on google street map from kph location - the view from the main street to the antenna fields is terrific (receive site)... so I enjoyed my virtual visit...hi
However WSPRDAEMON.log throws some errors regularly:
do I need to 'repair' something ?
-----
...
Sat Feb 8 07:29:06 CET 2020: let_kiwi_get_gps_lock() kiwi 'NW_170mLW' reports '2' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi
Traceback (most recent call last):
File "/tmp/wsprdaemon/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
mv: cannot stat '/tmp/wsprdaemon/wd_tmp.png': No such file or directory
Sat Feb 8 07:31:10 CET 2020: let_kiwi_get_gps_lock() kiwi 'E_170mBev' reports '0' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi
Sat Feb 8 07:31:11 CET 2020: let_kiwi_get_gps_lock() kiwi 'NW_170mLW' reports '2' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi
Traceback (most recent call last):
File "/tmp/wsprdaemon/noise_plot.py", line 69, in
noise_vals = genfromtxt(csv_file_path, delimiter=',')[:,1:]
IndexError: too many indices for array
mv: cannot stat '/tmp/wsprdaemon/wd_tmp.png': No such file or directory
Sat Feb 8 07:33:10 CET 2020: let_kiwi_get_gps_lock() kiwi 'E_170mBev' reports '0' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi
Sat Feb 8 07:33:10 CET 2020: let_kiwi_get_gps_lock() kiwi 'NW_170mLW' reports '2' good GPS which is less than the min of 4 we require. So GPS is bad on this Kiwi
...
-----------------
Ulli
I'm also getting one of the same errors as Ulli above:
Traceback (most recent call last): File "/tmp/wsprdaemon/noise_plot.py", line 72, in noise_vals=noise_vals.reshape(n_recs,14) # reshape to 2D array with n_recs rows and 13 columns ValueError: cannot reshape array of size 78 into shape (5,14) mv: cannot stat '/tmp/wsprdaemon/wd_tmp.png': No such file or directoryGrafana data is updating fine. But, the local .png noise graph is not updating post-update, and I think that is leading to the graph on graphs.wsprdaemon.org also not being updated.
Quick additional note from the update instructions. Not all of us have the alias "wd" as a shortcut to the wsprdaemon script, and the instructions if followed exactly as stated will fail for people. Hopefully most people will be able to figure out that one should specify the script, but you might get more questions.
I've seen a significant drop in CPU usage on both of my PIs with ~20% less overall CPU used, and loads dropping from ~3.1 to ~2.1 - a not-insignificant reduction in processing usage.
I should better look also for a raspberry PI mini-computer. Still running the App on my main PC in virtual machine with Linux..
Ulli
As for the plotting problems, here is more information than may interest you on the cause.
v2.7 adds c2_fft data to each line of the signal-levels.log file which is used to create the noise graphs.
In 2.6 the 13th column was the source of fft data which was created by running the sox fft command
In 2.7 there are 14 column in each line and by default sox fft is not run, so that column is set to -999.9. The the newly implemented c2_fft value is always calculated and put in to the newly populated column 14 (the last one of each line).
In v2.7 the python plotting program /tmp/wsprdaemon/noise_plot.py has been modified to use the values in column 14. I think I regenerate that program when the plotting is done each odd 2 minutes, but if a 2.6 noise_plot.py is run it will plot column 13 which is now -999.9 and thus off the bottom of the graph.
In addition, legacy lines in /home/pi/wsprdaemon/signal_levels/*/*/signal-levels.log files might not contain 14 columns and crash the new noise_plot.py
So the safest way to upgrade probably involves:
1) flushing your old signal-level.logs files with: 'rm -rf /home/pi/wsprdaemon/signal_levels/*'
then
2) rebooting your server which will recreate /tmp/wsprdaemon/....
Of course it will then be several WSPR cycles before there are enough measurements for the noise graphs to show visible points.
Does anyone know details ?
What is going on - it seems that WSPRNET.org doesn't like Wsprdaemon script and blocks the Curl-uploads...
No spots from OE9GHV either (check the VK7JJ site for example), although his Kiwis are running in WSPR mode...
Ulli
I deleted all the old spots WD decoded, but could not upload because of the server problems. Now as the server works again WD wanted to upload hundreds of spots from WD.... that was the reason WSPRnet.org, didn't liked me...hi. The server simply refused to take hundreds of spots from the past...
Now, while I deleted the old spots, obviously there is no problem with the 20-30spots per 2min in realtime (not from the past)
the learning curve....hi
Ulli
In 2.7, WD caches spots in ~/wsprdaemon/uploads.d/... and every odd 2 minutes it tries to upload a block of no more than 200 spots starting with the oldest spots in the cache. I set these limits so that WD clients don't swamp wsprnet.org after an extended outage.
When wsprnet.org is down for many hours like it was last night, top spotting sites like you and Ulli will have cached 1000's if not 10000's of spots. So it can take hours for WD to flush its cache, but all spots will eventually be uploaded. I will add a 'cache status' message in v2.8 to give users some feedback about the upload status.
Some interesting information: I am testing a new antenna, comparing it with the beverages. It works very well so far! I really recommend it to build, especially for broadband wspr and general SWLing...
I explained the details in the antenna section of this forum:
http://forum.kiwisdr.com/discussion/1913/broadband-antenna-with-gain-on-high-bands#latest
Ulli