Check to see if your /tmp/wspr-captures file system has overflowed. Alternatively, reboot your Pi. I will add checks for file overflow condition in future versions.
I can see a problem when WD caches many spots due to an internet outage or outage at wsprnet.org. But that should be solved by a reboot Also, spots may not get reported if the /tmp/wspr-captures file system fills up. Run 'watch df /tmp/wspr-captures' for several minutes to see if overflowing is a problem and increase the size of that file system if it is.
Update - my missing spots have now appeared on wsprnet.
I think part of the problem may be the response time of wsprnet as it's taking progressively longer to get any response back from the website. I noticed this a while ago but now it's getting to be much worse.
I think it's suffering from it's own success and all those automated systems now running that are uploading huge volumes of spots :-)
I have been watching the WD upload log and see many, many timeouts on upload attempts to wsprnet.org. WD caches it spots until a successful upload so when wsprnet.org goes offline for 10 minutes or more WD can be caching 100 or more spots. wsprnet.org seems to reject uploads of 100+ spots (I'm not sure what is that limit, so that is a guess). At that point WD gets into a continuous upload failure loop while its cache of spots grows larger and large. I am going to add a limit of upload size to WD 2.6b to try to address this problem. But the larger problem is the frailty of wsprnet.org. I maintain a research grade database of noise data at wsprdeamon.org and I am thinking I will add a spots database as well. WD users will be able to make real queries of their spot data and create graphs from queries.
If you did make a database one thing I'd really like is the ability to run two or more systems and break down which works better on a band. I have one KiwiAI on a LZ1AQ loop (14Ch) and a second KiwiBBG on a Wellbrook (running 7 so I have a slot free to check the antenna). Currently it takes a long time to really get a feel for adding a band from the more limited BBG.
As usual a very personal reason for a feature but I do wonder if someone had say three loops (+ nulls) with otherwise identical setups, they could detect an opening or atmospheric anomaly in a specific direction.
I have made antenna comparisons at KPH by configuring the second antenna system to log spots as KPH/N. Then Jim's site shows a comparison by band of the two using: "curl www.wa2zkd.net:8088/today.html | html2text | grep KPH" More detailed comparisons can be made at Phil's excellent site: wspr.vk7jj.com/"
I have just checked in V2.6b to https://github.com/rrobinett/wsprdaemon where you will find installation and upgrade instructions. Among other improvements, this version addresses an upload problem which may be causing WD users to pound the wsprnet.org database. So I strongly encourage existing user to upgrade to V2.6b
In the notes it recommends "mv ~/wsprdaemon ~/wsprdaemon.save" Should not be something like "mv ~/wsprdaemon/wsprdaemon.conf ~/wsprdaemon.save"
On my non standard VM I tried a couple of unsucessfull git pulls and obviously got it wrong (nothing changes eh) so for speed just ended just removing the whole directory (graphs and all). Figured a days worth of QRM was fine to lose if it meant not hammering wsprnet.org
Might want to update the title of this thread? (Both kiwis are now providing audio streams to a pair of Pi4s, both now happily appearing to decode at 2.6a version.)
Yes, I know that one pi4 can happily take the output of of two BBG Kiwis, but my Kiwis are one in Limerick and one in Zurich, so not to easy to do one for both.
I have just checked in V2.6b code which further improves the upload operations. The code is also more wsprnet.org server friendly, so hopefully it will help suppress some of the long hangs of the past few weeks. Get it with 'git pull'
Good afternoon, after some time, I am again a happy user of WD installed on my VMware Workstation Player 15 (Virtual machine with Ubuntu 18 LTS).
The updates posted on Github can be easily installed on a regualar basis by "git pull"... I like that easy maintainance...
Now I have a question: I am running 2 kiwi's and have configured them in the .conf file for configuration of WD However what do I need to do to always spot the strongest decode of a single particular call at a given moment. Or in other words, do I need to configure the 'merge' parameter ? What is the purpose of the merge parameter and how to use it ? Can someone give an example, please.
Currently I am testing a lot with various antennas - the problem is the heavy interference and broadband noise from local later afternoon to late evening from strong LED-Noise and strong radiation from the neighbors VDSL2 telecom connection. Trying to phase-out the noise sources without succes, unfortunately. Afternoon till late evening is almost useless for the moment. Summer is much better here, when LED-lamps are not needed in the evenings (due to late daylight hours..)
I owe to Jim the suggestion to implement this important MERG feature in WD.
Here is a fuller explanation and example of the use of MERGed receivers: 1) MERG_K0_K1 merges the spots decoded from KIWI_0 with those decoded from KIWI_1 2) MERG_K0_A0 merges the decodes from KIWI_0 with decodes from the analog audio output of a 10M receiver 3) MERG_K0_A1 merges the decodes from KIWI_0 with decodes from the analog audio output of a 20M receiver
This is *NOT* diversity reception where the RF or audio signals are mixed. The decodes are performed separately on the signals from each of the MERGed receivers, from which WD picks the best set of decodes to be posted to wsprnet.org.
As a configuration example, here is a section of the conf for my Pi4 with two local Kiwis and two USB audio dongles attached:
declare RECEIVER_LIST=(
"KIWI_0 192.168.1.20:8073 AI6VN CM88mc NULL" ### Kiwi fed by 160M - 10M antenna
"KIWI_1 192.168.1.21:8073 AI6VN CM88mc NULL" ### Kiwi fed by 2200M - 160M antenna
"AUDIO_0 localhost:1,0 AI6VN CM88mc foobar" ### Audio from a receiver tuned to 10M. The id AUDIO_xxx is special and defines a local audio input device as the source o
"AUDIO_1 localhost:2,0 AI6VN CM88mc foobar" ### Audio from a receiver tuned to 20M
"MERG_K0_K1 KIWI_0,KIWI_1 AI6VN CM88mc foobar" ### The antennas useful ranges overlap on 160M
"MERG_K0_A0 KIWI_0,AUDIO_0 AI6VN CM88mc foobar" ### A 'virtual' receiver useful only for 10M WSPR band
"MERG_K0_A1 KIWI_0,AUDIO_1 AI6VN CM88mc foobar" ### A 'virtual' receiver useful only for 20M WSPR band
)
declare WSPR_SCHEDULE=(
"00:00 MERG_K0_A0,10 KIWI_0,15 KIWI_0,17 MERG_K0_A1, 20 KIWI_0,30 KIWI_0,40 KIWI_0,80 MERG_K0_K1,160 KIWI_1,630 KIWI_1,2200"
When so configured, on 20M WD merges the signals received by AUDIO_1 with those from KIWI_0 and posts only one copy (the strongest) of each received signal. Similarly on 10M WD merges the signals received by AUDIO_0 with those from KIWI_0 , and on 160M WD merges signals from KIWI_0 with those from KIWI_1.
Not only will MERGed rxs avoid double posting spots to wsprnet.org, but as importantly when WD is run at verbosity level 1 (start as ./wsprdaemon.sh -va'), WD generates decode reports each two minutes which show all the spots in a MERG rx and which spot was chosen to be posted to wsprnet.org. Here is an example from another WD site of what you can learn in those posting.log files:
odroid@wspr:/tmp/wspr-captures$ ls -lrt /tmp/wspr-captures/*/*/posting.log
-rw-r--r-- 1 odroid odroid 0 Jan 9 21:10 /tmp/wspr-captures/MERGED_RX_0/2200/posting.log
-rw-r--r-- 1 odroid odroid 32232 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/17/posting.log
-rw-r--r-- 1 odroid odroid 87501 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/20/posting.log
-rw-r--r-- 1 odroid odroid 22376 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/12/posting.log
-rw-r--r-- 1 odroid odroid 23300 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/60/posting.log
-rw-r--r-- 1 odroid odroid 26226 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/10/posting.log
-rw-r--r-- 1 odroid odroid 28614 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/15/posting.log
-rw-r--r-- 1 odroid odroid 27073 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/80eu/posting.log
-rw-r--r-- 1 odroid odroid 47324 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/160/posting.log
-rw-r--r-- 1 odroid odroid 88536 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/30/posting.log
-rw-r--r-- 1 odroid odroid 24427 Jan 10 04:41 /tmp/wspr-captures/K74/2200/posting.log
-rw-r--r-- 1 odroid odroid 53565 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/630/posting.log
-rw-r--r-- 1 odroid odroid 65804 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/80/posting.log
-rw-r--r-- 1 odroid odroid 24378 Jan 10 04:41 /tmp/wspr-captures/MERGED_RX_0/60eu/posting.log
-rw-r--r-- 1 odroid odroid 172637 Jan 10 04:42 /tmp/wspr-captures/MERGED_RX_0/40/posting.log
odroid@wspr:/tmp/wspr-captures$ tail -n 20 /tmp/wspr-captures/MERGED_RX_0/40/posting.log
Fri Jan 10 04:39:59 UTC 2020: 7.039999 W4ENN -16 -27 -16p
Fri Jan 10 04:39:59 UTC 2020: 7.040049 N6KOG -28 -28p
Fri Jan 10 04:39:59 UTC 2020: 7.040050 ZS3RF -13 -13p -14
Fri Jan 10 04:39:59 UTC 2020: 7.040069 WA8KNE 1 -0 1p
Fri Jan 10 04:39:59 UTC 2020: 7.040073 KJ4IHN -23 -23p
Fri Jan 10 04:39:59 UTC 2020: 7.040106 K4APC -5 -14 -5p
Fri Jan 10 04:39:59 UTC 2020: 7.040156 N9QDS -20 -21 -20p
Fri Jan 10 04:39:59 UTC 2020: 7.040160 -12 -17 -12p
Fri Jan 10 04:42:10 UTC 2020: FREQUENCY CALL POSTED_SNR K74 K76 TOTAL=19, POSTED=11
Fri Jan 10 04:42:10 UTC 2020: 7.039999 KD9JYV -19 -22 -19p
Fri Jan 10 04:42:10 UTC 2020: 7.040000 -28 -28p
Fri Jan 10 04:42:10 UTC 2020: 7.040014 AF4MP -26 -26p
Fri Jan 10 04:42:10 UTC 2020: 7.040028 W4DJW -9 -9p -10
Fri Jan 10 04:42:10 UTC 2020: 7.040039 KK1D -22 -22p -22p
Fri Jan 10 04:42:10 UTC 2020: 7.040058 W5MWI -23 -24 -23p
Fri Jan 10 04:42:10 UTC 2020: 7.040068 VE3OWV -23 -26 -23p
Fri Jan 10 04:42:10 UTC 2020: 7.040126 VE6DAI -22 -24 -22p
Fri Jan 10 04:42:10 UTC 2020: 7.040141 KK4YEL -24 -26 -24p
Fri Jan 10 04:42:10 UTC 2020: 7.040158 W1CEW -21 -21p
Fri Jan 10 04:42:10 UTC 2020: 7.040162 W9RAN -1 -3 -1p
odroid@wspr:/tmp/wspr-captures$
You can see from that table that on 40M at 04:42 UTC rx "K76" is generally 1-3 dB more sensitive than "K74"
Thanks to Rob for a better post..... when I put that up earlier, I lacked the time to be more thorough. Here's a simple bash script I wrote to make comparing RX easier
There's also a tool/method to compare the posts (SNR ) from each of Merged receivers. This is interesting because it gives insight into angle of arrival, noise and other aspects of each receiving system. Unfortunately I've forgotten how to invoke it!
Comments
Alternatively, reboot your Pi.
I will add checks for file overflow condition in future versions.
I had wondered if it could be some sort of overflow problem as the reboot seems to have fixed it, at least in the short term.
I'll keep my eye on it.
Regards,
Martin - G8JNJ
It's done it again, the noise plots are uploading but not the WSPR spots.
I've tried rebooting but no luck.
Which logs should I inspect ?
Regards,
Martin - G8JNJ
Also, spots may not get reported if the /tmp/wspr-captures file system fills up.
Run 'watch df /tmp/wspr-captures' for several minutes to see if overflowing is a problem and increase the size of that file system if it is.
I tried that command but I'm not sure what I should be seeing.
I got the response /tmp/wspr-captures: No Such File or Directory
and then lines and lines of blocks.
I'm not a software guy, so I don't know the command string to increase the file size, and any information would need to be spelled out for me.
I'm running the 2.6 version, but I think I'll rebuild the Raspi OS from scratch and then load 2.5a instead.
Regards,
Martin - G8JNJ
I think part of the problem may be the response time of wsprnet as it's taking progressively longer to get any response back from the website. I noticed this a while ago but now it's getting to be much worse.
I think it's suffering from it's own success and all those automated systems now running that are uploading huge volumes of spots :-)
Regards,
Martin - G8JNJ
WD caches it spots until a successful upload so when wsprnet.org goes offline for 10 minutes or more WD can be caching 100 or more spots.
wsprnet.org seems to reject uploads of 100+ spots (I'm not sure what is that limit, so that is a guess). At that point WD gets into a continuous upload failure loop while its cache of spots grows larger and large.
I am going to add a limit of upload size to WD 2.6b to try to address this problem.
But the larger problem is the frailty of wsprnet.org.
I maintain a research grade database of noise data at wsprdeamon.org and I am thinking I will add a spots database as well.
WD users will be able to make real queries of their spot data and create graphs from queries.
I have one KiwiAI on a LZ1AQ loop (14Ch) and a second KiwiBBG on a Wellbrook (running 7 so I have a slot free to check the antenna).
Currently it takes a long time to really get a feel for adding a band from the more limited BBG.
As usual a very personal reason for a feature but I do wonder if someone had say three loops (+ nulls) with otherwise identical setups, they could detect an opening or atmospheric anomaly in a specific direction.
>
>If you did make a database one thing I'd really like is the ability to run two or more systems and break down which works better on a band.
>
I've started a new thread on this subject.
http://forum.kiwisdr.com/discussion/1817/using-wspr-data-to-compare-antennas
Regards,
Martin - G8JNJ
Then Jim's site shows a comparison by band of the two using: "curl www.wa2zkd.net:8088/today.html | html2text | grep KPH"
More detailed comparisons can be made at Phil's excellent site: wspr.vk7jj.com/"
Anyone else been contacted by Corrie - M0XDK - WSPRNet Admin, regarding optimising uploads ?
Edit -Rob I have copied you in on an email regarding this.
Regards,
Martin - G8JNJ
I have not received an email on this subject. My email address is in 'wsprdaemon.sh -h' output
thanks
rob
That's the address I included you on as a cc - I've just resent it please check spam etc.
Regards,
Martin - G8JNJ
Cheers
Stu
Among other improvements, this version addresses an upload problem which may be causing WD users to pound the wsprnet.org database.
So I strongly encourage existing user to upgrade to V2.6b
Should not be something like "mv ~/wsprdaemon/wsprdaemon.conf ~/wsprdaemon.save"
On my non standard VM I tried a couple of unsucessfull git pulls and obviously got it wrong (nothing changes eh) so for speed just ended just removing the whole directory (graphs and all).
Figured a days worth of QRM was fine to lose if it meant not hammering wsprnet.org
Thanks for the project
Stu
That's what I ended up doing too.
Probably a good idea to have a proper clean out anyway :-)
My thanks to Rob for the updates.
Regards,
Martin - G8JNJ
(Both kiwis are now providing audio streams to a pair of Pi4s, both now happily appearing to decode at 2.6a version.)
Get it with 'git pull'
after some time, I am again a happy user of WD installed on my VMware Workstation Player 15 (Virtual machine with Ubuntu 18 LTS).
The updates posted on Github can be easily installed on a regualar basis by "git pull"... I like that easy maintainance...
Now I have a question:
I am running 2 kiwi's and have configured them in the .conf file for configuration of WD
However what do I need to do to always spot the strongest decode of a single particular call at a given moment. Or in other words, do I need to configure the 'merge' parameter ? What is the purpose of the merge parameter and how to use it ?
Can someone give an example, please.
Currently I am testing a lot with various antennas - the problem is the heavy interference and broadband noise from local later afternoon to late evening from strong LED-Noise and strong radiation from the neighbors VDSL2 telecom connection. Trying to phase-out the noise sources without succes, unfortunately. Afternoon till late evening is almost useless for the moment.
Summer is much better here, when LED-lamps are not needed in the evenings (due to late daylight hours..)
Thanks for info about the :'merge' parameter.
Ulli, ON5KQ
"MERGED_RX_0 K74,K76 WA2ZKD FN13ed XXXXX"
K74 and K76 I declared as receivers
declare WSPR_SCHEDULE=(
"00:00
K74,2200
MERGED_RX_0,630
MERGED_RX_0,160
MERGED_RX_0,80
MERGED_RX_0,80eu
MERGED_RX_0,60
MERGED_RX_0,60eu
MERGED_RX_0,40
MERGED_RX_0,30
MERGED_RX_0,20
MERGED_RX_0,17
MERGED_RX_0,15
K74,12
MERGED_RX_0,10
"
Here is a fuller explanation and example of the use of MERGed receivers:
1) MERG_K0_K1 merges the spots decoded from KIWI_0 with those decoded from KIWI_1
2) MERG_K0_A0 merges the decodes from KIWI_0 with decodes from the analog audio output of a 10M receiver
3) MERG_K0_A1 merges the decodes from KIWI_0 with decodes from the analog audio output of a 20M receiver
This is *NOT* diversity reception where the RF or audio signals are mixed. The decodes are performed separately on the signals from each of the MERGed receivers, from which WD picks the best set of decodes to be posted to wsprnet.org.
As a configuration example, here is a section of the conf for my Pi4 with two local Kiwis and two USB audio dongles attached: When so configured, on 20M WD merges the signals received by AUDIO_1 with those from KIWI_0 and posts only one copy (the strongest) of each received signal.
Similarly on 10M WD merges the signals received by AUDIO_0 with those from KIWI_0 , and on 160M WD merges signals from KIWI_0 with those from KIWI_1.
Not only will MERGed rxs avoid double posting spots to wsprnet.org, but as importantly when WD is run at verbosity level 1 (start as ./wsprdaemon.sh -va'), WD generates decode reports each two minutes which show all the spots in a MERG rx and which spot was chosen to be posted to wsprnet.org. Here is an example from another WD site of what you can learn in those posting.log files: You can see from that table that on 40M at 04:42 UTC rx "K76" is generally 1-3 dB more sensitive than "K74"
#!/bin/bash
#
mrx=MERGED_RX_0
#
cat /tmp/wsprdaemon/$mrx/*/posting.log | grep -v FREQ | awk '{ printf ("%5s\t %2s\t %8s\t %.3f\t %9s\t %3s\t %3s\t %3s\n", $2, $3, $4, $7, $8, $9, $10, $11)}' | sort -k4,5 |sort -u
printf "%5s\t %2s\t %8s\t %4s\t %9s\t %3s\t %3s\t %3s\n" "Month" "Day" "Time" "Freq" "Call" "POST" "RX1" "RX2"