No problem, that is what QA is for. The newest code does work without the -T I've had my share of bugs and I'm sure there are more bug lurking in kiwiwspr.sh
Also, are you sure we are receiving the raw uncompressed decoded audio stream? We are seeing 2-3 dB worse WSPR SNRs from the Kiwi than from an Apache ANAN attached to the same antenna. It appears the SNR degradation is somewhere between the Kiwi's RF input and when I hand the wave file output by kiwirecorder to 'wsprd'. kiwirecorder is a small section of the processing chain, but you know it well.
The threshold option should be only needed if you want to use the squelch. The fact that you use -101 as a threshold (which a relative value) suggests that you really do not intend to use the squelch. Note that if there is no -T option, the squelch is disabled.
I saw that until this morning (in your local time) there were kiwirecorder clients on kph:8072 running for more than 60 hours.
Re compression: as long as you're using "--ncomp" compression should be turned off.
You can sanity check this as follows. With no other active connections on a Kiwi make an admin connection. The stats on the default "status" tab should say "audio 0 kB/s ... total 0 kB/s" (these numbers are bytes/second). Then make a kiwirecorder connection. With --ncomp (compression off) you should see 24 kB/s, without --ncomp (compression on) it should drop to 6 kB/s (4x difference). The stats are only updated every 10 seconds so the number take a while to settle.
I am seeing much more frequent crashes with the latest version but there is no diagnostic output from any of them. My printouts happen at odd minutes, the crashes could have happened any time in the previous two minutes.
I just noticed this orphan line on one of my ssh windows: *** stack smashing detected ***: python terminated =========== My watchdog.log file: ========= Sun Oct 7 11:50:26 PDT 2018: INFO: Schedule has changed. A new schedule has been applied: 'KPH_LF_MF_0,2200 KPH_LF_MF_0,630 KPH_LF_MF_0,160 KPH_HF_2,80 KPH_HF_2,80eu KPH_HF_2,40 KPH_HF_2,30 KPH_HF_2,20 KPH_HF_2,17 KPH_HF_2,15 KPH_HF_2,10 KPH_HF_3,80 KPH_HF_3,80eu KPH_HF_3,40 KPH_HF_3,30 KPH_HF_3,20 KPH_HF_3,17 KPH_HF_3,15 KPH_HF_3,10' Sun Oct 7 12:13:02 PDT 2018: WARNING: there is a stale capture job file with pid 2666. Deleting file ./capture.pid and starting capture Sun Oct 7 12:55:01 PDT 2018: WARNING: there is a stale capture job file with pid 3142. Deleting file ./capture.pid and starting capture Sun Oct 7 14:35:02 PDT 2018: WARNING: there is a stale capture job file with pid 3065. Deleting file ./capture.pid and starting capture Sun Oct 7 14:35:05 PDT 2018: WARNING: there is a stale capture job file with pid 2988. Deleting file ./capture.pid and starting capture
After 5 hours I have seen one of these new error messages:
Sun Oct 7 15:38:49 PDT 2018: INFO: Schedule has changed. A new schedule has been applied: 'KPH_LF_MF_0,2200 KPH_LF_MF_0,630 KPH_LF_MF_0,160 KPH_HF_2,80 KPH_HF_2,80eu KPH_HF_2,40 KPH_HF_2,30 KPH_HF_2,20 KPH_HF_2,17 KPH_HF_2,15 KPH_HF_2,10 KPH_HF_3,80 KPH_HF_3,80eu KPH_HF_3,40 KPH_HF_3,30 KPH_HF_3,20 KPH_HF_3,17 KPH_HF_3,15 KPH_HF_3,10' *** stack smashing detected ***: python terminated Sun Oct 7 17:23:02 PDT 2018: WARNING: there is a stale capture job 'KPH_HF_3,30' with pid 3453. Deleting file ./capture.pid and starting new capture
I don't know anything about Python really, but these all sound like weird runtime memory corruption problems. What's the ambient temperature of the RPi this is running on? Does the SoC have a heat sink? Airflow?
The Pi has a heat sink but no fan and was reaching 72C when the 19 wsprd jobs were running and using all the CPUs. I have moved the SW to a second Pi, PI85 which has passive heat sink. That cpu is reaching only 52C during the wsprd jobs. Lets see if there are fewer failures.
I use metal cases like Amazon product B074MSHJND, admittedly I don't have the latest PI so didn't have to make any changes to the CPU lug height. I might modify one for the later PI when I buy one but would not recommend to non-tinkerers it will stress (break) the PCB due to the extra metal shim on the 3B+ unless the lug is modified. Wouldn't not use a fan type case when the entire enclosure can be a heatsink, without moving parts, and never get more than warm. I do have a FLIRC metal case and that is OK but more money for less dissipation.
Resisting starting a new thread, I'll ask here if wsprd+kiwiwspr, like the WSPR decoder in the Kiwi, has the ability to effectively quash duplicate decodes?
I ask this having just switched the KiwiSDRs at the Northern Utah WebSDR site from the built-in wspr decoder to using kiwiwspr. Specifically, I noticed that during the daylight hours - when it is operationg - my 630 meter WSPR signal is extremely strong into this receiver (it is only 80 miles/130km away: Ain't groundwave across a giant salty lake wonderful?), often reporting >=+20 dB, which means that power mains related IMD that is 30-50 dB down results in a plethora of spurious decodes being reported on wsprnet.
The wsprd "-v" option doesn't seem be being used (as far as I can tell from the script) so I wonder if there is some way that I can suppress them?
Thanks for your post. I have looked at your reports for 14:02 and see some interesting results. The first column is the frequency offset of the spot from your carrier at 0.475373 and the second column is the signal level relative to your +22 carrier. You can see sidebands at +-60 Hz at a respectable -50/-47 dB from your carrier which are likely generated by your transmitter's AC power supply. The rest of the sidebands are +/- 23*N Hz, and those type of sidebands have been observed on other signals at KPH and elsewhere, so they are almost certainly *not* in your tx signal. Since your tx frequency is offset 37 hz from the center of the WSPR band, additional sidebands are almost certainly present at +70, +120 and +140 Hz but but not reported because the are above the band.
A group of us are investigating such phantom sideband signals which are only observable on carriers at +15 SNR or greater. At that level the sidebands are >-30 dB and thus above the WSPR decode threshold. It appears those sidebands are generated within the Kiwi, but we are still gathering data on that subject. So I think the solution to those sideband reports is in a revision of the Kiwi SW/FW.
In the attached Version 1.1d I have added a feature which permits the user to supply the wsprd command line flags in kiwiwspr.conf. The default remains '-d' (deep decode). However it isn't clear to me that the '-v' flag suppresses duplicates, nor can I find any place in the kiwi/admin where 'suppress duplicates' is configurable. If -v or other wsprd flag doesn't suppress duplicates, then I would have to enhance the code to examine the output of wsprd for dups and suppress them before posting to wsprnet.org. Modifying the output of wsprd would seem to violate one of my design goals for this project to mimic as much as possible the operation of WSJT-x WSPR mode.
In the attached V1.1d, you can define the wsprd command line flags by adding a line to the end of the kiwiwspr.conf like:
declare DECODE_CMD_FLAGS="-d -v" ### Overwrites the default '-d'. Must include "s around the flags
There are also a number of minor bug fixes in this version which I think are unlikely to be encountered by most users.
Thanks for elaborating on that (and, of course, thank you very much for your scripts!). A couple years ago I investigated this same thing and determined that what would seem to be a useful feature in the wspr software was lacking. I, too, was confused by the "-v": I interpreted its function as effectively disabling the detection of duplicates - but was puzzled that the dupes weren't being suppressed in the first place!
* * *
One issue at the Northern Utah WebSDR - and, I'm sure other sites - is that there is a bit of "space modulation" of the local E-field due to power lines, putting a very slight AC bias on the (DC coupled) antenna, its balun, and any magnetics in the signal path. In the case of the antenna at that site, it is normally well-balanced, but due to rain and contamination, leakage sometimes appears - likely on both the antenna itself, which has hundreds of wires and insulators (it's a TCI-530 wire omni log periodic, about a 94 foot tower) and the nearby power lines - and this would cause a strong enough mains-frequency bias on one of the RF input transformers that caused a low-level mains frequency modulation on all received signals, the effect being terrible at AM BCB and below but disappearing by the time one got to 5 MHz. The measured current of this induced signal was very low, but apparently it doesn't take much at all to to turn a ferrite device into a low-level reluctance modulator. The "fix" for this was to install a 50 kHz high-pass filter on the receive antenna's feedline. There may be similar effects elsewhere with little that can be done about it.
I suspect that a similar field exists at my home QTH with the "tophatted vertical" arrangement of my LF/MF transmit antenna where it is picking up fields in the urban morass, causing a bit of modulation in... who knows what? In monitoring my own signal, I can also hear "digital noises" way in the background of the generated WSPR audio signal - perhaps artifacts in a DDS/NCO somewhere: My transmitter (an FT-817) uses at least two DDSs in the production of the RF (the IF BFO and the main LO via PLL multiplication synthesis) and it's possible that this could account for at least some of the non-mains-frequency-related low-level sidebands.
Now that we have fixed kiwiwspr.sh, a group of us volunteering at KPH are trying to clean up the RF coming from the antenna systems.
Our 530 seems pretty clean, although your observations suggest we should run a vector analyzer on it to see if there are any broken or corroded wires.
More troublesome for us has been the RF coming from the Marconi T feeding our LF/MF Kiwi. During daytime I see a little -90 dBm IMD at 1780 from 680+1100 and -105 dBm at other nearby 10 Khz frequencies, but in the evening there is -80dBm IMD every 10 Khz above 1700 to 3000+ and probably in LF/MF as well. Since a 10 dB attenuator at on the T's feed line in the station reduces daytime IMD and carrier levels equally, it appears the IMD is being generated in the T or the feed line from the T. But it is very strange that the IMD would be worse at night when I see no signals stronger than those during the day.
Fortunately I have the help of RF engineers much more experienced than I in these matters, so we hope to soon have the both antennas cleaned up.
One this to keep in mind is that any device that is susceptible to non-linearity can fall victim to the net energy applied. Sometimes even a group of lesser strong signals can sum to drive something into compression. Stations who shift their pattern at night can be an issue, even though they may also reduce power, they may re-pattern and be aimed right at you.
In using kiwiwspr, I've noticed that the scheduling doesn't seem to work: It seems only to do the "daytime" schedule (below) 24/7. I'm wondering if I've missed a permission somewhere? (The scripts were installed under a user, not root.)
Perhaps I'm dense, but I'm not quite sure what the "CAPTURE_JOBS" array is supposed to be for based on its description alone. Based on the example in the default kiwiwspr.conf file, it looks as though it's supposed to contain all of the possible KIWI,BAND combinations that would ever be used.
Your array printouts look fine to me, but is your watchdog deamon running? Verify that is it with '-w s' and run '-w l' to look at your watchdog.log for logged schedule changes. CAPTURE_JOBS is no longer used by kiwiwspr.sh , so you can remove references to it from your kiwiwspr.conf
CAPTURE_JOBS functionality has been incorporated into arrays defined in the four files derived from kiwiwspr.conf:
1) 'suntimes' which contains the sunrise and sunset times for the grid defined for the first Kiwi in KIWI_LIST. suntimes is updated by the watchdog daemon once a day or any time there is a change to kiwiwspr.conf
2) kiwiwspr.sched where WSPR_SCHEDULE is converted to HHMM_SCHED in which sunrise and sunset schedules are converted to HH:MM times. If the user doesn't specify a schedule for 00:00, the latest (in time) HH:MM schedule is copied into a new 00:00 schedule so the daemon knows what jobs should be run between 00:00 and the earliest configured job. The result is that the same schedule runs through the night. kiwiwspr.sched is updated any time the kiwiwspr.conf or suntimes files are changed. Naming consistency suggests this file should be called hhmm.sched.
3) 'expected.jobs' contains EXPECTED_JOBS which is the list of jobs defined in kiwiwspr.sched which should be running now. It is updated any time the daemon determines that a new set of jobs should be running. It depends of current time and changes in kiwiwspr.conf or suntimes
4) 'running.jobs' contains RUNNING_JOBS which is the list of currently active jobs. Any time expected.jobs changes, the daemon compares each job in RUNNING_JOBS against all the EXPECTED_JOBS. A job is terminated (both the recording and decoding daemons are killed) If it finds the job is no longer scheduled, then any newly scheduled jobs are started.
Thanks for that, Rob: I somehow missed that the WDT didn't start with everything else, but reading between the lines I can see that to be so and it started with the "-w a" arguments - and it kinda says as much in the help.
The "00:00" makes sense, being a convenient and easy way to work around the time-of-day roll-over: I've used that sort of trick before so I feel silly for not figuring that, myself.
I'll see how things go - but they look good - Thanks again for your help!
Has anyone got an install guide from say Debian net install to working Kiwirecorder and WSPR script? I set up a small virtual machine rather than update the previously used PI, started with headless Debian net install (64Bit) but I found it was a right pain to find all the dependencies for WSJT-X and was not sure if the Ubuntu guides were 100 compatible as they were often adding Ubuntu PPA's.
I found an Ubuntu guide which helped with things I surprisingly failed to guess like "sudo apt install libqt5multimedia5-plugins libqt5serialport5 libfftw3-single3 libqt5printsupport5 libgfortran3" - https://linuxer.eu/wsjt-x-ubuntu-18-04-installation/ Later I had some issue with the kiwiwspr script not finding kiwirecorder.py or other files in the directory above the script and just powered the machine down for the evening. I don't consider my self very competent with Linux but I can normally install software without having to bounce around websites and forum posts searching errors, for hours.
What would be great is a list of steps from a completely vanilla (even headless) Ubuntu or Debian to working setup, in one place, too much to ask?
kiwiwspr requires only the 'wsprd' binary from the WSJT-x distribution. When I was unable to compile the WSJT-x sources on the odroid, I copied wsprd from the Pi on to the odroid and it ran perfectly.
Its a Debian VM on VMware ESXi or Ubuntu, I'm not bothered, can be anything but I have an ESXi host running so though it would be a fifteen minute job. You're (respectfully) missing the point, IMHO it would be great to have a run through of steps to get from a bare ISO install to a working script for anybody. If I get this going it won't help anyone but me, it should be a simple list of steps, it might be that there are steps which work on an install that has been going for a while but will be missing stuff on a first install. I'm also known as Stu, I'm not really that grumpy.
Lets say x64, if I have VM's they are generally x64. SDRPlay do a PI image, gets someone going with their kit and a base PI as quickly as writing the image and booting, I'd actually prefer to export myself a clean an OVA file I could import to ESXi/Virtualbox as I can't see the point of another PI running when I have a host on all the time for other stuff. Have you ever used TurnkeyLinux? excellent for getting a machine going with the software you want to test in moments, literally. I'll have to check Shackbox for a laugh see what that has going on.
Anyway sorry for hijacking the thread, if you were thinking of doing a guide that would make an excellent sticky now the 2+6 mode is here.
I have already invested several man months in kiwiwspr and I think it is in a state where a user with a normal linux skill set can install and configure it. Creating installation images is beyond my skill set, so I am hoping that someone with the time and those skills would take on that task if there is a widespread desire for it.
If it works OK with a normal Linux skill set it should be easy to list the steps and have others check those actually work on a fresh install. Yesterday I was looking for a file that someone had included as part of a software build but it was (only) on their machine, the software "always built OK for them", not so much for the guy that had to test build in another location on the authors day off. I sound ungrateful and I don't mean to be, it's just that really like the work you have done, with the Kiwi it is a force for RF monitoring. I'd just like to be able to use it knowing I've the current version and a text file to copy paste into putty to go from fresh install to working so that if I want to do the same elsewhere or wipe a "too fiddled-with" VM I can.
Comments
A fix is on the way.
I've had my share of bugs and I'm sure there are more bug lurking in kiwiwspr.sh
Also, are you sure we are receiving the raw uncompressed decoded audio stream? We are seeing 2-3 dB worse WSPR SNRs from the Kiwi than from an Apache ANAN attached to the same antenna. It appears the SNR degradation is somewhere between the Kiwi's RF input and when I hand the wave file output by kiwirecorder to 'wsprd'. kiwirecorder is a small section of the processing chain, but you know it well.
The threshold option should be only needed if you want to use the squelch. The fact that you use -101 as a threshold (which a relative value) suggests that you really do not intend to use the squelch. Note that if there is no -T option, the squelch is disabled.
I saw that until this morning (in your local time) there were kiwirecorder clients on kph:8072 running for more than 60 hours.
Today's fix is in service at KPH.
You can sanity check this as follows. With no other active connections on a Kiwi make an admin connection. The stats on the default "status" tab should say "audio 0 kB/s ... total 0 kB/s" (these numbers are bytes/second). Then make a kiwirecorder connection. With --ncomp (compression off) you should see 24 kB/s, without --ncomp (compression on) it should drop to 6 kB/s (4x difference). The stats are only updated every 10 seconds so the number take a while to settle.
I just noticed this orphan line on one of my ssh windows:
*** stack smashing detected ***: python terminated
===========
My watchdog.log file:
=========
Sun Oct 7 11:50:26 PDT 2018: INFO: Schedule has changed. A new schedule has been applied: 'KPH_LF_MF_0,2200 KPH_LF_MF_0,630 KPH_LF_MF_0,160 KPH_HF_2,80 KPH_HF_2,80eu KPH_HF_2,40 KPH_HF_2,30 KPH_HF_2,20 KPH_HF_2,17 KPH_HF_2,15 KPH_HF_2,10 KPH_HF_3,80 KPH_HF_3,80eu KPH_HF_3,40 KPH_HF_3,30 KPH_HF_3,20 KPH_HF_3,17 KPH_HF_3,15 KPH_HF_3,10'
Sun Oct 7 12:13:02 PDT 2018: WARNING: there is a stale capture job file with pid 2666. Deleting file ./capture.pid and starting capture
Sun Oct 7 12:55:01 PDT 2018: WARNING: there is a stale capture job file with pid 3142. Deleting file ./capture.pid and starting capture
Sun Oct 7 14:35:02 PDT 2018: WARNING: there is a stale capture job file with pid 3065. Deleting file ./capture.pid and starting capture
Sun Oct 7 14:35:05 PDT 2018: WARNING: there is a stale capture job file with pid 2988. Deleting file ./capture.pid and starting capture
Sun Oct 7 15:38:49 PDT 2018: INFO: Schedule has changed. A new schedule has been applied: 'KPH_LF_MF_0,2200 KPH_LF_MF_0,630 KPH_LF_MF_0,160 KPH_HF_2,80 KPH_HF_2,80eu KPH_HF_2,40 KPH_HF_2,30 KPH_HF_2,20 KPH_HF_2,17 KPH_HF_2,15 KPH_HF_2,10 KPH_HF_3,80 KPH_HF_3,80eu KPH_HF_3,40 KPH_HF_3,30 KPH_HF_3,20 KPH_HF_3,17 KPH_HF_3,15 KPH_HF_3,10'
*** stack smashing detected ***: python terminated
Sun Oct 7 17:23:02 PDT 2018: WARNING: there is a stale capture job 'KPH_HF_3,30' with pid 3453. Deleting file ./capture.pid and starting new capture
Linux PiKPH 4.14.67-v7+ #1139 SMP Wed Aug 29 15:17:05 BST 2018 armv7l GNU/Linux
pi@PiKPH:/tmp/kiwi-captures $ python -c 'import sys,numpy; print(sys.version); print(numpy.__version__); numpy.__config__.show()'
2.7.13 (default, Nov 24 2017, 17:33:09)
[GCC 6.3.0 20170516]
1.12.1
lapack_info:
libraries = ['lapack', 'lapack']
library_dirs = ['/usr/lib']
language = f77
lapack_opt_info:
libraries = ['lapack', 'lapack', 'blas', 'blas']
library_dirs = ['/usr/lib']
language = c
define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
openblas_lapack_info:
NOT AVAILABLE
blas_info:
libraries = ['blas', 'blas']
library_dirs = ['/usr/lib']
define_macros = [('HAVE_CBLAS', None)]
language = c
atlas_3_10_blas_threads_info:
NOT AVAILABLE
atlas_threads_info:
NOT AVAILABLE
atlas_3_10_threads_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
atlas_3_10_blas_info:
NOT AVAILABLE
atlas_blas_threads_info:
NOT AVAILABLE
openblas_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['blas', 'blas']
library_dirs = ['/usr/lib']
language = c
define_macros = [('NO_ATLAS_INFO', 1), ('HAVE_CBLAS', None)]
blis_info:
NOT AVAILABLE
atlas_info:
NOT AVAILABLE
atlas_3_10_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
pi@PiKPH:/tmp/kiwi-captures $ pip list installed
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
automationhat (0.0.4)
beautifulsoup4 (4.5.3)
blinker (1.3)
blinkt (0.1.1)
buttonshim (0.0.2)
Cap1xxx (0.1.3)
chardet (2.3.0)
Cheetah (2.4.4)
click (6.6)
colorama (0.3.7)
cryptography (1.7.1)
cycler (0.10.0)
decorator (4.0.11)
drumhat (0.1.0)
enum34 (1.1.6)
envirophat (0.0.6)
ExplorerHAT (0.4.2)
Flask (0.12.1)
fourletterphat (0.1.0)
functools32 (3.2.3.post2)
gpiozero (1.4.1)
html5lib (0.999999999)
idna (2.2)
ipaddress (1.0.17)
itsdangerous (0.24)
Jinja2 (2.8)
keyring (10.1)
keyrings.alt (1.3)
lxkeymap (0.1)
lxml (3.7.1)
MarkupSafe (0.23)
matplotlib (2.0.0)
mcpi (0.1.1)
microdotphat (0.1.3)
mote (0.0.3)
motephat (0.0.2)
networkx (1.11)
numpy (1.12.1)
oauthlib (2.0.1)
pantilthat (0.0.5)
phatbeat (0.1.0)
pianohat (0.1.0)
picamera (1.13)
picraft (1.0)
piglow (1.2.4)
pigpio (1.38)
Pillow (4.0.0)
pip (9.0.1)
pyasn1 (0.1.9)
pycrypto (2.6.1)
pygame (1.9.3)
pygobject (3.22.0)
pygraphviz (1.4rc1)
pyinotify (0.9.6)
PyJWT (1.4.2)
PyOpenGL (3.1.0)
pyOpenSSL (16.2.0)
pyparsing (2.1.10)
pyserial (3.2.1)
python-dateutil (2.5.3)
pytz (2016.7)
pyxdg (0.25)
PyYAML (3.12)
pyzmq (16.0.2)
rainbowhat (0.0.2)
requests (2.12.4)
requests-oauthlib (0.7.0)
RPi.GPIO (0.6.3)
RTIMULib (7.2.1)
scipy (0.18.1)
scrollphat (0.0.7)
scrollphathd (1.2.0)
SecretStorage (2.3.1)
sense-emu (1.0)
sense-hat (2.2.0)
setuptools (33.1.1)
simplejson (3.10.0)
six (1.10.0)
skywriter (0.0.7)
sn3218 (1.2.7)
spidev (3.3)
subprocess32 (3.2.7)
touchphat (0.0.1)
twython (3.4.0)
unicornhathd (0.0.3)
urllib3 (1.19.1)
webencodings (0.5)
Werkzeug (0.11.15)
wheel (0.29.0)
wxPython (3.0.2.0)
wxPython-common (3.0.2.0)
pi@PiKPH:/tmp/kiwi-captures $
pi@PiKPH:/tmp/kiwi-captures $ cat KPH_HF_2/40/capture.log
XXX lineno: 411, opcode: 36
Fatal Python error: GC object already tracked
pi@PiKPH:/tmp/kiwi-captures $
I have moved the SW to a second Pi, PI85 which has passive heat sink. That cpu is reaching only 52C during the wsprd jobs.
Lets see if there are fewer failures.
Wouldn't not use a fan type case when the entire enclosure can be a heatsink, without moving parts, and never get more than warm.
I do have a FLIRC metal case and that is OK but more money for less dissipation.
I ask this having just switched the KiwiSDRs at the Northern Utah WebSDR site from the built-in wspr decoder to using kiwiwspr. Specifically, I noticed that during the daylight hours - when it is operationg - my 630 meter WSPR signal is extremely strong into this receiver (it is only 80 miles/130km away: Ain't groundwave across a giant salty lake wonderful?), often reporting >=+20 dB, which means that power mains related IMD that is 30-50 dB down results in a plethora of spurious decodes being reported on wsprnet.
The wsprd "-v" option doesn't seem be being used (as far as I can tell from the script) so I wonder if there is some way that I can suppress them?
Thanks,
Clint
KA7OEI
Thanks for your post. I have looked at your reports for 14:02 and see some interesting results. The first column is the frequency offset of the spot from your carrier at 0.475373 and the second column is the signal level relative to your +22 carrier. You can see sidebands at +-60 Hz at a respectable -50/-47 dB from your carrier which are likely generated by your transmitter's AC power supply.
The rest of the sidebands are +/- 23*N Hz, and those type of sidebands have been observed on other signals at KPH and elsewhere, so they are almost certainly *not* in your tx signal. Since your tx frequency is offset 37 hz from the center of the WSPR band, additional sidebands are almost certainly present at +70, +120 and +140 Hz but but not reported because the are above the band.
A group of us are investigating such phantom sideband signals which are only observable on carriers at +15 SNR or greater. At that level the sidebands are >-30 dB and thus above the WSPR decode threshold. It appears those sidebands are generated within the Kiwi, but we are still gathering data on that subject.
So I think the solution to those sideband reports is in a revision of the Kiwi SW/FW.
pi@KPH-Pi2:~ $ cat ka7oei.log | sort -k4,4n | awk '{printf"%4d %3d %s\n", (($4-0.475737)*1000000), -(22 - $5), $0}'
-141 -36 2018-10-09 14:02 KA7OEI 0.475596 -14 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
-120 -36 2018-10-09 14:02 KA7OEI 0.475617 -14 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
-71 -47 2018-10-09 14:02 KA7OEI 0.475666 -25 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
-60 -52 2018-10-09 14:02 KA7OEI 0.475677 -30 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
-47 -45 2018-10-09 14:02 KA7OEI 0.475690 -23 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
-24 -44 2018-10-09 14:02 KA7OEI 0.475713 -22 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
0 0 2018-10-09 14:02 KA7OEI 0.475737 +22 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
22 -38 2018-10-09 14:02 KA7OEI 0.475760 -16 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
46 -43 2018-10-09 14:02 KA7OEI 0.475784 -21 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
60 -47 2018-10-09 14:02 KA7OEI 0.475797 -25 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
69 -46 2018-10-09 14:02 KA7OEI 0.475807 -24 0 DN40ao 0.5 KA7OEI-1 DN31uo 115 346
pi@KPH-Pi2:~ $
In the attached Version 1.1d I have added a feature which permits the user to supply the wsprd command line flags in kiwiwspr.conf. The default remains '-d' (deep decode).
However it isn't clear to me that the '-v' flag suppresses duplicates, nor can I find any place in the kiwi/admin where 'suppress duplicates' is configurable.
If -v or other wsprd flag doesn't suppress duplicates, then I would have to enhance the code to examine the output of wsprd for dups and suppress them before posting to wsprnet.org. Modifying the output of wsprd would seem to violate one of my design goals for this project to mimic as much as possible the operation of WSJT-x WSPR mode.
In the attached V1.1d, you can define the wsprd command line flags by adding a line to the end of the kiwiwspr.conf like:
declare DECODE_CMD_FLAGS="-d -v" ### Overwrites the default '-d'. Must include "s around the flags
There are also a number of minor bug fixes in this version which I think are unlikely to be encountered by most users.
Rob
Attachments:
https://forum.kiwisdr.com/uploads/Uploader/e3/d0d3e164fd69c6d06da3ca41e319b7.sh
Thanks for elaborating on that (and, of course, thank you very much for your scripts!). A couple years ago I investigated this same thing and determined that what would seem to be a useful feature in the wspr software was lacking. I, too, was confused by the "-v": I interpreted its function as effectively disabling the detection of duplicates - but was puzzled that the dupes weren't being suppressed in the first place!
* * *
One issue at the Northern Utah WebSDR - and, I'm sure other sites - is that there is a bit of "space modulation" of the local E-field due to power lines, putting a very slight AC bias on the (DC coupled) antenna, its balun, and any magnetics in the signal path. In the case of the antenna at that site, it is normally well-balanced, but due to rain and contamination, leakage sometimes appears - likely on both the antenna itself, which has hundreds of wires and insulators (it's a TCI-530 wire omni log periodic, about a 94 foot tower) and the nearby power lines - and this would cause a strong enough mains-frequency bias on one of the RF input transformers that caused a low-level mains frequency modulation on all received signals, the effect being terrible at AM BCB and below but disappearing by the time one got to 5 MHz. The measured current of this induced signal was very low, but apparently it doesn't take much at all to to turn a ferrite device into a low-level reluctance modulator. The "fix" for this was to install a 50 kHz high-pass filter on the receive antenna's feedline. There may be similar effects elsewhere with little that can be done about it.
I suspect that a similar field exists at my home QTH with the "tophatted vertical" arrangement of my LF/MF transmit antenna where it is picking up fields in the urban morass, causing a bit of modulation in... who knows what? In monitoring my own signal, I can also hear "digital noises" way in the background of the generated WSPR audio signal - perhaps artifacts in a DDS/NCO somewhere: My transmitter (an FT-817) uses at least two DDSs in the production of the RF (the IF BFO and the main LO via PLL multiplication synthesis) and it's possible that this could account for at least some of the non-mains-frequency-related low-level sidebands.
TNX and 73,
Clint
Thanks for the insights into the TCI-530.
Now that we have fixed kiwiwspr.sh, a group of us volunteering at KPH are trying to clean up the RF coming from the antenna systems.
Our 530 seems pretty clean, although your observations suggest we should run a vector analyzer on it to see if there are any broken or corroded wires.
More troublesome for us has been the RF coming from the Marconi T feeding our LF/MF Kiwi. During daytime I see a little -90 dBm IMD at 1780 from 680+1100 and -105 dBm at other nearby 10 Khz frequencies, but in the evening there is -80dBm IMD every 10 Khz above 1700 to 3000+ and probably in LF/MF as well. Since a 10 dB attenuator at on the T's feed line in the station reduces daytime IMD and carrier levels equally, it appears the IMD is being generated in the T or the feed line from the T. But it is very strange that the IMD would be worse at night when I see no signals stronger than those during the day.
Fortunately I have the help of RF engineers much more experienced than I in these matters, so we hope to soon have the both antennas cleaned up.
73,
Rob
Perhaps I'm dense, but I'm not quite sure what the "CAPTURE_JOBS" array is supposed to be for based on its description alone. Based on the example in the default kiwiwspr.conf file, it looks as though it's supposed to contain all of the possible KIWI,BAND combinations that would ever be used.
In the WSPR_SCHEDULE array, I have this:
### Daytime schedule
"sunrise+00:30 NUT_KIWI1,630 NUT_KIWI1,30 NUT_KIWI1,17 NUT_KIWI1,40 NUT_KIWI2,20 NUT_KIWI2,15 NUT_KIWI2,10 NUT_KIWI2,12"
### Nighttime schedule
"sunset-00:30 NUT_KIWI2,2200 NUT_KIWI1,630 NUT_KIWI1,80 NUT_KIWI1,80eu NUT_KIWI1,30 NUT_KIWI2,160 NUT_KIWI2,40 NUT_KIWI2,20 NUT_KIWI2,10"
While in the kiwiwspr.sched I have this:
declare HHMM_SCHED=( \
"00:00 NUT_KIWI2,2200 NUT_KIWI1,630 NUT_KIWI1,80 NUT_KIWI1,80eu NUT_KIWI1,30 NUT_KIWI2,160 NUT_KIWI2,40 NUT_KIWI2,20 NUT_KIWI2,10" \
"08:14 NUT_KIWI1,630 NUT_KIWI1,30 NUT_KIWI1,17 NUT_KIWI1,40 NUT_KIWI2,20 NUT_KIWI2,15 NUT_KIWI2,10 NUT_KIWI2,12 " \
"18:31 NUT_KIWI2,2200 NUT_KIWI1,630 NUT_KIWI1,80 NUT_KIWI1,80eu NUT_KIWI1,30 NUT_KIWI2,160 NUT_KIWI2,40 NUT_KIWI2,20 NUT_KIWI2,10 " \
)
I'm not quite sure where the "00:00" schedule is coming from.
Any suggestions?
Thanks.
Verify that is it with '-w s' and run '-w l' to look at your watchdog.log for logged schedule changes.
CAPTURE_JOBS is no longer used by kiwiwspr.sh , so you can remove references to it from your kiwiwspr.conf
CAPTURE_JOBS functionality has been incorporated into arrays defined in the four files derived from kiwiwspr.conf:
1) 'suntimes' which contains the sunrise and sunset times for the grid defined for the first Kiwi in KIWI_LIST. suntimes is updated by the watchdog daemon once a day or any time there is a change to kiwiwspr.conf
2) kiwiwspr.sched where WSPR_SCHEDULE is converted to HHMM_SCHED in which sunrise and sunset schedules are converted to HH:MM times. If the user doesn't specify a schedule for 00:00, the latest (in time) HH:MM schedule is copied into a new 00:00 schedule so the daemon knows what jobs should be run between 00:00 and the earliest configured job. The result is that the same schedule runs through the night. kiwiwspr.sched is updated any time the kiwiwspr.conf or suntimes files are changed. Naming consistency suggests this file should be called hhmm.sched.
3) 'expected.jobs' contains EXPECTED_JOBS which is the list of jobs defined in kiwiwspr.sched which should be running now. It is updated any time the daemon determines that a new set of jobs should be running. It depends of current time and changes in kiwiwspr.conf or suntimes
4) 'running.jobs' contains RUNNING_JOBS which is the list of currently active jobs. Any time expected.jobs changes, the daemon compares each job in RUNNING_JOBS against all the EXPECTED_JOBS. A job is terminated (both the recording and decoding daemons are killed) If it finds the job is no longer scheduled, then any newly scheduled jobs are started.
The "00:00" makes sense, being a convenient and easy way to work around the time-of-day roll-over: I've used that sort of trick before so I feel silly for not figuring that, myself.
I'll see how things go - but they look good - Thanks again for your help!
I set up a small virtual machine rather than update the previously used PI, started with headless Debian net install (64Bit) but I found it was a right pain to find all the dependencies for WSJT-X and was not sure if the Ubuntu guides were 100 compatible as they were often adding Ubuntu PPA's.
I found an Ubuntu guide which helped with things I surprisingly failed to guess like "sudo apt install libqt5multimedia5-plugins libqt5serialport5 libfftw3-single3 libqt5printsupport5 libgfortran3" - https://linuxer.eu/wsjt-x-ubuntu-18-04-installation/
Later I had some issue with the kiwiwspr script not finding kiwirecorder.py or other files in the directory above the script and just powered the machine down for the evening.
I don't consider my self very competent with Linux but I can normally install software without having to bounce around websites and forum posts searching errors, for hours.
What would be great is a list of steps from a completely vanilla (even headless) Ubuntu or Debian to working setup, in one place, too much to ask?
Grumpy of GrumpyLinuxshire
You're (respectfully) missing the point, IMHO it would be great to have a run through of steps to get from a bare ISO install to a working script for anybody.
If I get this going it won't help anyone but me, it should be a simple list of steps, it might be that there are steps which work on an install that has been going for a while but will be missing stuff on a first install.
I'm also known as Stu, I'm not really that grumpy.
SDRPlay do a PI image, gets someone going with their kit and a base PI as quickly as writing the image and booting, I'd actually prefer to export myself a clean an OVA file I could import to ESXi/Virtualbox as I can't see the point of another PI running when I have a host on all the time for other stuff.
Have you ever used TurnkeyLinux? excellent for getting a machine going with the software you want to test in moments, literally.
I'll have to check Shackbox for a laugh see what that has going on.
Anyway sorry for hijacking the thread, if you were thinking of doing a guide that would make an excellent sticky now the 2+6 mode is here.
Creating installation images is beyond my skill set, so I am hoping that someone with the time and those skills would take on that task if there is a widespread desire for it.
Yesterday I was looking for a file that someone had included as part of a software build but it was (only) on their machine, the software "always built OK for them", not so much for the guy that had to test build in another location on the authors day off.
I sound ungrateful and I don't mean to be, it's just that really like the work you have done, with the Kiwi it is a force for RF monitoring.
I'd just like to be able to use it knowing I've the current version and a text file to copy paste into putty to go from fresh install to working so that if I want to do the same elsewhere or wipe a "too fiddled-with" VM I can.