Some things I have noticed [problems with a large DX/label database]

Comments

• jks

  January 2018

  Do things get better if you reboot the Beagle? Not just restart the Kiwi server but reboot Linux running on the Beagle entirely. My Kiwi didn't exhibit any of the problems you mentioned (my dx.json file is only 50 kB). But the server depends on Linux keeping out-of-the-way as much as possible.

  
  

  Below is a dump of the virtual memory statistics after 12 days of Linux uptime (not Kiwi server uptime) and after a fresh reboot. After 12 days up there was still plenty of free memory (out of 500 MB total). No swapping, or excessive interrupts or more than one process running compared to the statistics after a reboot. More memory had been tied down in buffers and the cache but that's not unusual.

  
  

  The same can be said for the computer and browser you're using. You might see if the problems clear on a full reboot. On very rare occasions (like twice a year) I've seen Firefox get into a weird state where audio problems start occurring. They cleared when I restarted the browser. I know this sounds like voodoo you'd get on a phone call to first-level tech support, but it's true in a lot of cases.

  
  

  Also worth doing is a careful ping test over a period of time. Look at not just the average ping times but the maximum as well to try and catch cases of bad latency behavior which could be caused by any number of networking problems.

  
  

  root@kiwi:~/Beagle_SDR_GPS# uptime

   17:37:03 up 12 days, 16:45,  1 user,  load average: 1.00, 1.01, 1.05

  root@kiwi:~/Beagle_SDR_GPS# vmstat 1

  procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa

   1  0      0 247208  26436 159760    0    0     1     2   21   11 89  5  6  0

   1  0      0 247208  26436 159760    0    0     0     0  588   18 88 12  0  0

   1  0      0 247208  26436 159760    0    0     0     0  577   26 87 13  0  0

   1  0      0 247208  26436 159760    0    0     0     0  583   14 89 11  0  0

   1  0      0 247208  26436 159760    0    0     0     0  579   16 88 12  0  0

  
  

  (reboot)

  
  

  root@kiwi:~/Beagle_SDR_GPS# uptime

   17:39:23 up 1 min,  1 user,  load average: 0.57, 0.21, 0.07

  root@kiwi:~/Beagle_SDR_GPS# vmstat 1

  procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa

   1  0      0 419116   6692  26832    0    0   333    10  319  314 31  8 59  3

   1  0      0 419116   6692  26836    0    0     0     0  580   32 90 10  0  0

   1  0      0 419116   6692  26836    0    0     0     0  576   32 87 13  0  0

   1  0      0 419116   6692  26836    0    0     0     0  592   38 87 13  0  0

   1  0      0 419116   6700  26828    0    0     0    48  605   50 89 11  0  0

  
  
• G8JNJ

  January 2018 edited January 2018

  Hi John,

  
  

  OK I'll try and grab some stats.

  
  

  It's currently like this

  
  

  procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

   1  0      0 369292   9224  62696    0    0     1     0  384  549 94  4  2  0  0

   1  0      0 369260   9224  62720    0    0     0     0  387  544 94  5  1  0  0

   1  0      0 369260   9224  62720    0    0     0     0  356  439 94  5  1  0  0

  
  

  Although I turn my PC off each evening and fully reboot the KiWi's every time I update the software (or do other maintenance tasks) in order to reset the Ant_Switch GPIO ports.

  
  

  Regards,

  
  

  Martin - G8JNJ

  
  

  
  
• G8JNJ

  January 2018

  Hi John,

  
  

  A quick check at WF6 level with three successive page up / right arrow button presses to tune HF, results in ~3 seconds worth of 'stuttering'

  
  

  Regards,

  
  

  Martin - G8JNJ
• jks

  January 2018 edited January 2018

  Well, I don't know -- works fine for me. Anyone else have problems?

  
  

  I tried it on the world's worst laptop: an hp 650 (Pentium) using Windows 10, running the latest build (1709 16299.192). And Firefox 57.0.4 ("Quantum"). I had to use a wired Ethernet connection because the latest W10 update permanently killed the WiFi driver. I have no FF plugins running and a perfmon says about 50% cpu use when connected to a Kiwi. Since the laptop screen is relatively small I used the FF zoom function to double the screen size to effectively move more bits during a page up/down. It worked fine. I could hit page/up down in quick succession or at any other speed and the audio kept going. The waterfall was always delayed slightly until I stopped paging. I tried several zoom levels.

  
  

  If you look at the "stats" tab are you getting audio underruns while paging? Alternatively, open another tab and look at the log tab on the admin page for underrun messages. If not then it isn't the Kiwi failing to produce audio under these conditions but the browser, or something in the path between the Kiwi and the browser, failing to keep up. What connection does the host running the browser have to the Kiwi? Is there WiFi anywhere along the path? Try a wired Ethernet connection.

  
  
• G8JNJ

  January 2018

  Hi john,

  
  

  I'm using Win 7 Pro 64 bit.

  
  

  "look at the "stats" tab are you getting audio underruns while paging? "
  

  
  

  The audio stats 'freeze' whilst the audio is stuttering.

  
  

  "look at the log tab on the admin page for underrun messages"

  
  

  No underrun messages seen.

  
  

  "Is there WiFi anywhere along the path? Try a wired Ethernet connection."

  
  

  It's all wired back to the main VDSL router.

  
  

  I've just dug out an old laptop running XP and Chrome. The stuttering at zoom level 6 when I perform one page up, lasts for nearly six seconds. 

  
  

  I'm wondering if the stuttering and dx tag editing are related.

  
  

  I think tomorrow I'll try replacing the dx.json file with something smaller and see if that changes the behaviour in any way.

  
  

  Are there any more commands or logging options I can set via the console to more specifically try and trap some of this ?

  
  

  Regards,

  
  

  Martin - G8JNJ

  
  

  
  

  
  

  
  

  
  

  
  

  
  

  
  
• jks

  January 2018

  Can you email me your dx.json file as a text attachment? I'm beginning to think it's related to that.

  
  
• G8JNJ

  January 2018

  Hi John,

  
  

  It's associated with the 'density' of DX tags in the chosen zoom level.

  
  

  I now have a lot of tags  >5,000.

  
  

  I edited the dx.json file and stripped out all the tags between 500KHz and 29MHz leaving me with a file size of 290 lines = 19KB

  
  

  I tried tuning across 10MHz to 20MHz at various zoom levels and it was instantaneous with no audio 'suttering' 

  
  

  However as soon as I got the 400KHz NBD band where there are lots of tags within a small frequency range, it started 'suttering' again. As soon as I tuned below this band where there are relatively few tags 'per MHz', it improved once again. 

  
  

  Zooming in and out on the NDB band produced different duration 'stuttering' so it definitely seems to be related to the number of tags that are in view at any particular zoom level. I think zoom 6 is the worst, because as you zoom out further a lot of the tags are omitted and no longer displayed.

  
  

  This is also the case when adding new tags. If there are already a lot of tags on the screen, there is a delay and some audio 'stuttering' when a new tag is committed. The same test in a part of the frequency range where there are no tags is instantaneous, with no loss of audio.

  
  

  So I think this is fairly conclusive evidence that it's the number of tags, per MHz, per zoom setting, that is the problem.

  
  

  Another thought - how does the KiWi sort between the dx.json and dx.min.json files ? Could this be slowing things down too ?

  
  

  Regards,

  
  

  Martin - G8JNJ

  
  

  
  
• jks

  January 2018

  dx.min.json is not referenced by the software. It's just included in the distribution as a "cleaner" version of dx.json. There was a request for this I believe.

  
  

  The current label system was never designed to handle thousands of entries. So I guess this behavior doesn't surprise me.

  
  
• G8JNJ

  January 2018 edited January 2018

  Hi John,

  
  

  "The current label system was never designed to handle thousands of entries."

  
  

  Agreed it's partially down to the way in which I use the KiWi.

  
  

  I like to spot 'odd 'transmissions - those that are unusual or unidentified.

  
  

  I typically spend about 1/2 hour each day clunking up the frequency range looking for stuff that I haven't already labelled. If I spot something new I'll tag it.

  
  

  By doing this I can quickly spot new signals or ID fleeting ones.

  
  

  The downside is that I've heard a lot of signals and added a lot of tags. So this is why I think I'm noticing this and others are not (yet).

  
  

  Maybe it would be possible to have a two tier tag system. At the moment it's not easy to edit the band markers, and even if I did it would be over written on the next update.

  
  

  As a thought, maybe it could be possible to add a second 'higher level' tier of 'marker' tags instead of the current band markers. 

  
  

  These new markers could have upper and lower frequency limits and different colour options, and could be placed where the current band markers reside. 

  
  

  The new markers could be used to identify ITU allocations such as broadcast, marine, aeronautical, amateur and others, pretty much like the existing band markers.

  
  

  When the KiWi is zoomed out >9 to display a wide frequency range, only these 'allocation' markers would be visible. But perhaps if the KiWi was tuned to a frequency that already has an individual tag, that tag could be displayed at this zoom level along with any accompanying notes. 

  
  

  The current 'tier' of tags would then only need to be visible at close in zoom levels around 9 or 10.

  
  

  I think this would allow users to more quickly identify specific 'bands of interest' (such as marine) without the current 'blur' of tags, and then be able to zoom in for more detailed information (such as individual channel allocations).

  
  

  Basically it's all your fault John......................................

  
  

  You designed something that works too well and causes 'tagging' addiction :-)

  
  

  Regards,

  
  

  Martin - G8JNJ
• Earl

  January 2018

  Hello,
  i have the same Problem on my Laptop ThinkPad T61 Intel Core2 Duo Processor T5870. With Windows 7 and Firefox. I have many NDB Labels on my Kiwi. When i go in the NDB Utility Area and zoom in stocks the Kiwi Screen for a long while. But on my other Laptop Thinkpad X220 with I7 Processor Win 7 all OK. On the X220 Notebook all runs great.
  
  Regards
  
  Sven DC9FD
  
  
• jks

  January 2018 edited February 2018

  Today's release, v1.168, has some improvements to the speed of dx labels during normal operation. Please let me know how the changes work for you. Problems during on-screen label editing are still being considered.

  
  

  
• G8JNJ

  February 2018

  Hi John,

  
  

  The change has made a huge difference no sign of any audio 'stuttering' and a very fast and smooth scroll up and down in frequency at all zoom levels.

  
  

  Thanks for that :-)

  
  

  Regards,

  
  

  Martin - G8JNJ

  
• Earl

  February 2018

  Hi John,
  on my notebooks works the new Version very fine. No stockings and very Fast.
  Thanks for that.
  
  Regards
  
  Sven DC9FD