missing samples [not just a Kiwi problem]
I searched but I didn't find anything. I hope this isn't a duplicate.
I've been seeing gaps in the audio sample stream. The gaps seem to be on the order of 30 samples. [Edit: on the order of 128 samples at 48 ksps.] They happen randomly. They seem to happen on average every 8 seconds. But there could be three gaps in a second or no gaps in 30 seconds.
If you are listening to voice or music you'll hardly notice them. To make them more noticeable tune to an AM station with little sideband content. Such as WWV. Then select USB mode and tune below the signal so that the carrier is about in the middle of the passband. You'll hear the strong tone but with occasional glitches. You might even see them in the waterfall. [Edit: It's not visible in the waterfall.]
I've tried this on two different computers. I see it happen on every receiver/server.
Has anybody else seen this? Does anybody know what is happening?
Comments
This has been a problem in the past, but is not known to be an issue currently.
The most sensitive test of dropped samples is to run the FAX extension. The slightest dropped sample will cause the image to become progressively more shifted (i.e. lines looking broken) as the image rendering proceeds. I tried this just now on several Kiwi running the most recent software version and all seemed well. An example image is below. All those lines should not have any offsets or breaks. The 400 pixel tall FAX viewport at 120 LPM represents 3.33 min of reception. So no dropped samples over that time period.
Have you tried different browsers? One factor in the audio is that the browser's scheduling of the Javascript audio processing must not have any realtime performance hits. Otherwise samples can be lost. But that's a browser problem not a Kiwi problem.
The audio and waterfall use completely different mechanisms and so any glitching should not be correlated. If they are that may point to network issues.
Hello, John.
The dropouts seem to be about 2 ms in length. Timewise they seem to be randomly distributed with an average of about 1 dropout per 8 seconds. I see the same using two different computers and with both Chrome and Edge browsers. Attached are pictures of a couple Audacity captures. The kiwisdr screen shot shows the settings used for the Audacity shot with the purer constant tone.
I don't think you would see it in a fax. A dropout would be about 0.4% the width of one line. And they would happen on average every 16th line. And the fax decoder is continuously adjusting its sync, isn't it.
It seems to me that there are basically two conditions in which one would notice the gaps: listening to a pure tone in single sideband or passing the audio to an external PSK demodulator.
STANAG 4285 would be an example of such a PSK (8PSK) signal. A 2 ms gap would interfere with four symbols (and knock it a bit out of sync). These gaps wouldn't be fatal to a STANAG signal given the 10 second interleaver that is typically selected. The gaps would make weaker signals harder to copy.
Slightly off the topic - look how clean your weather fax is! 😮
About the FAX extension: It does not use the per-line sync signal. It relies on the precise known sample rate for timing (even though that rate is not a fixed number due to how the Kiwi frequency calibration works -- but that's another story). This is also why there is a manual horizontal framing adjustment. There is a "page stop" detector based on received tones.
I guess there's a more basic question as to whether you're seeing missing samples causing the subsequent samples to be shifted in time, or simply suppressed amplitude of samples not altering the timing. Probably more likely to be the latter? The other possibility is a phase reversal at that point. But your first image doesn't look like that. Just suppressed amplitude.
How are you recording? Browser audio through a virtual sound card to an app? Kiwi record function to a browser downloaded file? Kiwirecorder Python app to a file or netcat stream?
Let me do some similar recording here and examination with an audio processing program (Sound Studio on Mac in my case).
@smg From a development Kiwi-2 kindly hosted by Martin at his fantastic Wessex South West UK site which has the most incredible reception: wessex.zapto.org:8074
One day.... one day.......... :-)
If we are losing 2.6 ms of samples every 8 seconds then by the end of a 1200 line fax we should be off by 0.39 lines. Could that have been compensated for in the calibration factor and manual framing control without being aware of it?
I use Audacity and set the recording input as Stereo Mix (Realtek (R) Audio). I can hear the gaps even when Audacity is not running.
I've done some more measurements too. Every event is the same length. They are about 2.6 ms. The samples after the event are shifted by 2.6 ms. At certain frequencies that results in a 180 degree phase shift. Images attached. I get those waveforms by tuning below the carrier of WWV in USB mode and narrowing the passband (as best I can) to just include the carrier. If I tune 400 Hz below the carrier then I get a 400 Hz tone.
The offset frequencies that give 180 phase shift are those where an odd half multiple of the period is equal to the gap. f = (2n + 1) / (2 * 2.6ms) I.e. 192, 577, 961, 1346 Hz,...
Another thought occurs to me. Could this be a rate adapter underrun? The adapter is sending out samples at one rate but they are coming into the adapter at a slightly lower rate. Every once in a while the buffer underruns? But if that were the case I would expect the underruns to happen at a pretty regular interval, not the somewhat random interval that I am seeing.
(For the images I copied part of the continuous section and pasted it next to the gap to compare phase before and after.)
I need to know the answers to these questions: How are you recording? Browser audio through a virtual sound card to an app? Kiwi record function to a browser downloaded file? Kiwirecorder Python app to a file or netcat stream?
I looked at a couple of Kiwi (one local, one far) by using the Kiwi record function (record button left of mute button) which downloads a .wav file at the approx. 12 kHz sample rate. Doing so factors out the Javascript code that does upsampling from 12 kHz to the native audio rate the OS is expecting (e.g. 44.1 or 48 kHz). I looked through an entire minute of samples in each case and didn't see anything amiss.
Under/overruns of the resample buffer are reported in the control panel, "Stat" tab, "Audio" line. The system audio rate is shown followed by the audio queue size. If the size goes to zero there's an underrun and if it goes too high an overrun. But usually such an even produces a very large disturbance in the audio. Not just a few msec of drop.
(This is using) HP desktop or Lenovo laptop. (Windows 11 Home)
(Connecting to kiwi radio server via) Chrome or Edge.
I'm using the sound system that is native to the computer. Both computers describe it as Realtek. (Is that a virtual sound card?)
I'm using Audacity to record what is going out to the speakers ("Stereo Mix"). But Audacity is not a factor because I hear the things that Audacity is showing even before Audacity is brought into the picture.
I used the record button as you described. While I heard gaps in the live sound as it was recording, the recording is _clean_. When Media Player plays the .wav there are no gaps. Audacity shows no gaps in the .wav.
So that is kind of pointing at the javascript upsampler, isn't it?
(Thank you for your help with this.)
[Under Stat: Audio: I'm seeing it bounce between 48.0 and 48.1 kb/s (ksps?) and Qlen 5 and 6.]
So I would guess this is an issue with Windows and Chrome/Edge not being able to deliver the Javascript realtime requirements for the resampler. The fact that you're not getting any Kiwi under/overruns and the Qlen is stable kind of confirms that.
This entire project is built on the idea that browsers live up to their promise of providing stable, well documented platforms for delivering content/media. And over the years I've been constantly disappointed in the delivery on that promise. I guess I shouldn't be surprised.
Whats the alternative in these cases? The use of SDR Console and alike?
@smg No. The entire point of this project is for the Kiwi to not be yet-another-IQ-generator using external PC software. And all the associated headaches. There are plenty of existing solutions for that.
If ever I feel I have no alternative but to go that way then that's the day I pull the plug. There's no point in continuing.
OK. Thanks for your help, John.
I confess I was skeptical that the root cause was in Chrome/Edge/javascript. But then I tried WebSDR and OpenWebRX and they do exactly the same thing. They are totally different code, right? I'm totally blown away by all this.
I would like to hear from other Windows users. They should easily be able to reproduce the symptoms. Set USB mode, tune to a strong AM station without a lot of business in the sidebands (e.g. WWV), and narrow the passband to pass mostly just the carrier.
One more point. If 128 zero samples were occasionally inserted into the stream it would give the same symptoms. (At 48 ksps 128 samples is 2.666 ms.)
Yes, totally different code. Same principles though.
Unfortunately none of this surprises me. A number of years ago Chrome had a bad audio problem. There was someone else who went to battle on the Chrome bug tracker, so I just sat back and waited. It was an ugly fight, but months later Chrome finally admitted there was a problem and fixed it. I was much too busy to get involved directly. I just told everyone to use a different browser temporarily.
Interesting. Thanks. :)
Perfect!
There are a number of very lightweight browsers around, and not all built on Chrome.
I have used a few of these in the past. (I'm a die hard Firefox guy)
Mozilla Firefox - Win / Linux / MacOS
https://www.mozilla.org/
Librewolf - Win / Linux / MacOS
https://librewolf.net/
GNOME Web (Epiphany) - Linux
https://wiki.gnome.org/
Falkon - Lunix
https://www.falkon.org/
I notice you are connecting to your Kiwi via the proxy address. Do you see the same dropouts with a local connection (if possible)
Oh. I suppose those are directed at me.
I had Firefox but hadn't tried it. If Chrome and Edge can't hack it why would I think Firefox would be different. Well golly gee. I just tried with Firefox and it works. No glitches. Strange, but good to know.
I wasn't aware I was going through proxies. I just clicked on the links in the linkfanel map. I see that some of the links are to proxies. Some are not. They all act the same.
So from what I understand and have read, the underlying code for Firefox (called Gecko) is much better at handling media streams than Chromium based browsers.
Regarding proxies, and devices in the public maps, those are all streaming across the internet from far and wide, and some packet loss is to be expected.