Audio timing walkabout [FT8 decoder problems solved with tighter limits on audio buffering]

VR2BGVR2BG
edited March 2019 in Problems Now Fixed
I've dumped all the auto-run WSPR & am trying to use the Kiwi for monitoring FT8 on another computer until I have time to mess with the RPi.

It works fine for a bit, but after a while (sometimes a few minutes, sometimes much longer) the presented audio shifts in time too far for JTDX (like WSJT-X) to work with. If I then select the bookmark in my browser again for the frequency I'm monitoring, the presented audio jumps back to around the 1-2 sec delay.

Here it can be seen - JQ1UGE's CQs started to decode after 04:31:45.



This on a Lenovo X220 i7, XP SP3 & FF 52.9.0esr.

Any suggestions?

73, VR2BG.

Comments

  • VR2BGVR2BG
    Was watching A41ZZ doing alternating CQs on 17 & 15m today & on 15m (received with the Kiwi) after a while, his transmissions started to drift later & later whilst on 17m (received with another SDR) they didn't.

    I've also seen a few instances where re-selecting the bookmark for what the Kiwi was tuned to did NOT result in the additional delay going away. Re-selecting the bookmark a second or even once for a third time, would. Everything arrives around a second earlier when the Kiwi decides to cooperate.

    73, VR2BG.
  • WA2ZKDWA2ZKD
    any Kiwi will have a delay in the RF to audio path, it's the nature of the design. It is not like other SDR in that regard.
  • VR2BGVR2BG
    Yes, understood there will be some latency - nature-of-beast. But what's happening is that over time, the delay increases. Even without any users or anything else going on with the Kiwi, the delay eventually builds up to be too much. I wonder if others have seen the same thing? I often see folks using my Kiwi to listen to FT8 (based on the frequency tuned to), but have no idea if they experience it, too. Although I don't see how, being trailing-edge with XP here is often suggested to be the root of all untoward behavior experienced with other stuff...

    73, VR2BG.
  • KA7UKA7U
    I suggest you clear your browser cache and check again.
    Ron
    KA7U
  • VR2BGVR2BG
    Thanks for the suggestion, Ron. Did that, exited & restarted the browser & after a while the audio was again lagging by the additional second or so that kills copy of FT8.

    The other day I switched from VB-Audio to VirtualAudioCable to get the audio into JTDX (as with VAC I can have two instances of JTDX on the same "cable", so I might catch a bit of what JT9/JT65 activity is still out there) & it seemed the problem went away as it kept working for a number of hours - but come morning, I woke up to find the audio was again an additional second later.

    I have used both VB-Audio & VAC with multiple instances of JTDX (and sometimes also WSJT-X) on that computer 24/7 back in KH6 with the other SDR, and never seen this creeping delay - so perhaps what happened after switching to VAC was simply a coincidence... when I first started monitoring FT8 on the Kiwi, it did take quite some time (better part of a day?) before the lag increased & I noticed it wasn't copying anymore. The problem seems to happen when it feels like it. Compared to KH6, the computer is loafing now as when I'm there it's my only machine so is chock-a-block doing all sorts of other stuff.

    Grrrrrrr.

    73, VR2BG.
  • jksjks
    So the problem here is that FT8 has a much greater time synchronization requirement (i.e. "wall clock" time) than WSPR does. The timing of the audio hitting FT8 must match what time the FT8 program is getting from the host operating system within a second or two at most.

    The Kiwi produces a delay in the audio stream because it has some buffering. Buffering is required because it is the only defense against latency/interruption issues in the network, particularly when the audio is being delivered by the Internet over long distances with lots of potential points of interruption. So you are already being disadvantaged by the time delay of this buffering to begin with. Any additional cumulative delay by other software behaving badly (e.g. sample rate compensation by the VAC) may push the total timing over the FT8 limit.

    One thing you could try is reducing the Kiwi buffer size to reduce the fixed delay. The penalty for doing this is that you will have much less immunity to short-term network interruptions. That means it's important to make sure there are no highly variable latency devices in the path between the Kiwi and computer running the browser, like a WiFi router or a cheap Ethernet switch that might behave badly with heavy traffic on the other ports.

    Add "abuf=n.nn" to the URL where n.nn is a number in seconds of the minimum audio buffer size. Start with 0.5 and experiment with a range from 0.3 to 0.8

    It would also be interesting to know on the control panel "Stats" tab what the value of the Audio "Qlen" (queue length) is when you are having problems versus what the value is when starting (this value will vary when the abuf= value is changed)
    G0LUJWA2ZKDKA7UHB9TMCrz3dvp
  • sv1dhsv1dh
    GA jks. Can you explain a bit further how to perform your advice " Add "abuf=n.nn" to the URL " to test it with my KiWi, since I have same problem with VR2BG...
    73 SV1DH
  • WA2ZKDWA2ZKD
    see http://kiwisdr.com/quickstart/index.html#id-user-tune
  • VR2BGVR2BG
    Everything here is on the same switch that's also a WiFi AP for the home that's not got a lot of activity, so I'd be tempted to think that wasn't the source of the creeping additional delay. I didn't realize VAC did sample rate conversion. It's set for a range of rates & I thought it worked it at the rate of the source provided it was within that range. If the creeping delay came from VAC or VB-Audio, it's interesting they don't seem to add the delay to the other SDRs I'm running.

    Whatever it is, yes, all combined the delays can put the audio too late for decoding referenced to real time. W9MDB has a utility to let you offset the clock in order to copy those not synced & too far off, but it seems it doesn't run on XP. Before I try shortening the buffer, I'll watch Qlen. The Kiwi is on a dead 15m now, so dunno if they audio has indeed slipped... Qlen started at a steady 5 & now after about 15 minutes or so is 7, with the occasional 6 or 8.

    SV1DH: Thanks for confirming you observe the same behavior. Which OS/browser/"cable" are you using?

    73, VR2BG.
  • sv1dhsv1dh
    GA VR2BG and tnx for sharing ur experience about this problem...I using WIN7 X64, FIREFOX QUANTUM 65.0.2 and the virtual cable is VB-Audio Virtual Cable, driver 1.0.3.5/2014
  • sv1dhsv1dh
    GM WA2ZKD and tnx for related link, but the above proposed by jks " abuf= " parameter is NOT on the link list of parameters...Can I use it instead, pse??
  • jksjks
    Yes, "abuf" is not on that list because it's undocumented (not mentioned because of its potential to cause more problems than it solves). But you use it in the same way.

    You can also specify the parameter "audio" (or "audio=1") and every time you change frequency in the browser a message in console window will show you the measured delay between the change in audio on the Kiwi and the resulting audio change detected at the browser. Effectively measuring the audio delay of the entire path (Kiwi + network + browser/javascript). You'll note there is a lot of jitter in this number because there are so many variables involved.
  • sv1dhsv1dh
    jks, MANY TNX again! Will try it and reply here...
    73 SV1DH
  • jksjks
    I didn't realize VAC did sample rate conversion.
    It may or may not do large-scale sample rate conversion, but it always has to do small-scale compensation via sample rate adjustment ("pitch bending"). This is because there are always small differences in the input and output sample rates even though they are the same, e.g. 44.1 or 48 kHz. The Kiwi might be high by 0.01% and the "sound card", however it is implemented, might be low by 0.01%. If there is not an adjustment then the VAC buffer will slowly grow until it overflows or shrink until it underflows. And of course the absolute timing of the output samples will be offset by the size of the buffer.
  • VR2BGVR2BG
    Okay - have been able to get away with abuf=0.6 or 0.7, though eventually Qlen creeps up to 8-9 after starting around 4-5. Now have another machine running Ubuntu 18.04 LTS & also see the problem with it. Seems how long it takes can vary, but generally within an hour or two it's too much. I've yet to run the other SDRs long enough for this rate difference thing to become noticeable, let alone a problem.

    73, VR2BG.
  • jksjks
    edited March 2019
    When you specify a minimum with "abuf=min" the maximum by default is set to min * 3. I think this is too much for FT8. For min=0.6 max would be 1.8 seconds. I just released v1.273 which gives abuf an optional form "abuf=min,max" so you could try something like "abuf=0.6,1" and see if that behaves any better.
  • VR2BGVR2BG
    Stats shows audio sps jumping around quite a bit more on the Windows machine than the Ubuntu one - as well as 44.1k instead of 48k.
  • jksjks
    The first number of the two "audio sps" numbers is the one second averaged transfer rate of audio data coming over the network as seen by the Javascript code. This number has a high amount of jitter because the network traffic and processor scheduling of when the Javascript code runs is completely at the mercy of the OS process scheduling (Windows, Linux) and how the browser and specific Javascript implementation schedule events. There is nothing I can do about this other than to buffer the traffic to prevent over/underruns, i.e. not being able to supply the audio to the audio-out side when it is required.

    The OS/browser determines if a 48 or 44.1 kHz audio output rate is required and tells me what I must provide. I don't get a choice and I am required to do sample rate conversion in the Javascript code. I've seen cases where some OS/browser combinations request 22.05 kHz. So I have to be able to support sub-multiple rates as well. And we haven't even talked about decompression or the additional filtering that has to be done.
  • VR2BGVR2BG
    For a while there, it seemed like when the audio was 44.1 things were worse than at 48k, but with different hardware, different OSs & being tuned to different bands, there were too many variables to conclude anything...

    With 1.273, finding abuf=0.7,1.1 avoids under runs & places all but those with really late clocks within the decoding window. It's great being able to turn it loose & not have to keep checking on it all day - tnx!

    73, VR2BG.
  • vk5qivk5qi
    Is there a minimum value to the abuf setting? My main aim here being to lower the latency as much as possible, accepting the potential for unreliability. (In this case, the KiwiSDR in question is only 14ms away, via a very good link).

    I've been able to drop the latency from ~1.5 seconds to ~1 second by using abuf=0.3,0.5. Just wondering if it's possible to drop this even further...
  • jksjks
    The abuf min/max values have to be between 0.25 and 5.0, inclusive.
  • rrobinetrrobinet
    Running v1.333, after adding '&audio=1' I see no extra /var/log/messages lines. Should I be looking elsewhere?
  • jksjks
    Not the log, the browser Javascript console.
  • rrobinetrrobinet
    edited September 2019
    OK, got that now. It appears the delay numbers are in msecs. Is that correct?
  • jksjks
    Yes. The measurement scheme is itself subject to some jitter. So the results are not precise.
  • frohrofrohro

    I ran into this problem using Ubuntu 20.04 making a pulseaudio virtual audio sink a couple days ago. It was worse than when I used the actual playing audio channel. It usually didn't work for long at all. I'll have to try this solution, and see if it helps.

    Thanks for the pointer!

    Rob

    KL7NA

  • HB9TMCHB9TMC

    I stubled across a handy solution to compensate buffer delay with wsjt-x.

    Under linux you can run a command with "faketime", so it runs on a time offset. For example:

    faketime -f '-2.1s' wsjtx

    But obviously one doesn't want to transmit like that.

  • G8JNJG8JNJ

    I use BktTimeSync (free) with Windows, and set an offset of -1 second, which seems to be good for use with most KiWi's.

    https://www.maniaradio.it/en/bkttimesync.html

    Regards,

    Martin

Sign In or Register to comment.