Kiwirecorder resample problem [improved in most recent kiwirecorder]

n8ooun8oou
edited December 2023 in Problems and Issues

Back in 2019 i ask about resampling the output of Kiwirecorder to 6000 which is required by the Argo program. A change was made to Kiwirecorder to add the -r parameter which has worked fine up to last night. Last night with the -r 6000 set, the file output by Kiwirecorder was at 6011 samplerate. Of course Argo does not like that number. Files that I created two days before were ok at 6000.

This morning I tried running on my Kiwi again and got the 6011 file created. I simply changed the url to point to a remote Kiwi and got the expected 6000 samplerate file needed. Has a recent update introduced this problem? My Kiwi has not been updated in the last 7 days. On 10/10/23 I did download the latest Kiwiclient from github. My previous version of Kiwiclient downloaded on 01/14/22 produces the 6011 samplerate now from my Kiwi. I rerun Kiwirecorder dropping the -r parameter down by 1 until 5992. The output file remained at 6011. at 5991 the output went to 5972.

Help please,, can we get Kiwirecorder back to consistent output?


Thanks

Mike N8OOU

Comments

  • WA2ZKDWA2ZKD

    kiwirecorder resampler does not work as it should due to an issue with the related python libarary AFAIK. I pipe through sox to get what I want.

  • n8ooun8oou

    WA2ZKD

    Sox and several other external applications were discussed back in 2019. Yes I agree, Sox is an option. The -r option was added to Kiwirecorder back then. That provided a simple, no extra step process. My point is for the last 4 years Kiwirecorder has worked with no failure. It worked two days ago It does not work now, even though I think I am using the same software/Kiwi.

    I don't have the DSP or Python expertise to debug the why. That is why I am posting the problem and requesting assistance. If the resample option never worked in Kiwirecorder, I might agree with you. Being that it has worked, and I use it often, I am hoping the "experts" can find the tweak that is now causing the issue and I can get 4 more years of good results.


    Mike N8OOU

  • WA2ZKDWA2ZKD

    To the best of my knowlege -r never worked right in kiwirecorder. At best, it would do integer math. There may be more to this puzzle than kiwirecorder. Which kiwi system are you sourcing the wav file from?


    -Jim

  • jksjks
    edited October 2023

    Does your Kiwi have constant GPS reception? If not, it's possible the ADC XO has drifted due to temperature variation or something (uncompensated by the GPS) and is now producing an output sample rate that is triggering the bug in the python-samplerate library.

    There was a long thread about this a month ago: https://forum.kiwisdr.com/index.php?p=/discussion/comment/16689#Comment_16689/p1

    It seems there are a couple of ways kiwirecorder could compensate for the python-samplerate bug. But it will take some work. I certainly don't have any time at the moment.

  • n8ooun8oou

    JKS,

    Yes, I follow the threads fairly closely, and I did follow that one. I just checked the Kiwi and I currently am tracking 11, good 7, and have 365.7k fixes. My GPS antenna is blocked to the South a little, but these are the kind of numbers that I normally see.

    I understand your priorities, I would like to have a new Kiwi TOO! If I can do anything here to assist in this debugging I will help where I can.


    Mike N8OOU

  • jksjks
    edited October 2023

    Alright. Let me take a quick look. I suggested before that simply incorporating the python-samplerate source code into kiwiclient, and applying Christoph's fix, might be the simplest thing to do. That will either be simple or difficult due to the amount of support code needing to be stripped out.

  • n8ooun8oou

    JKS,

    I have an update which is good news, but maybe bad news. I did a power cycle on the Kiwi, and I am writing 6000 sps again. The bad side of that is what ever caused the problem likely went into the bit bucket. A few more runs continued making 6k files.

    if I have the issue again is there anything I can capture from the kiwi to help troubleshoot? Rebooting is an easy fix.

    Mike

  • jksjks
    edited October 2023

    Well, I worked all day on this. But it needed to be done. And as a side-effect I got a few things straightened out on my laptop that I'd been postponing.

    So please update your kiwiclient. The new version will use a local copy of the samplerate module containing Christoph's fixes if it has been built. See the README file for build and test instructions.

    If you can't build it then the code will fallback to using the prior workaround for the standard pip3 installed python-samplerate. And, as before, if that is not found it will fallback to low-quality linear interpolation resampling. Or you can simply pipe to sox's high quality resampler as Jim suggests.

    I have tested building the local samplerate module on macOS x86_64(intel) and arm64(m1). If someone gets it working on Windows please let us know.

  • n8ooun8oou

    JKS

    Again, thank you for taking your time and spending some of it on KiwiClient/Recorder (KCR") I grabbed the new version from github, and reinstalled following the steps in the readme.me file.

    Problem 1. I run Linux Mint 20.3 which when I run the step to apt install libsamplerate it reports that the package cannot be found. Looking into this closer I found this related info;

    https ://ubuntu.pkgs.org /20.04/ ubuntu-main-amd64/libsamplerate0_0.1.9-2_amd64.deb.html

    I had to add some spaces to that link to make your forum happy. I used the package name of libsamplerate0 and received the result that package was all ready installed.

    The samplerate compile and test steps worked as documented.


    Problem 2. I then run kiwirecorder to make the 6k .wav file, which run successful. I made additional runs adjusting the -r parm down in size. The output file now remains at 6k down to an -r 5989. At -r 5988 the output file becomes 5977.

    I started studying the Kiwirecorder source code, (I am a retired programmer, but not in c/python). I first find that I need to add --log debug to allow additional error messages to be printed. I then could see the message that "KCR" was resampling from 12001.1 to 5977.1.

    A sample rate of 5977.1 just can't be done! it is either 5977 or 5978. Yes, I do see the floating point calculations, and fully understand the precision implications. I don't know the exact why, but with floating point involved in the calculations I understand why specifying -r 5993 gets 6000 output, or -r5988 gets 5977.

    Here is the real problem I think. Ultimate scientific precision verses real life practicality. The Argo program expects the input file to be exactly 6000. It does not care if the original file was 12001.1 divided by 2 to get 6000 with the 1.1/2 samples dropped. The 1.1/2 samples that are lost won't impact the graphic Argo makes. I understand other uses may require better accountability of the 1.1/2 sample.

    I also saw in the code, error messages in the "warn" category documenting the reversion to the simpler resample methods. The next question was how do I make that happen.

    Problem 3. More code reading gets me to a completely undocumented parameter "Environment Variable USE_LIBSAMPLERATE=True/False"

    When I set the variable to "False", I find that "KCR" outputs a file with the exact sample rate as specified by the -r parm. (Based on limited testing -r5988 output 5988, -r5994 output 5994, -r6000 output 6000.) What ever the "KCR" code is doing in this case, Argo is happy with the 6k input file, and I don't see any issue with it's output.

    I think at the very minimum the readme.me and makefile need a caution about using the -r parameter if not fully documenting what happens behind the scene.

    I am sorry for this long post, but feel it is necessary for clear communications.

    Mike

  • jksjks

    The apt install libsamplerate in the GitHub README for Linux is more of suggestion as the exact details always vary. But glad you got it to work.

    What you describe about the sample rate numbers indicates that the latest Kiwi version of samplerate is not being used. Rather, the previous code which only allows limited sample rate values to get around the buffering bug in python-samplerate.

    Without changing anything about --log you should see a message at the beginning indicating which of the 3 resamplers is being used:

    • Using Kiwi high-quality samplerate module.
    • CAUTION: using python-samplerate instead of Kiwi samplerate module containing fixes.
    • CAUTION: libsamplerate not available; low-quality linear interpolation will be used for resampling.

    The code automatically switches to the second and third resamplers depending on what it finds: If the Kiwi sample rate module was compiled and installed successfully and if the python-samplerate module was installed successfully.

    I ran -r at 6000, 5989 and 5988 using Kiwi samplerate and got the expected results:

    Using Kiwi high-quality samplerate module.

    -r 6000: resampling from 12001.1 to 6000 Hz (ratio=0.499956)

    -r 5989: resampling from 12001.1 to 5989 Hz (ratio=0.499039)

    -r 5988: resampling from 12001.1 to 5988 Hz (ratio=0.498956)

    And sox --info *.wav via make info reports the correct file sample rates:

    Sample Rate  : 6000

    Duration   : 00:00:05.29 = 31744 samples ~ 396.8 CDDA sectors

    File Size   : 63.5k

    Bit Rate    : 96.1k


    Sample Rate  : 5989

    Precision   : 16-bit

    Duration    : 00:00:05.29 = 31682 samples ~ 396.752 CDDA sectors

    File Size   : 63.4k

    Bit Rate    : 95.9k


    Sample Rate  : 5988

    Precision   : 16-bit

    Duration    : 00:00:05.29 = 31682 samples ~ 396.819 CDDA sectors

    File Size   : 63.4k

    Bit Rate    : 95.9k

    However, if I force Kiwi samplerate not to be used (simulating a compile/install failure) I get what you describe:

    -r 6000: resampling from 12001.1 to 6000.53 Hz (ratio=0.500000)

    -r 5989: resampling from 12001.1 to 6000.53 Hz (ratio=0.500000)

    -r 5988: resampling from 12001.1 to 5977.09 Hz (ratio=0.498047)


    Sample Rate  : 6000

    Precision   : 16-bit

    Duration    : 00:00:05.27 = 31601 samples ~ 395.012 CDDA sectors

    File Size   : 63.2k

    Bit Rate    : 96.1k


    Sample Rate  : 6000

    Precision   : 16-bit

    Duration    : 00:00:05.27 = 31601 samples ~ 395.012 CDDA sectors

    File Size   : 63.2k

    Bit Rate    : 96.1k


    Sample Rate  : 5977

    Precision   : 16-bit

    Duration    : 00:00:05.27 = 31477 samples ~ 394.977 CDDA sectors

    File Size   : 63.0k

    Bit Rate    : 95.7k

    And similarly for the low-quality linear interpolation resampler if USE_LIBSAMPLERATE is set False.

    So the real question here is why doesn't the Kiwi resampler work for you? Do you see the message "Using Kiwi high-quality samplerate module" when you run? When you ran make samplerate_test did you get something that looked reasonably error free?

  • n8ooun8oou

    JKS

    Well DRAT!!!

    I must admit my mistake of not updating my test script to point to the version I installed on 10/15. I was still executing the 10/10 version on all the tests I run on Sunday and Monday, I corrected the script, and in the couple tests I have done I now see your resample messages, and get the 6000 sample rate on the output file

    mike@casper:~/kiwiclient101023$ ./n8oourec ## 10/10 version with -r 6000

    capture 184 at 192.168.0.9 to /home/mike/ws5m/n8oou20231017_204633

    Block: 000000b1, RSSI: -60.5   ## note... there are no messages

    mike@casper:~/kiwiclient101023$ cd ..

    mike@casper:~$ cd kiwiclient101523

    mike@casper:~/kiwiclient101523$ ./n8oourec ## 10/15 version with -r 6000

    capture 184 at 192.168.0.9 to /home/mike/ws5m/n8oou20231017_204728

    Block: 00000001, RSSI: -60.5 2023-10-17 20:47:28,822 pid 13557 Using Kiwi high-quality samplerate module.

    2023-10-17 20:47:28,822 pid 13557 resampling from 12001.1 to 6000 Hz (ratio=0.499955)

    Block: 000000b1, RSSI: -60.6

    mike@casper:~/kiwiclient101523$ nano n8oourec ##edit script to change -r 6000 to -r 5988

    mike@casper:~/kiwiclient101523$ ./n8oourec ## 10/15 version with -r 5988

    capture 184 at 192.168.0.9 to /home/mike/ws5m/n8oou20231017_205046

    Block: 00000001, RSSI: -60.5 2023-10-17 20:50:47,525 pid 13591 Using Kiwi high-quality samplerate module.

    2023-10-17 20:50:47,525 pid 13591 resampling from 12001.1 to 5988 Hz (ratio=0.498956)

    Block: 000000b1, RSSI: -60.5

    mike@casper:~/kiwiclient101523$


    I will do some additional testing but I believe the 10/15 version does correct the output size issue. I still believe a couple more lines in the readme or makefile is needed. The Environment Variable does need to be documented. It may be a good thing to have the simple resample method selectable.

    To answer your final questions: Why doesn't it work for me?....It does if I point my script to the corrected program. Did I see the High Quality Message? No, Here is what "make samplerate_test" says on both versions.

    mike@casper:~/kiwiclient101523$ make samplerate_test

    pytest --capture=tee-sys samplerate/tests/test_samplerate.py

    ================================================== test session starts ==================================================

    platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0

    rootdir: /home/mike/kiwiclient101523

    collected 40 items                                                                                                     


    samplerate/tests/test_samplerate.py ........................................                                     [100%]


    ================================================== 40 passed in 0.09s ===================================================

    mike@casper:~/kiwiclient101523$ cd ..

    mike@casper:~$ cd kiwiclient101023

    mike@casper:~/kiwiclient101023$ make samplerate_test

    make: *** No rule to make target 'samplerate_test'. Stop.

    mike@casper:~/kiwiclient101023$


    Thanks for reading. I believe I am good for the moment.

    Mike

  • jksjks

    Okay, I will mention the USE_LIBSAMPLERATE env variable in the README file.

    I only added the display of which resampler is being used in 10/15 version. So that's why you don't see it in your 10/10 version.

Sign In or Register to comment.