Sound recording from multiple opened browse windows for diversity reception?

zfyoungzfyoung
in KiwiSDR Discussion
Hi, I like to know if there is any way to grab the IQ samples from multiple browse windows and record them onto wave files for diversity reception? I don't want to mix all sound samples from different SDRs, but individual source to individual sound track.

Best Regards
zfyoung
«1

Comments

  • jksjks
    I didn't originally write the code, but with changes to kiwiclient (kiwirecorder.py) you could open multiple connections to a Kiwi and record IQ samples from each connection separately: https://github.com/jks-prv/kiwiclient/tree/jks-v0.1

  • zfyoungzfyoung
    Thanks for your fast response, jks. I'll take a look into the URL your provided and see if I can make it work.
  • zfyoungzfyoung
    Hi, I've looked into the python code. Unfortunately this code is not working for IQ recording (other modes are fine), and since I'm new with python, I would greatly appreciate any expert in that regard help to add a patch to that script?

    Regards
    zfyoung
  • jksjks
    I received the patch for IQ recording just 4 days ago. I looked at the changes but I haven't tried it myself. Does it crash or just give you a file that doesn't seem to have the correct IQ data?


  • zfyoungzfyoung
    No, it just stuck before it could print out the "Block" and "RSSI" values and went on to generate wave file. I looked into the script and it turns out the callee "process_iq_samples" in "kiwiclient.py" defined as "pass" ie. void? I think a few added commands should be suffice to get it up and running. Another potential issue may be related to stereo/mono format with regard to IQ mode?
  • jksjks
    edited November 2017
    On my Mac with "Python 2.7.3 -- EPD_free 7.3-2 (32-bit)" I was able to do this:

    python kiwirecorder.py -s kiwisdr.local -p 8073 -f 1521 -m am

    python kiwirecorder.py -s kiwisdr.local -p 8073 -f 1521 -m iq



    And generate these two files:

    20171118T225528Z_1000000_am.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 12001 Hz

    20171118T225615Z_1000000_iq.wav: RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, stereo 12001 Hz


    They played back fine, but the iq file didn't sound quite right as you mentioned. It sounded like the I (or Q) channel was just being duplicated in both left/right channels. It was missing the "phase shifting" effect you normally hear between the stereo channels. I'll have a look at this.

    WA2ZKD
  • jksjks
    edited November 2017
    Well, I'm getting nowhere with this. I changed the Kiwi code to send data test patterns in the audio IQ stream and they show up correctly in kiwiclient/kiwirecorder. But the audio stream, when viewed as a complex Fourier (FFT) with Baudline on a Mac, is clearly not an IQ signal. Just as it doesn't sound like an IQ signal when tuned to an AM station and played in headphones. I also checked the .wav file header and all the fields look correct (http://soundfile.sapp.org/doc/WaveFormat). And the file length is correct. So I don't get it. Will keep looking though. The spectrum in Baudline looks like the sample rate is not correct even though the correct sample rate is shown in the info panel for the file. Really strange..

  • zfyoungzfyoung
    Thank, jks. I've tried your updated script, and indeed the wave file sounds not quite right. Specifically, I found the recorded AM station signal losing its carrier peak in its spectrum plot. Wonder if the kiwi server ever respond with the correct "raw" IQ data in the first place.
  • jksjks
    edited November 2017
    Well, the identical code on the Kiwi server side is sending IQ data over a web socket to the Javascript code when you're connecting with a browser. And that seems to work. The python code opens the same web socket, sends the same commands to setup the connection (frequency, mode, etc.).

    The only difference is that the browser javascript is doing an up-sample from the 12 kHz of the IQ stream to the 44.1 kHz required by the browser audio output. And that must be okay for IQ data because Dream works when VAC'd to the browser audio output. The python code isn't doing this of course -- it just stores the IQ stream in the .wav file directly with the sample rate set to 12 kHz.

    Another difference is that the normal 4:1 compression on the Kiwi server side is turned off when IQ data is being sent (because the compression is not lossless).

    I bought some audio tools so I can look at this in more detail.

  • jksjks
    edited November 2017
    Aww, I'm such an idiot. The answer is so simple. The passband isn't wide enough! When you click the IQ button on the Kiwi it sets IQ demod mode, but also sets a default passband. But that's just how the Kiwi user interface works.

    With kiwirecorder when you change the mode with "-m iq" the low-level command it's sending on the web socket only changes the mode. It doesn't know anything about the passband. You'll get whatever the last passband was. The default for a new Kiwi connection is USB 300 - 2700 (2400 wide).

    This is of course not what you want with IQ mode. So try something like:
    python kiwirecorder.py -s fenu-radio.ddns.net -p 8073 -f 3965 -m iq -L -5000 -H 5000

    Which sets a DSB passband from -5 to +5 kHz for use with DRM. Using this I was able to feed the recorded file to Dream with no problems decoding at all.

    KA7U
  • zfyoungzfyoung
    Great! And thanks for the hard work.
  • KA7UKA7U
    I tried this and the file plays in my Dream receiver just fine. I am currently recording and when I open a web browser pointed to the above URL, it opens and works just fine, but there is no indication of the recording connection. Does this recording connection count as a client on the KiwiSDR? Or can there be any number of kiwirecorder.py connections to a given kiwiSDR?

    This is a great idea for recording at set times and days. The wav file size seems quite compact as well. This is good stuff.
    Ron - KA7U
  • jksjks
    edited November 2017
    No, it should show as a connection with a location of "Antarctica" (lol). And it does count as one of the four connections. It doesn't somehow not use the resources of a standard connection. The file is fairly compact because it's recording at the audio sample rate (12 kHz) instead of the typical internal audio system rate of 44.1 or 44 kHz.


  • KA7UKA7U
    Another interesting aspect to this kiwirecorder.py is that the Dream receiver will play the file while the file continues to be recorded. This effectively makes a near real time connection for replay while recording, or you can come into a program late and start at the beginning while the program continues to record.  

    Now for a cron job to start and stop the recorder. I think I might manage that... Is there a way to specify the recording length time on this recorder?
    Ron - KA7U

  • jksjks
    You can see all the current options by doing: python kiwirecorder.py --help
    I didn't write this code, and I don't know python, so requests for improvement should go to the original author: https://github.com/dev-zzo/kiwiclient

    KA7U
  • KA7UKA7U
    John,
    Thanks for all the work you do!  I just might write the author.

    ron@linux-4cdz:~/kiwiclient-jks-v0.1> grep -nri antarctica .
    ./kiwifax.py:372:        self.set_geo('Antarctica')
    ./kiwirecorder.py:46:        self.set_geo('Antarctica')
    ron@linux-4cdz:~/kiwiclient-jks-v0.1> 

    But in the short term, I can hack a bit, even if I don't know Python. Hi Hi
    Ron
    KA7U
  • jksjks
    edited November 2017
    So with that commented out the Kiwi status panel shows "(unknown location)". I guess he didn't want that to appear. The server is supposed to figure out the geolocation when this occurs so I should figure out why that's not happening.


    KA7U
  • jksjks
    edited November 2017
    @zfyoung,

    Your original post mentioned diversity reception, but also IQ mode. What exactly did you have in mind that was IQ-related? Were you thinking of traditional frequency diversity where kiwirecorder would open two connections on two different frequencies but merge the mono audio from each into a 2-channel file? (as L/R stereo) Or something more complicated?

    KA7U
  • jksjks
    Ron, when I was looking through the code I noticed he seemed to go to some trouble to keep the .wav header information updated as the file length grows. Probably so external readers of the file always see a consistent state as you noticed.

    KA7U
  • jksjks
    edited November 2017
    Okay, so v1.149 going out today fixes some geolocation problems. If you comment out the set_geo() calls then the ip of the connection should get geolocated. There was a bug causing it to fail most of the time and return "unknown location".

    Also, if you are on a local/private network talking to your own Kiwi then the public ip address of the Kiwi (if any) will be geolocated instead of the local address of the computer running the browser (which has no geolocation due to its private ip address).

  • KA7UKA7U
    jks,
    My error. The file does not play as it is recorded. The players load the file to the point it is recorded and then loop giving that impression, if I'm not really listening. So to listen to a complete recording, I should wait until the file is recorded and closed, then listen to it. 
    Ron - KA7U
  • jksjks
    What I would expect to happen is that when you start your file "player" (the program that reads the file) it is essentially taking a snapshot of the file at that moment. And doesn't necessarily capture what the recorder is continuing to stream into the file. It would have to be designed differently to keep up with a changing file.

    What I was concerned about is that the file be in a consistent state so at the time of the snapshot the .wav header information be updated, match the file length etc. Some of this depends on how Linux (or whatever) handles file I/O with multiple readers and one writer in addition to how the player is written. I am no expert about any of that..

  • zfyoungzfyoung
    Glad that new release is out. When I mentioned " diversity reception", I actually have frequency AND/OR spatial AND/OR polarization diversity in mind and the targeting signal is DRM digital radio signal. Since kiwiSDR can coordinated signals from different locations all across the world in real time, this platform will offer a very good test bed to all those strategies.

    Here, I have a question/request: If due to network congestion, we have lost IQ packet. Can you keep writing to the output wave file with "zero" as a placeholder, because otherwise the receiver will lose sync and try re-sync which is costly in time and performance. With multiple streams coming in-sync, I can live with some part of it "null out".

    Another suggestion is for future hardware architecture upgrade: reserve the possibility for spatial / polarization diversity for a single kiwiSDR device. ie multiple antenna at the same frequency (maybe at the sacrifice of sampling bandwidth)
  • jksjks
    edited November 2017
    Wow, that's pretty interesting. An improvement to HF DRM reliability would be most welcome from what little I know about it.

    There is a Github issue thread you should read: https://github.com/jks-prv/Beagle_SDR_GPS/issues/130
    Christoph has GPS-derived timestamps on the IQ stream mostly working. That could go a long way to solving your phase synchronization and dropout patching issues. Adding a buffer size control to kiwiclient / kiwirecorder would also be helpful as you point out. I can try working on that.

    Future hardware is a very interesting question. There are certainly 14 and 16-bit ADCs now with multiple channels on a single chip. It's just a question of cost. I've had some ideas. I think most people want me to fix the software before spending time on new hardware, lol.

  • G8JNJG8JNJ
    Hi All,

    Interesting thread.

    Does adding the time stamps to the I/Q stream now open up possibility for Time Of Arrival location of transmitter sites ?

    Regards,

    Martin - G8JNJ


  • jksjks
    Having timestamps is a requirement. But the actual solution to the HF TDOA problem seems to be very complicated from the little information I have found.

  • G8JNJG8JNJ
    Hi John,

    Agreed it is complex.

    I'll let Peter G3PLX know that this has now been implemented in case it's of interest to him.

    Regards,

    Martin - G8JNJ
  • jksjks
    You might want to hold off on that for a bit. The timestamp support is very new and I have not looked at it in detail (it was all contributed work).

    Also, I am moving QTH next week and have a lot of time off scheduled at the EOY. So if Peter gets interested and needs my help with changes etc. that's going to be a problem in the near term.

  • G8JNJG8JNJ
    Hi John,

    Ooops too late.

    Never mind, I don't think he will use it as it stands, but it's worthwhile keeping an eye on for future developments.

    Regards,

    Martin - G8JNJ
  • KA7UKA7U
    edited November 2017
    Now for a cron job to start and stop the recorder. I think I might manage that... Is there a way to specify the recording length time on this recorder?

    So how to start and stop the kiwirecorder.py using crontab (ie. cron job)
    add these commands in crontab:
    ron@linux-4cdz:~> crontab -e

    # DO NOT EDIT THIS FILE - edit the master and reinstall.

    # (/tmp/crontab.JBGmN4 installed on Thu Nov 23 10:37:32 2017)
    # (Cronie version 4.2)
    # .---------------- minute (0 - 59)
    # |  .------------- hour (0 - 23)
    # |  |  .---------- day of month (1 - 31)
    # |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
    # |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7)  OR sun,mon,tue,wed,thu,fri,sat
    # |  |  |  |  |
    # *  *  *  *  *  command to be executed
    # *  *  *  *  *  command --arg1 --arg2 file1 file2 2>&1
    00 12 * * *  python ~/kiwiclient-jks-v0.1/kiwirecorder.py -s southwest.ddns.net -p 8073 -f 5955.00 -m iq -L -5000 -H 5000
    30 12 * * *  for pid in $(ps -ef | awk '/python/ {print $2}'); do kill -9 $pid; done

    Ron - KA7U
Sign In or Register to comment.