Wideband pulsing noise & Noise Blanker absence: I'm in trouble! [NB added in v1.152]

Markone

Hi to all,

some days ago i received my brand new KiwiSDR KIT from a UK official reseller.

After manual assembling, cable setup and software update i put it on test with a magnetic loop antenna (Wellbrook ALA 1530+), head to head with other SDR radios (Elad FDM SW2, SDRplay RS2pro, etc) .

During day time it behaved in a good way, functionally wise is a very interesting device with a lot of potential.

I like a lot the possibility to use four WSPR decoders on four different bands placed without constrictions as well as the possibility to get a prompt snapshot of the whole shortwave spectrum.

But as soon as nightfall hit and street light turned on ... everything went south :-(

My receiving site, like many others,  is subject to broadband pulsing noise coming from sodium street light during night time, that mostly affects lower shortwave bands.

Usually this is not a problem for my other SDR radios, programs like HDSDR, SDR#, FDM-SW2, fully defeat this noise cleaning the spectrum, the waterfall and the demodulated audio, with the integrated Noise Blanker functions.

KiwiSDR, right now, does not have NB or NR functions, so i got the tipical buzzing sound, that i was expecting knowing this temporay lack.

What i was not expecting is the fully scratched  waterfall screen, the FFT function seems to overreact to this condition fullfilling with several thick & bright horizontal lines, making the function near to useless.

Other SDR programs, in the same conditiom, provide a much better plot even with the noise blanker turned off.

I attached some screens with comparison between KiwiSDR and Elad FDM S2 & HDSDR, operated with same antenna, through a magnetic RF power splitter; the HDSDR "RF NB" function was turned OFF and ON to show its action.

I got same results (but not showed in images) for RSP2 and RTL-SDR with HDSDR and SDR# (with "Baseband Noise Blanker" plugin).

So ... the question is ... there is the hope that a wide band Noise Blanker function, similar to HDSDR "NB RF" or SDR# "Baseband Noise Blanker" will be implemented for KiwiSDR in the near future ?

Is it possible, in the mean time, to modify the behaviour of the waterfall screen to avoid the thick bright lines ?


Attachments:
https://forum.kiwisdr.com/uploads/Uploader/27/81da9c9aaaeb8b7db6eaf470f6c934.jpg
https://forum.kiwisdr.com/uploads/Uploader/28/1dc9f2361595029859f7cb247bd7ed.jpg
https://forum.kiwisdr.com/uploads/Uploader/1e/f3378ad60ec61dadc32599fcecd557.jpg
https://forum.kiwisdr.com/uploads/Uploader/12/957db2a850e47b81a93c7823e0abb5.jpg
https://forum.kiwisdr.com/uploads/Uploader/d0/63e15208fc22ddc0fbc853250436cd.jpg
https://forum.kiwisdr.com/uploads/Uploader/47/02f18ce6e82ee858b650ddde187a90.jpg
https://forum.kiwisdr.com/uploads/Uploader/99/564bf2a320683ab4eefa9824dd2cac.jpg
https://forum.kiwisdr.com/uploads/Uploader/7a/eb65ab127c2b45876ba5bfcdc10128.jpg

jks

Some questions:
How many simultaneous connections were there to the Kiwi when these screenshots were taken? Just one?
Does the red "OV" indicator ever appear at the top right of the S-meter? Currently, if the ADC overloads for even a single sample OV will appear.
Are you able to add some attenuation at the Kiwi input? Say 10 - 30 dB? I'm wondering if the +20 dB ADC preamp is reacting badly to these pulses.
If you add 10 dB of attenuation, but the horizontal noise lines drop more than 10 dB, this indicates a possible problem.
When you tune to a quiet frequency do the pulses on the Kiwi sound substantially worse than the other devices/programs with their NB off? On the Kiwi the audio and waterfall paths have less hardware & software in common than other SDRs and SDR software.

Also, why with HDSDR does the top bit of the waterfall have a slightly different colormap or setting? I'm just curious why it looks different to the rest.

jks

Okay, I am working on adding the simple noise blanker from CuteSDR. I am going to need root ssh access to the Beagle of a Kiwi that has this issue for development and testing. You'd just have to open port 22 on your router and point it at the Kiwi. Set a password on the root and "debian" accounts and let me know what the root password is (email support@kiwisdr.com)

Until then, there were some suggestions in this thread for using AGC settings to reduce the audio effects: http://forum.kiwisdr.com/discussion/852/noise-blanker-and-noise-reduction

Markone

Hi John, thanks for your prompt answer, here belowe my answers to your questions :

"How many simultaneous connections were there to the Kiwi when these screenshots were taken? Just one?"
- yes, just only one.

"Does the red "OV" indicator ever
appear at the top right of the S-meter?"
- OV indicator never light up among those tests, the  power splitter insertion loss seems enough for most day condition.

"I'm wondering if the +20 dB ADC preamp is reacting badly to these pulses."
- i was wondering about it too, i tried to put inline a 10 dB attenuator with no success, of course i can try greater values. 

"If you add 10 dB of attenuation, but the horizontal noise lines drop more than 10 dB, this indicates a possible problem."
- i will check this.

"When you tune to a quiet frequency
do the pulses on the Kiwi sound substantially worse than the other
devices/programs with their NB off?"
- sound wise, with NB off, Kiwi & other SDRs  are pretty similar in most conditions, the real difference is in waterfall image; of course turning on NB function gives you a perfect audio, spectrum and waterfall screen.

"Also, why with HDSDR does the top bit of the waterfall have a
slightly different colormap or setting? I'm just curious why it looks
different to the rest."
- not sure to exactly undertand this question (my bad, of course),  i will try to explain how i set waterfall in HDSDR hoping to cover your point.

Usually i select  the "Spectran" color palette (it's based on  black-blue-white color curve), then i increase both waterfall brightness & contrast settings to obtain a crisp and clear spectrum representation; palettes with more colors  are pretty but tend to hide fine gradients and to be messy in noisy condition.
For my personal taste "Spectran" is the right compromise.

One thing that makes the HDSDR waterfall screen pretty and easy to watch is its autoscale function : if the noise floor changes but the overall dynamic range is maintained, you do not see any chromatic variation, while other programs require continuous offset settings change (this is especially true for SDR#) to avoid black crushing or color top saturation. This feature, in addition to high frequency resolution and very fine/linear brightness rendering, makes it my first choice for small signal hunting. Also the fact that spectrum plot has separate settings from waterfall helps a lot.

Markone

"Okay, I am working on adding the simple noise blanker from CuteSDR. I am
going to need root ssh access to the Beagle of a Kiwi that has this
issue for development and testing. You'd just have to open port 22 on
your router and point it at the Kiwi. Set a password on the root and
"debian" accounts and let me know what the root password is (email
support@kiwisdr.com)"

Here the issue is currently present from about 17.30 to 07.30 (UTC + 1 hour time).

I will do required settings and i will provide you access information by email.

Thanks for your attention.

jks

Thanks for the answers. I'm at UTC+12, so the timing for a problem occurring at 1730-0730 UTC+1 works well.

G8JNJ

Hi John,

A couple of thoughts on this subject.

SDR's seem particularity prone to high level pulses of coherent energy spread across a wide spectrum. Even if the pulse doesn't overload the ADC, the combined power is often sufficent to overide signals that occupy a much narrower chunk of spectrum.

Conventional noise blankers tend to chop out the pulse but also remove the wanted signal too.

Is it possible to 'sense' short duration high level pulses that are well above the ambient level prior to the digital filtering, and use the time delay through the filtering to our advantage. So that pulses that are chopped out are replaced with samples that were obtained prior to the pulse. I think this would be less noticeable than simply blanking out the signal for the duration of the pulse length.

Secondly, thinking about the previous comments on diversity reception and the TOA DF work that Christoph is developing. Would it be possible to have a PC application that takes in time stamped streams from multiple KiWis and substitutes alternate streams when noise pulses occur.

Regards,

Martin - G8JNJ

jks

Okay, today's v1.152 update contains a first try at a simplistic noise blanker. Use the "NB" button above the "More" button. On the expanded control panel (via the More button) is a slider called "Noise Gate" that varies the noise gate (blanking) time between 100 - 5000 usec. Use as low a setting as possible (narrowest possible gate). The blanker is applied to both the audio and waterfall. A "Test" button injects 1PPS high-level pulses into both audio and waterfall streams.

This is just a first try and I can already see issues that need to be worked on. I haven't tried it on a lot of "real world" noise signals yet.

Markone

Hi John,

i was going to bed but i saw your forum's post update on my phone, so i promptly updated  my KiwiSDR for a fast try ... BINGO !

It's working !

Around 5Mhz, one of the the frequency band most affected by the problem, is obtaining decent results.

Right now (03.30 AM) it's not the worst noise condition of the day, the peak is around 18.00 - 24.00 (local time), so tomorrow i will do several other test at different time and frequency band.  

If you look to the attached image, the upper portion of the waterfall is "cured" with NB function turned ON, while bottom was taken with NB turned OFF.

Demodulated audio quality changes in consequence. 

As a first try, it's a very good result, THANKS a LOT.

In the mean time, i prepared my Kiwi SDR for full WEB access, email with IP addr & login data is on the way.

See you tomorrow.


Attachments:
https://forum.kiwisdr.com/uploads/Uploader/3d/e90eb4294935464b4b33e1f7ac10d8.jpg

jks

Good to hear. There are still some significant problems I'm looking at. For example when the NB is on waterfall distortion is created under some conditions. There is also the problem with "blanking to zero" causing its own noise when receiving strong signals. This is what Martin was referring to: picking better values than zero to insert into the blanking interval.

Markone

Hi John,

i just sent to you (support@kiwisdr.com) the email with my KiwiSDR web access data.

KA7U

Working with the new noise blanker. It does work well with the test. I haven't had noise on my KiwiSDR units that seems to be reduced by the noise blanker, but then maybe I don't usually have noise of that type. Maybe if I hook up a fencer...

The menu has a WF rate that pushed the WF max into the more menu. I use the WF min and WF max all the time. It is doubtful that I will use the WF rate. How about changing the behavior to have the WF min and WF max available in the small menu and let the new stuff show up with the more button.

Ron

KA7U

G8JNJ

Hi John,

It works very well.

I've tried it on some broadband noise (2-4MHz) I have from the nearby quarry during the working week, and it virtually eliminates it.

I also tried it on a couple of SDR's with some Switched mode power supply noise, and although it didn't completely remove it, it was reduced it by about 6dB, so it's definitely worth having.

However I agree with Ron the control panel is now looking a bit messy. I use the waterfall max and min controls a lot and don't think I'd use the speed control than much. So I'd also prefer the speed to be on the second page.

Some thoughts.

Could the Waterfall Max and Min sliders be half the length and placed on the same line ?

Could a waterfall auto max and min be added. Set the waterfall limits so that the top and bottom are 10dB higher and lower than the max signal and noise floor ?

Could the S-Meter be moved to be at the top of the grey control panel, so that it stays in the same place when the panel is expanded.

Could the modes be a single drop down box as previously suggested as this would give a lot more space.

Could the control panel be spread along the bottom of the waterfall so that it doesn't take up as much space across the active part of the waterfall ?

Perhaps something like this mockup.

Regards,

Martin - G8JNJ

sv1btl

At last a Noise Blanker is added! It is working good enough for common noise.

Markone

In my peculiar situation (receiving site nearby sodium street light) NB function is a must.

For what i quickly saw last night, this early implementation is surprisingly effective, beyond my expectations for a first try.

@ John : i was expecting a slide for the Threshold Level setting in addition to Noise Gate Time, are you using a fixed threshold value for the current version ?

Regarding the user web interface control panel, i agree with Martin, a large horizontal one, spread along the bottom would be the best.

Right now i'm reducing the browser font size (ie to 80%) to minimize the waterfall screen portion hidden by control menu panel.

sv1btl

Please continue to improve the NB and eventually built a Noise Reduction. I think that this is a very basic option to every receiver, which can make the difference.

Suppressing noise level will immediately improve S/N ratio, something that everyone is expecting from a good receiver.

jks

I found that on some weaker sources increasing the gate time to 1 msec (1000 usec on the slider) improved the noise reduction.

@sv1btl ;I was going to add a threshold slider but ran out of time before yesterday's update deadline.

Markone

Hi John,

if you can, give a try to my receiver (see the email that i sent to you for address), it's online with one of the worst noise condition ever seen, NB struggle to work.

jks

I'm adding a threshold control now. So I'll try it soon when I can vary both gate and threshold to see if that helps.

Markone

I opened the SSH port (22) on the router, you should be able to fully manage the receiver.  

jks

Okay. v1.153 is out with a threshold slider (get it now by rebooting). @Markone your Kiwi is already updated. Here is a screenshot with three different settings. The waterfall shows from bottom to top: no NB, NB 100 usec 50% thresh, 1000 usec 65% thresh. The last two sound the same but note how the waterfall is cleaned up a bit more.

[re-uploaded image below]

Markone

Thanks a lot John, i will put in test the new version later and then i will report my feedback.

Just out of curiosity, in your screenshot is visible a "Pref" button aside the "WF rate" slide, not present in my 1.153 version.

Is it a special developer menu  ?

jks

Yes, "user preferences" coming someday. Very complicated..

sv1btl

jks

Re-uploaded image:

Attachments:
https://forum.kiwisdr.com/uploads/Uploader/ab/66d7a4739ccb6a5e2c3dabd8f5d3d8.png

G8JNJ

Hi John,

I've been trying the noise blanker against a variety of different noise sources.

I notice that in most cases the noise gate is better at 100uS and (for me) actually worsens the performance at much higher values >3000uS.

I wondered if other users found the same ?

Would it be possible to change the minimum setting to say 10uS so that I can try the effect of much shorter durations ?

Also, would it be possible to add a 'default' button to return to standard settings, or perhaps toggle the NB button between off / normal / user ?

Regards,

Martin - G8JNJ

Markone

Usually i play more with threshold slide than gate time, anyhow there are so many variables in play that i think it's hard to establish a common approach to the problem.

In my personal case, values beyond 1000us, while cleaning better waterfall, start to be disruptive against decoded audio.

It would be interesting to test lower values than 100us, maybe this is the lower limit imposed by the implementation. 

About multiple states NB toggle function, i would instead prefer some direct selection user memories like "NB1" - "NB2" - "NBx", but i understand that adding new items to the control panel requires a change in its size, shape and structure.

I have to thank John for the prompt response to my situation, that saved me from the need to put on sale this SDR.

  

KA7U

Noise blanker works good! I have some noise tonight across most of the spectrum. tuned in and used he NB and what a difference. Nice NB.

Ron

KA7U

jks

1 / 12 kHz = 83 usec. And the NB needs a couple of gate samples to work properly. So going below 100 usec doesn't get you any improvement.

Moving the NB to a higher sampling rate in the FPGA is one option. But that will take some effort. Or increasing the audio bandwidth, which has its own problems.