Proxy service down or unstable [currently okay]

jksjks
edited May 2020 in Problems and Issues
The proxy service is having serious problems. There is some evidence that it is under attack, but this is not yet confirmed.
W9SPY

Comments

  • jksjks
    edited May 2020
    I woke up this morning fully expecting to find the proxy looping again. But it was fine. It had run overnight (EU daytime) without crashing even once. EU daytime is the majority of the proxy traffic load. So this doesn't make any sense.

    A number of people have said their Kiwis are not connecting through the proxy when they should be. Please fully restart/reboot your Kiwi. Clicking the
    "(re)register" button is not sufficient it seems.
  • jksjks
    If you have a Kiwi at a remote location that requires a substantial effort to visit for a manual restart/reboot (e.g. a two hour automobile drive) then please email support@kiwisdr.com Under some circumstances we have a way to force a Kiwi restart/reboot even if the proxy connection is not working.
  • jksjks
    So there has been significant port scanning of kiwisdr.com recently. For example, over the last 24 hours 173 thousand scan requests were made by a single ip address from China. Also from a Digital Ocean server farm in the USA.

    Some of these scans occur on the proxy ports. If the URL or packet data is particularly pathological it might trigger bugs in the proxy server code (e.g. buffer overruns). This could explain the looping and crash/restart behavior we've seen in the last several days.

    Rather than try and debug the proxy code (which we didn't write), and cause significant additional proxy downtime, it was easier to strengthen our existing firewall filtering rules. Port scanning is now more reliably detected and the associated ip addresses banned.

    We will closely monitor the situation and see how effective the new rules are.
    W9SPY
  • rz3dvprz3dvp
    Maybe possible add form for using personal frp server on KiwiSDR control panel? It can be interesting for reduce proxy.kiwisdr.com server load and more shorter geo path for this proxy server (lower distance and lower latency).
  • jksjks
    Such a change is already in v1.390. Look at the bottom of the admin page network tab. Also see this commit: https://github.com/jks-prv/Beagle_SDR_GPS/commit/889224ad8f620cdbcaefc735ed5778ac95206cc3

    What I haven't yet done is the hard part: Packaging up the backend pieces of the proxy service that normally live on kiwisdr.com, i.e. the frp server and associated PHP code.

    This change was made at the request of someone who wants to run a private proxy service themselves for a bunch of Kiwis.

    Also, I have considered for a long time opening a second proxy somewhere in the EU. But there are of course issues in how to migrate specific Kiwis to that server without owner/admin intervention. I haven't worked that out yet. It was easier to just pay more money to increase the network transfer cap on our California server. We now use about 2TB/month.

    The server itself is well balanced in terms of cpu/memory/disk use between proxy/TDoA/webserver/other tasks. Unless things go wrong it is not that heavily loaded at all. It's quite remarkable actually how well it has worked.
  • rz3dvprz3dvp
    Now KiwiSDR can works with standard frp server or I need special frp version? If need special version - where I can download it?
  • jksjks
    edited May 2020
    Special version. You can't download it yet -- that's what I need to do. You also need lots of PHP to administer it.
  • W9SPYW9SPY
    Thank you for sending out this alert. For your records, I have not noticed any problems connecting to my KiwiSDR via your proxy service. (Located in Naples, Florida, United States).
  • JimoJimo
    @John, could you also consider upgrading to the latest version of frp. I see you are a few versions behind.
    Thanks
  • jksjks
    @Jimo: Not so simple. Not because of the actual code issues, but because of the validation and deployment issues (as usual). Can you imagine the disruption? Especially if something goes wrong?

    This is probably best handled when there is the eventual need to open a second proxy server (we're getting close and it will probably be in the EU). That way the migration can be done in a controlled way.
    Jimo
  • JimoJimo
    edited October 2020
    Thanks for this update.
    Looking forward to your special version as well.
Sign In or Register to comment.