Status of proxy service instability [should be good now]
Update Mar 2:
The stability problems may have been solved, at least in the short term. The proxy server has been running now for 40 hours without crashing. Let's see how it goes.
---------------------------
Original post Feb 27:
Just to keep everyone up to date:
The proxy service (frp) has been crashing & restarting every few hours, to multiple times per hour, around the time of greatest use (Europe early morning). I found and fixed one problem but there are still issues.
It seems the proxy runs into limits at around 525 active connections. I don't understand why. There are many possible solutions to consider. But many of them are hampered by the realities of the Kiwi environment.
For example old proxy clients have to be supported on the Kiwi side for people who don't stay updated to the latest Kiwi software. This precludes moving to the latest frp version in the hopes it might fix the issue. Because the latest frp server is incompatible with old frp clients. What I am doing is going through all the frp github commits looking for applicable bug fixes that could be retroactively applied to our current frp server.
The most obvious thing is to use two servers and split the proxy connections across two machines. We have two machines already: one for the proxy/website and one for the forum. The biggest issue here is that references to xxxxx.proxy.kiwisdr.com must continue to work even though a Kiwi might actually be using proxy2.kiwisdr.com (note "proxy2") on the forum machine to get its proxy service. If all of a sudden the link 21234.proxy.kiwisdr.com people entered into a browser stopped working and had to be 21234.proxy2.kiwisdr.com instead my inbox would be full and I'd have people with pitchforks at my front door.
But this is tricky. It can only be done by modifying the proxy code on the primary server to do URL redirection when applicable. And frp is code I don't know written in a language new to me ("go" aka "golang"). So it's been very slow to figure out how to do this. But it seems the best approach to take, i.e. a solution most transparent to users.
Comments
is nginx not an option, as it offers load balancing?
No. As I mentioned, there are hundreds (thousands?) of Kiwis out there with old frp clients that cannot just stop working.
The other problem is that the frps server claims the proxy port (8073), so there is no opportunity for a web server (we use lighttpd) to act as a load balancer on top of (in front of) the proxy service. Any port redirection has to come from the frp server code itself.
I have this working now. But now there have to be changes on the Kiwi-side to decide when to use proxy2.kiwisdr.com. Probably something like if the proxy name starts with a digit (the new Kiwi-2 auto proxy feature using the serial number as default proxy name). Since currently there are roughly as many of those as there are Kiwis with alphanumeric names (changed by their owner/admins).
Proxy stability update in the first post of this thread.