Galileo down [alert flag added in v1.302]

jksjks
edited July 2019 in Problems Now Fixed
Since last Thursday/Friday:
NOTICE ADVISORY TO GALILEO USERS (NAGU) 2019026
DATE GENERATED (UTC): 2019-07-13 20:15

NAGU TYPE: GENERAL
NAGU NUMBER: 2019026
NAGU SUBJECT: SERVICE OUTAGE
NAGU REFERENCED TO: 2019025
START DATE EVENT (UTC): 2019-07-12 01:50
END DATE EVENT (UTC): N/A
SATELLITE AFFECTED: ALL

EVENT DESCRIPTION: UNTIL FURTHER NOTICE, USERS EXPERIENCE A SERVICE OUTAGE. THE SIGNALS ARE NOT TO BE USED.
https://www.gpsworld.com/galileo-down-over-weekend

This may have implications for the Kiwi as the Galileo code does not currently recognize a sat being in an "alarm" state preventing its use in position/timing solutions. For example the alarm state of QZSS sats is correctly detected and an option exists on the GPS tab of the admin page to include alarmed sats or not (Q195 is in alarm as of this writing).

The Galileo code does look for alert pages in the I/NAV messages (page type = 1). But none of these are currently seen. We are looking more closely at the Galileo ICD to see what other bits might be present the code should be checking.

Below is an image where the position solutions differ substantially when Galileo is included in the pool of sats (red markers) versus not (green markers). The green markers are close to the true position.

image

Attachments:
https://forum.kiwisdr.com/uploads/Uploader/26/c9594facd659cddcda1c4b584dbabd.png
PowernumptyHB9TMC
«1

Comments

  • Okay, v1.302 adds the alert flag to Galileo sats. It's interesting to note that not all Galileo sats are currently in alert state. Even with alerted sats excluded solutions using Galileo seem to wander around much more than Navstar + QZSS only.
  • HF Reception over UK was very strange yesterday, global seismic unrest without much solar activity (energy showing as nice sine waves on long period UK seismographs) something seems afoot.
    I wonder if this intentional disruption.

    Sorry I meant to say "Nothing to see here, everything is normal, move along"
  • One of mine is in danger of being shot down, it currently thinks it is about 12,000ft over Turkey, which can't be right...



    --for info--
    I did turn on "always aquire" just before the screen shot, wanted to see if it recovers, other Kiwi just booted up to much more sensible locations
  • edited July 2019
    Not sure if this is related, but very few KiwiSDRs are available for TDoA right now.

    Mine was not appearing even though I had 29 fixes/min, although it just appeared now on the TDoA map. Still not as many receivers as usual.
  • edited July 2019
    That makes sense.

    It's probably a ransomware attack at the EU control, someone got an email with subject "Glonass wants to be your friend, click here to confirm".
    Joking apart I fully expect the BBC to hang it on the Russians tomorrow.

    Just checked, I have two GPS antennas, one on a pole at about 8ft and one close to a wall at about 6ft, the higher one is fine, the lower now thousands of miles out.
  • Something is very wrong ... Lat/Long way out.
  • Mine are in the sensible range now, not sure if enough GPS came into view, signals just got better, current software ran for long enough, for whatever reason mine seem OK at the moment.
    If you have had the admin page open for a while refresh it, check "always aquire" is enabled (unless the Beagle is heavily loaded) and check GPS connections are tight as mine was very marginal with the normal GPS antenna (not very high or in the clear).
  • Now >30 minutes after starting to use 1.302 & still no GPS fixes...

  • edited July 2019
    Refresh the admin page (F5) just to make sure nothing on the page is locked up?
    I found the map freezes on mine if I leave the admin page open too long while there are good few channels in use.
  • edited July 2019
    That map freezing occurs on my Linux Chrome browser too..... has for a long time.... not thought to be a kiwi issue
  • I think I've seen it before but figured it was a good first step.
    The RSSI figures are not great, on my kit antenna they are all above 500 and 6 of the 8 "good" ones over 1100.
  • edited July 2019
    It took over an hour before mine got the first fix... and now at 1:36 there are just over 200. Seems like far fewer fixes now for "good" birds. Every time I looked, at most two channels tied up by those in "alarm".
  • jksjks
    edited July 2019
    Yeah, something is not right. Both Kiwis I test GPS code on were in a strange state this morning. They were getting subframes (colored blocks appearing on GPS page). But the "S" column (solution) showed red instead of green on each computation which means an out-of-bounds position (e.g. ridiculous altitude etc.) There were never any green solutions.

    Please be sure you have "Include alerted?" set to off and try restarting. When non-alerted Galileo sats are present I'm now seeing the red and green position markers on the map back in close agreement as before. This in NZ and the USA. I'd be interested in other reports.
  • UK (southwest)
  • The TDoA-eligible Kiwi count is slowly coming back up. It was 70 an hour ago and is now 113. Since the default value for the "include alerted" switch is false I wonder if bad data from some non-alert flagged Galileo sats was causing many Kiwis to stop delivering good solutions for a period of time. This would cause them to be removed from the TDoA list. The solutions-per-minute (and hour), which TDoA looks at, is not displayed on the GPS page. Only the total solution count.
  • I noticed the solution per minute were not on the GPS page so earlier I got the average per minute for the run time.
    worked out to roughly 21, 24, 13, 20.
    Did look like bad data was getting through, quite possible as there is little fault information from the Galileo squad (other than "don't use").
  • v1.303 is out with a new switch "Include Galileo?". Also fixed a bug where "Include alerted?" didn't have immediate effect each time you changed it. Also fixes-per-minute is shown as "f/min" at the bottom of the page.

    Now I've seen cases where including non-alerting Galileo sats will change solutions ("S") from green to red reliably. So that seems to be an indication that not all sats capable of causing problems are marked alerted. But I've also seen plenty of cases where Galileo sats are providing perfectly fine solutions too.
    HB9TMC
  • Thanks for v1.303 but unfortunately with "Include Galileo" disabled it's still filling many of the 12 slots with Galileo satellites, meaning that I'm only getting 3-5 GPS sats with a good antenna outdoors in the clear. It would help if the switch could stop the Rx acquiring Galileo sats.
  • That's a separate problem requiring a separate solution. If I add that to the "include" switch it becomes useless for debugging. Already I believe it may have helped uncover a long-standing bug in the GPS solution code (working on it now).

    We probably need individual selection of which sat systems (Navstar, QZSS, Galileo) to include/ignore during sat acquisition.
  • All right, v1.304 has individual enabling of Navstar/QZSS/Galileo acquisition.
    PowernumptyHB9TMC
  • edited July 2019
    Things were looking good this afternoon with Galileo, when suddenly the kiwi thought I was being in central America although the Galileo sats were not being flagged as alerted. After disabling Galileo aquisition, it immediately jumped to the correct location.
    The problem is, due to mountains I have limited coverage of the sky. Sometimes I have only two Navstar SVs visible.
    Didn't the kiwi had GLONASS support once?
  • Unfortunately GLONASS, BeiDou, NAVIC et al. don't transmit signals on the L1 frequency (1575.42 MHz, 2 MHz b/w) used by the GPS front-end chip of the Kiwi.
    HB9TMC
  • I am not sure if this is coincidental or related, but ever since this started, my KiwiSDR has been doing very poorly in the GPS department. Today for example I don't think it has produced a single fix since it rebooted overnight with the latest update.

    Anyone else seeing this?
  • jksjks
    edited July 2019
    The Galileo inaccuracies may be triggering a bug in the Kiwi code (uncertain at this point). If you are updated to v1.304 try a few different settings. Always have the Kalman filter on. Try with Galileo removed from the "acquire" list on the left and see if just Navstar/QZSS performs reasonably. When you remove a sat system from the list you currently have to restart to free up slots taken by the system you are removing (if necessary).

    If this works okay then add Galileo back and try with and without the "Include Galileo?" switch set.

    The bug is that sometimes when Galileo is enabled and not giving solutions the code appears to get "stuck" and you never get any good solutions again without a restart. Even after setting "Include Galileo?" to off which normally works as expected.
  • If the code appears to be "stuck", disabling and then re-enabling the Kalman filter should fix it.
  • In v1.304 changing the Kalman setting requests a restart. But this is unnecessary and will be fixed in today's release. The Kalman option can be changed on-the-fly like all the other settings.
  • jksjks
    edited July 2019
    So other sources are reporting that most of the Galileo ephemerides seem to have returned to normal. I have noticed this in my testing and the "TDoA capable" count on kiwisdr.com/public has been > 250 all day.

    These same sources also said there was little correlation between sats with the alert flag set and sats with ephemerides "issue of data" (IODE) values indicating invalid navigation data. When I added code to track the age of updates to the IODE I noticed the same thing. So this criteria needs to be added in addition to just looking at the alert flag. This is apparently how professional receivers work.
    Powernumpty
  • edited July 2019
    It's very strange, I am still getting just 1 or 2 satellites at a time. It started around the same time as the outage and recent updates, but apparently that was coincidental since other KiwiSDRs seem to have returned to normal GPS operation. I am running the latest 1.305. Over the entire 6 hours since the overnight restart after updating the KiwiSDR software it had less than 500 total fixes. The Az/El map was completely blank, I don't think it was ever able to locate the position of any of the sats?

    I have all the checkboxes on the GPS settings tab checked.

    EDIT: OK... now all of the sudden it is working (better anyway). 6 sats, 4 good and I am finally getting frequency corrections. This is very strange.

    One question - I seem to recall GPS operation is somewhat related to the channel load on the KiwiSDR? Mine is fairly heavily used, although I am not sure how much use there is in the wee hours. It had 3 or 4 users when I first checked earlier this morning it had few sats. Now it had just one user. Oh, and now 8 sats and 5 good. And sats are appearing on the Az/El map. Weird.
  • Adding a plausibility check might be also a good idea. Moving 1000s of km in a second is an unlikely thing to happen.

    I have device with a commercial GPS timing receiver. After powering it up it runs a discovery mode where it is determining the location. Once this is done, it is assumed that the position is stationary. Apparently this helps to get better timing precision.
    Powernumpty
Sign In or Register to comment.