Nightscout BG updates fail after a period Following

[4gra]

History

I have a primary phone (iOS 15) running xdrip4io5 4.11.5 (reading Miaomiao/Libre data as part of a Loop 2.x build), reporting to a self-hosted Nightscout (15). This combination has been stable for a number of years, incrementally upgrading xdrip occasionally.

Investigation

For a period of about three days Nightscout updates took place from a secondary phone (iOS16, Loop build WITHOUT xdrip) for development reasons while the primary awaited rebuilds. I switched the primary phone's xdrip4io5 to Follower mode, and it remained in this mode for about two days.

On returning the primary phone (xdrip4io5 4.12) to Master mode I found that everything worked perfectly as per each prior except uploading of BG values. Treatments were being uploaded fine, and Loop was uploading pump data fine, so it wasn't an authentication or generalised Nightscout problem. The phone logs show lots of this sort of large upload being attempted repeatedly:

2024-07-22 15:39:33.7620 4.11.3 4187 NightScoutUploadManager        in uploadBgReadingsToNightScout
2024-07-22 15:39:33.8440 4.11.3 4187 NightScoutUploadManager            number of readings to upload : 1257
2024-07-22 15:39:33.8450 4.11.3 4187 NightScoutUploadManager            restricting readings to upload to 300
2024-07-22 15:39:34.3270 4.11.3 4187 NightScoutUploadManager            size of data to upload : 72843
2024-07-22 15:39:34.3300 4.11.3 4187 NightScoutUploadManager            calling urlSessionUploadTask.resume

and occasionally this sort of error:

2024-07-22 15:39:35.7960 4.11.3 4187 NightScoutUploadManager            finished upload
2024-07-22 15:39:35.7980 4.11.3 4187 NightScoutUploadManager            failed to upload, statuscode = 500
2024-07-22 15:39:35.8000 4.11.3 4187 NightScoutUploadManager            data received = {"status":500,"message":"Mongo Error","description":{"driver":true,"name":"MongoError","index":0,"code":16837}}

other uploads seemed to be succeeding but over the course of a day the BG uploads didn't return.

Nightscout logs weren't very useful but confirmed that the large uploads were being attempted. I didn't manage to alter logging either on NIghtscout or mongod enough to actually see the POSTed values or the full error. But all my manual (curl) test SGV uploads worked fine, as did (Loop / xdrip4io5) uploads from other sources.

Diagnosis

It became apparent that the problem was either the size of the upload, or malformed / unexpected data amongst the stored BG history on the primary phone. I have examined the recent BG data (not attached for obvious reasons) but couldn't find anything particularly unusual. So because I couldn't see the fault, I'm not sure if it was Nightscout's "fault" for not accepting/filtering this data, or xdrip4io5's fault for attempting to send malformed data.

Workaround, and Fix

The solution was presumably to delete the faulty/excessive BG history, but xdrip4io5 doesn't provide a way to do this automatically, and deleting the app is undesirable in this case (I believe would need to delete the entire app group including Loop?). My workaround was to rebuild xrip4io5 with a minimum retention value of 1 day instead of 90 so as to clear the last week's data.

Setting the value to 1 on the live phone started a cleanup process which took about five minutes and shredded enough data that, eventually, uploads started working again and BG values (and a day's history) appeared in Nightscout. Afterward I restored the retention to its previous value of 365 and the problem was solved.

Bug?

There was clearly some undesirable behaviour. xdrip4io5 should be able to seamlessly return to being a nightscout Master having been a Follower for a while. I'm not sure what the source was, and don't have enough equipment to conduct further tests. I'm not sure what to suggest at this point as I don't know if the data was faulty, or just not accepted by Nightscout. So there's potentially a bug writing or not clearing garbage data when switching mode, and I appreciate just clearing the queue might not be a fix given that users might temporarily switch modes and be surprised when their old data doesn't get uploaded.

Feature requests

I can't see a good reason why the minimum couldn't be set to zero or near-zero. On the contrary I can see good use cases where users might want control over their BG data so that their BG history is not stored on a device for a long period. I don't understand why such a high value needs to be hard-coded, and I don't see any technical issue with keeping the value low even if it's not a good way to collect stats.

So:

1. could this minimum value be reduced - perhaps only requiring a certain number if statistics are enabled?
2. perhaps it would be prudent to add a "Clear History" Button to clear ALL stored data in case of trouble or even just user preference?