deleter: inconsistency after AGNOS11 update (NVMe related)

[AlexandreSato]

Describe the bug

I suspect that the deleter after AGNOS11 update is no more consistent. I haven't tested it yet to confirm, but I've already caught four people who had to delete realdata/* manually (because a noEnter "out of storage" event)

Provide a route where the issue occurs

dongleIDs: 0b54d0594d924cd9, 075b133b6181e058, a0e5673b89222047

075b133b6181e058/000000cc--7d1fff3f36/42

openpilot version

1376096

Additional info

Not confirmed

[adeebshihadeh]

Got any routes or dongles?

[AlexandreSato]

Got any routes or dongles?

only some dongs, added to description

[nelsonjchen]

Got this error on this route:

https://connect.comma.ai/fe18f736cb0d7813/0000040d--3a202cd3b9

It actually increased by 1% after I couldn't engage anymore either. Logs/data are uploading.

I pretty much only run master-ci so this was surprising.

[AlexandreSato]

got one segment when it's happened:
075b133b6181e058/000000cc--7d1fff3f36/42

Interestingly when I connected to the cell phone hotspot to ssh and run the deleter test script, the deleter worked the moment the connection was established without me running any command and the out of storage event cleared, when I get home I upload the route

[korhojoa]

My routes don't appear in connect so can't really help there, but got it with dongle a954cbd0682c58bd

[michaelhonan]

Yup mine's doing the same too. Hoping the new build of AGNOS in master will address...

My dongle ID is likely useless as I don't have a data connection anymore with my C3.
dc30ef2ad294a732

[adeebshihadeh]

The new AGNOS doesn't address this. Just looking into this now.

So far seems like it's all devices with NVMe drives, so likely a race condition (or failure in) mounting it. Have you guys gotten the "failed to mount" alert at all?

[michaelhonan]

Thanks for confirming!
I personally have not seen an error like that

[nelsonjchen]

I have not seen that error.

[AlexandreSato]

I also have not seen that error too.

[AlexandreSato]

grab more one: a0e5673b89222047/0000003d--2ac5e373e1/8 or in a0e5673b89222047/0000003c--0133a69a87/3 according user description

[pbassut]

Not sure if this is related but my C3X wasn't uploading anything. After deleting realdata it started uploading again.

[adeebshihadeh]

Just started looking into this:

075b133b6181e058/000000cc--7d1fff3f36/42

• eventually recovers towards the end of the drive, almost like the deleter was stuck
• no rlogs
• bootlog for this route is missing, not on the NVMe?

fe18f736cb0d7813/0000040d--3a202cd3b9

• doesn't recover like the other route did
• thanks for uploading all the logs @nelsonjchen!
• has the bootlog, and NVMe is mounted for that bootlog

[AlexandreSato]

I missed it when I did the AGNOS flash in the sound issue test

[AlexandreSato]

got some routes:

075b133b6181e058/0000003e--67e218c522/1
075b133b6181e058/0000003d--2e512616b9/3
i think 0000003f have something interesting too

075b133b6181e058/0000003a--3043e775b6/25