Forward video has tearing artifacts

[nelsonjchen]

Describe the bug

The first drive's after a cold boot's forward video is fine. The next ONROAD's forward video is not. Might need the screen to go dark in-between drives. The wide video seems fine on all the drives. Third drive and up seemed to be fine in one of the cases.

---

this one's first boot's drive is fine: https://connect.comma.ai/fe18f736cb0d7813/00000404--83516f16c4

second boot: fe18f736cb0d7813/0000040b--78d26612d7/2

replicate-prediction-08npcdqtxdrgw0ck3zarcbkw38.mp4

replicate-prediction-e6ka7a91qsrgt0ck3zbapd3f5m.mp4

Context: Vehicle started from cold boot, had a bit of a drive, saw TPMS low tire pressure, went back to house to get tire pump, pumped all four tires, started second drive, saw errors.

Did not attempt to drive the rest of the route with openpilot engaged. It was ping-ponging like mad, which isn't unexpected for this issue.

---

first boot is fine: https://connect.comma.ai/fe18f736cb0d7813/0000040a--0dbe63a02e

second boot: fe18f736cb0d7813/00000405--2b441abe00/9

replicate-prediction-wka85genthrgt0ck3zbbarsdhr.mp4

replicate-prediction-bnq25cnhbsrgw0ck3zb984fvhc.mp4

third to fifth drive seems fine?:

• https://connect.comma.ai/fe18f736cb0d7813/00000406--f82bf78ec4

• https://connect.comma.ai/fe18f736cb0d7813/00000407--a56e0a2bec

• https://connect.comma.ai/fe18f736cb0d7813/00000408--9eb4d4c06f

Context: Just went out, got something to eat, groceries. Did use openpilot for rest of drive though. It didn't seem to be badly affected like the route above.

Provide a route where the issue occurs

fe18f736cb0d7813/0000040b--78d26612d7/2 , fe18f736cb0d7813/00000405--2b441abe00/9

openpilot version

master-ci

Additional info

Discord links:

• feedback: https://discord.com/channels/469524606043160576/1305837783490891826

• openpilot-experience: https://discord.com/channels/469524606043160576/616456819027607567/1305838954825777162

[nelsonjchen]

While not yet reported yet, a447729c1d15ff89 might have the same issue (patienc3 on Discord). They might come in with a report or data of their own.

[choongsoo]

Same issue observed in the route https://connect.comma.ai/a447729c1d15ff89/0000001b--0585570655

[adeebshihadeh]

Thanks for the great report! We made a bunch of relevant changes here recently (mostly ISP + tinygrad), but nothing that particularly points at the fcam.

• confirmed on both your drive's that it's just fcam (ecam and dcam are fine)
• friom commit 97c990d
• both are comma three with AR0231, edit: reproduced on an AR0231 device

[adeebshihadeh]

Reproduced on an AR0231 device. Repro procedure is reboot and cycle ignition a few times.

• on master, hit 4/6
• master + model daemons disabled (no tinygrad), hit 0/6
• master + ISP debayer disabled, hit 0/6
• 127c922 (right before tinygrad update), 2/7
• OX03C10 on master: 0/7

---

• only happens on the fcam
  • it's the only one still processed by the GPU
• only seems to happen on AR0231
  • has slightly larger raw MIPI frames due to the stats height
• it's every 4th frame, which points to camerad (FRAME_BUF_COUNT)

Appears to be in the raw bayer stream. Output fence not working properly? It's plausible that never quite worked correctly, but processing three cams masked over it.

Output fence wait times on a bad start:

i 3 23.64ms
i 0 0.01ms
i 1 23.61ms
i 2 0.01ms
i 3 23.61ms
i 0 0.01ms
i 1 23.54ms
i 2 0.01ms
i 3 23.63ms
i 0 0.01ms
i 1 23.62ms
i 2 0.01ms
i 3 23.60ms
i 0 0.01ms
i 1 23.62ms
i 2 0.01ms
i 3 23.59ms
i 0 0.01ms
i 1 23.61ms
i 2 0.01ms

Can't repro running camerad alone in a tight loop.

And another pattern

i 3 23.61ms
i 0 0.01ms
i 1 23.61ms
i 2 23.60ms
i 3 23.60ms
i 0 0.01ms
i 1 23.61ms
i 2 23.62ms
i 3 23.57ms
i 0 0.01ms

[MarcoTheDingo]

Had the same issue on my trip tonight. Lower third of the forward camera has this issue. Route: 5130484aa8069bad/00000007--b9a36bd4fa

[adeebshihadeh]

Reverting until this is fixed #34020.

[jakethesnake420]

I had this happen to me earlier this year. Have not noticed it since. https://connect.comma.ai/164080f7933651c4/2024-04-07--11-04-32