SyntaxError unexpected token in JSON breaks baking

[larsyencken]

Problem

Baking occasionally fails with SyntaxError: Unexpected token < in JSON at position 0

Stack trace

bake grapher page [] 5/4513 0.5s 11/s 421.9s Baked chart accessibility-of-statins-in-the-public-health-sector
--
  | buildLocalBake.js [baseUrl] [dir]
  |  
  | Bake the site to a local folder
  |  
  | Positionals:
  | baseUrl  Base URL of the site     [string] [default: "http://localhost:3000/"]
  | dir      Directory to save the baked site      [string] [default: "localBake"]
  |  
  | Options:
  | --version  Show version number                                   [boolean]
  | -h, --help     Show help                                             [boolean]
  | --steps    Steps to perform during the baking process
  | [array] [choices: "assets", "blogIndex", "embeds", "googleScholar",
  | "redirects", "rss", "wordpressPosts", "specialPages", "countries",
  | "countryProfiles", "explorers", "charts", "gdocPosts", "gdriveImages", "dods",
  | "dataInsights", "removeDeletedPosts"]
  |  
  | SyntaxError: Unexpected token < in JSON at position 0
  | at JSON.parse (<anonymous>)
  | at parseJSONFromBytes (node:internal/deps/undici/undici:6571:19)
  | at successSteps (node:internal/deps/undici/undici:6545:27)
  | at node:internal/deps/undici/undici:1211:60
  | at node:internal/process/task_queues:140:7
  | at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
  | at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
  | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
  | 🚨 Error: The command exited with status 1
  | user command error: exit status 1

https://buildkite.com/our-world-in-data/owid-deploy-code-master/builds/360#018dc107-523c-4305-82b5-88b44046e2e4

[larsyencken]

Hypotheses:

• Http 500 from DigitalOcean spaces
  • since...
    • It seems to happen when baking charts
    • Chart baking does tons of JSON requests for data and metadata
    • The error is transient and disappears on retry, just like these 500s
  • but...
    • The code paths fetchS3MetadataByPath and fetchS3DataValuesByPath both seem to check for 200 responses and catch exceptions, logging them more gracefully
• Invalid JSON in the DB
  • but...
    • MySQL is strict and does not allow invalid JSON to be saved
    • The error does not occur on retry

[Marigold]

Related discussion on Slack and this PR that might shed some light on the error.

[larsyencken]

Tried a local bake, I got a HTTP 500 from DigitalOcean spaces with a nice stack trace.

[Marigold]

After getting rid of workerpool in baking, the stack trace is clearly visible, showing 500 errors from CF workers. The bug is pretty dumb - retryPromise doesn't wrap response.ok check, so if there's a 500 (that doesn't result in error in fetch), it won't retry anything. The fix is here.