Scraper exits with 137 exit code

[benoit74]

See e.g. https://farm.openzim.org/pipeline/1b6d8178-864c-4736-9b64-58d7df44e295, https://farm.openzim.org/pipeline/b6bdec4d-b64e-4cd0-ac6f-790e18f96474, https://farm.openzim.org/pipeline/8a3b159d-17e8-42f0-9154-a7bb8d6d53dc or https://farm.openzim.org/pipeline/fe7ac08c-43e0-497c-bb05-0bf53df2514f

[2024-08-05 00:34:14,552: INFO] starting zimfarm task-worker for 8a3b159d-17e8-42f0-9154-a7bb8d6d53dc.
[2024-08-05 00:34:14,552: INFO] configuration:
	username=michaelblob
	webapi_uris=['https://api.farm.openzim.org/v1']
	workdir=/data
	task_id=8a3b159d-17e8-42f0-9154-a7bb8d6d53dc
[2024-08-05 00:34:14,552: INFO] testing workdir at /data…
[2024-08-05 00:34:14,553: INFO] 	workdir is available and writable
[2024-08-05 00:34:14,553: INFO] testing private key at /etc/ssh/keys/zimfarm…
[2024-08-05 00:34:14,626: INFO] 	private key is available and readable (db0be1c33347c1b3b37a9565e619db62)
[2024-08-05 00:34:14,626: INFO] testing authentication with https://api.farm.openzim.org/v1…
[2024-08-05 00:34:15,265: INFO] 	authentication successful
[2024-08-05 00:34:15,266: INFO] testing docker API on /var/run/docker.sock…
[2024-08-05 00:34:15,303: INFO] 	docker API access successful
[2024-08-05 00:34:15,325: INFO] Host hardware resources:
	CPU : 24 (total) ;  18 (avail)
	RAM : 160 GiB (total) ;  144 GiB (avail)
	Disk: 300 GiB (configured) ; 574.04 GiB (avail) ; 68.29 GiB (reserved) ; 231.71 GiB (remain)
[2024-08-05 00:34:15,325: INFO] registering exit signals
[2024-08-05 00:34:15,325: INFO] Fetching task details for 8a3b159d-17e8-42f0-9154-a7bb8d6d53dc
[2024-08-05 00:34:15,631: INFO] Updating task-status=started
[2024-08-05 00:34:15,942: INFO] Setting-up workdir
[2024-08-05 00:34:15,949: INFO] Starting DNS cache
[2024-08-05 00:34:16,251: DEBUG] DNS Cache started using IPs: ['172.17.0.4']
[2024-08-05 00:34:16,251: INFO] Starting scraper. Expects files at: /nfs/zimfarm/data/8a3b159d-17e8-42f0-9154-a7bb8d6d53dc 
[2024-08-05 00:34:16,253: DEBUG] Pulling image ghcr.io/openzim/ted:3.0.3
[2024-08-05 00:34:16,691: INFO] Updating task-status=scraper_started
[2024-08-05 06:21:25,448: INFO] Updating task-status=scraper_completed. Exit code: 137
[2024-08-05 06:21:25,957: DEBUG] Dumping docker logs to file…
[2024-08-05 06:21:26,187: DEBUG] Starting log uploader container…
[2024-08-05 06:21:26,480: DEBUG] No artifacts configured for upload
[2024-08-05 06:21:35,605: INFO] Scraper log upload complete: 0
[2024-08-05 06:21:35,605: INFO] Stopping and removing log_uploader
[2024-08-05 06:21:35,620: INFO] Sending scraper log filename: 8a3b159d-17e8-42f0-9154-a7bb8d6d53dc_ted.log
[2024-08-05 06:21:35,930: INFO] Updating task-status=failed

I'm not sure what we could do in such situations, exit code 137 usually means RAM depletion, so it looks like under some circumstances make_zim_file function is consuming too much RAM.