Error creating task api socket: bind: invalid argument

[Himura2la]

Versions

• Nomad: v1.8.2
• CNI Plugins: v1.1.1

Operating system and Environment details

Debian 12, amd64

Issue

Clients logs are flooded with messages like that:

[WARN] client.alloc_runner.task_runner.task_hook.api: error creating task api socket: alloc_id=d2c89842-2332-9471-f996-a041e5e41e00 task=foo path=/opt/nomad/alloc/d2c89842-2332-9471-f996-a041e5e41e00/foo/secrets/api.sock error="listen unix /opt/nomad/alloc/d2c89842-2332-9471-f996-a041e5e41e00/foo/secrets/api.sock: bind: invalid argument"

[WARN] client.alloc_runner.task_runner.task_hook.api: error creating task api socket: alloc_id=e93c99d3-78c0-a3f7-5f06-03bd262e743d task=foo path=/opt/nomad/alloc/e93c99d3-78c0-a3f7-5f06-03bd262e743d/foo/secrets/api.sock error="listen unix /opt/nomad/alloc/e93c99d3-78c0-a3f7-5f06-03bd262e743d/foo/secrets/api.sock: bind: invalid argument"

Affected tasks seem random...

Reproduction steps

IDK, we did not catch the moment when these messages starts appearing. Looks like they appear only in those clusters where we have CSI volumes configured with RBD (Ceph).

Expected Result

no warnings

Actual Result

warnings

[jrasell]

Hi @Himura2la and thanks for raising this issue.

I have not been able to reproduce this using Nomad v1.8.2 and checked that the UNIX socket path length is not exceeding the maximum limit. You mention they only appear on certain clusters with certain jobs; could you therefore provide more information about this or even steps for a reproduction?

$ nomad alloc exec f9e6de8c ls -ltr /secrets/
total 4
-rw-r--r-- 1 root root 827 Nov  4 09:19 nomad_token
srw-rw-rw- 1 root root   0 Nov  4 09:19 api.sock

I'll mark this issue for further investigation and await your reply. It is worth noting, this is a warning and doesn't impact the running on the allocations on the cluster itself. The impact is that they will not have the API socket available.

[Himura2la]

I looked closer at the affected tasks, and they all are batch jobs, or connect-proxy- of batch jobs. Hope this helps.