[Bug]: fs.inotify.max_user_watches is too low

[pierreozoux]

What happened

In some containers, we see logs like:

too many open files

to fix it, we run on the nodes:

sudo sysctl fs.inotify.max_user_instances=1280
sudo sysctl fs.inotify.max_user_watches=655360

Step to reproduce

Install loki promtail,and several pods, and should see the issue.

Expected to happen

The issue is well documented here:
https://gitlab.mim-libre.fr/rizomo/cluster-prod/-/issues/68

Add anything

My question is what do you think is the best way to fix that?

Is it at the OMI level?

Is it in the cluster manifest, I could add a line of bash that would be executed when the node joins or when the node restarts.

Or should it be some kind of daemon set?

What is you opinion on this?

Thanks for your help!

cluster-api output

.

Environment

.

[pierreozoux]

In the meantime, I found how to fix locally on my cluster:

in the kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io:

spec:
  template:
    metadata: {}
    spec:
      files:
      - content: |
          #!/bin/sh

          curl https://github.com/opencontainers/runc/releases/download/v1.1.1/runc.amd64 -Lo /tmp/runc.amd64
          chmod +x /tmp/runc.amd64
          cp -f /tmp/runc.amd64 /usr/local/sbin/runc
        owner: root:root
        path: /tmp/set_runc.sh
        permissions: "0744"
      format: cloud-config
      joinConfiguration:
        discovery: {}
        nodeRegistration:
          imagePullPolicy: IfNotPresent
          kubeletExtraArgs:
            cloud-provider: external
            node-labels: ingress=true
            provider-id: aws:///'{{ ds.meta_data.placement.availability_zone }}'/'{{
              ds.meta_data.instance_id }}'
          name: '{{ ds.meta_data.local_hostname }}'
      preKubeadmCommands:
      - sh /tmp/set_runc.sh
      - sudo sysctl fs.inotify.max_user_instances=1280
      - sudo sysctl fs.inotify.max_user_watches=655360

I could also label the node the way I needed.

I let you close if you want to keep track of this.