[BUG] ovs-ovn daemonset access denied when creating pid file in /var/run/openvswitch

[EdwardCooke]

Kube-OVN Version

kube-ovn-1.13.0

Kubernetes Version

Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.3

Operation-system/Kernel Version

"Ubuntu 24.04 LTS"
5.15.167.4-microsoft-standard-WSL2

Description

When using the helm chart, the ovs-ovn pods go into a crashloopbackoff state and cannot start reporting this error:

ovsdb-server: /var/run/openvswitch/ovsdb-server.pid.tmp: create failed (Permission denied)

When I set the permissions to 777 on that directory the pods starts successfully. Which is definitely not the most ideal permission set.

Steps To Reproduce

Install kubeovn using the helm chart with these values:

networking:
  vlan:
    VLAN_INTERFACE_NAME: enp5s0f1
MASTER_NODES: 10.0.0.6,10.0.0.7,10.0.0.8

Current Behavior

ovs-ovn pods do not start due to filesystem permissions

Full logs of ovs-ovn pod failing:

edward@edslaptop:~$ kubectl logs ovs-ovn-ffcmn --all-containers

+ chown -R nobody: /var/run/ovn /var/log/ovn /etc/openvswitch /var/run/openvswitch /var/log/openvswitch
+ iptables -V
iptables v1.8.10 (nf_tables)
OVN_REMOTE_PROBE_INTERVAL is set to 10000
OVN_REMOTE_OPENFLOW_INTERVAL is set to 180
filename:       /lib/modules/6.8.0-49-generic/kernel/net/openvswitch/openvswitch.ko.zst
alias:          net-pf-16-proto-16-family-ovs_ct_limit
alias:          net-pf-16-proto-16-family-ovs_meter
alias:          net-pf-16-proto-16-family-ovs_packet
alias:          net-pf-16-proto-16-family-ovs_flow
alias:          net-pf-16-proto-16-family-ovs_vport
alias:          net-pf-16-proto-16-family-ovs_datapath
license:        GPL
description:    Open vSwitch switching datapath
srcversion:     F25440F5110EB1159F86ACA
depends:        nf_conntrack,nsh,nf_nat,nf_conncount,libcrc32c
retpoline:      Y
intree:         Y
name:           openvswitch
vermagic:       6.8.0-49-generic SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        46:BC:40:18:10:63:93:9D:F5:E0:B2:46:81:F0:4D:BE:63:1E:65:99
sig_hashalgo:   sha512
signature:      4F:4E:28:36:28:BB:E4:CE:30:2B:B7:96:EA:D4:E7:9C:8E:03:AC:D0:
                91:C5:3E:3B:77:00:25:6B:AA:3C:C7:ED:CC:5A:42:2B:90:62:B2:DF:
                9A:FB:4C:4B:CB:5B:42:69:DC:D7:ED:0B:A9:5B:7B:06:0B:90:65:EA:
                B4:DA:85:2E:25:DE:FE:2A:91:CC:99:77:ED:DF:59:88:66:73:27:CA:
                05:5C:FD:95:A3:08:D2:98:CC:9A:16:3D:17:05:24:CF:6F:32:85:D2:
                ED:B3:CD:67:04:56:C2:4D:FF:B0:7D:2D:D9:12:86:88:A9:B7:01:0D:
                26:66:0E:57:BB:E3:22:91:1C:9F:82:02:0F:0A:F6:08:51:48:BA:E9:
                9B:C7:FD:69:43:EE:C6:22:A6:1F:1D:FE:F5:76:40:78:2C:F9:2F:55:
                BE:34:CB:46:A3:57:59:DA:C8:D8:34:F2:95:B3:58:A8:43:8D:AB:18:
                8F:87:FA:AD:70:9B:EB:6A:95:9A:0D:CD:E7:3B:D0:9C:0C:10:8A:99:
                63:1F:9E:8F:F0:0A:B1:F2:9C:FF:9A:C5:A9:2E:10:9B:6D:9F:78:4D:
                1F:93:92:77:27:72:D9:47:6D:C5:CF:40:8F:ED:D3:5D:65:8D:1E:E3:
                F4:32:7F:8C:26:7D:0A:C7:D6:0D:C1:84:30:10:8D:E7:97:44:9E:5A:
                E1:CE:37:05:35:96:A7:4A:DB:B8:1C:79:9B:F7:B4:97:0F:8F:22:AF:
                B2:3F:2B:E0:DC:F9:B5:03:F9:25:5B:BA:87:C9:3D:9E:25:B4:86:3D:
                20:CD:25:E4:0D:63:72:CE:07:4A:C6:AC:86:9E:A4:51:84:CB:65:AA:
                A9:01:DD:D9:4F:B7:24:8B:87:64:0A:20:75:44:3E:EE:0F:F1:A9:F3:
                7D:B0:EE:56:C2:BA:6A:34:46:26:B9:FF:43:E4:BF:97:CD:6D:56:AF:
                A7:DC:84:E5:A4:80:DB:59:04:D1:45:9E:15:B7:3E:B8:B2:9F:1F:49:
                DF:85:5B:94:22:79:46:DB:7B:A8:86:B9:3C:1F:CC:1B:AA:30:1B:7A:
                87:56:EA:CA:A7:A6:FF:E0:65:6B:E9:18:F4:28:E9:4F:ED:B3:44:44:
                4C:3D:A2:3F:1C:80:C8:36:C6:87:06:C3:B8:32:C2:16:74:8C:47:D3:
                DB:BA:E9:23:3D:3A:47:1B:F3:A7:61:05:2F:4A:9F:A7:95:C7:58:2E:
                EF:5B:0D:A6:3E:4D:68:06:FC:CF:25:2E:EE:0E:56:C2:85:9C:69:1F:
                D5:A4:9B:2F:E5:C2:13:59:83:24:01:DD:D4:DA:FB:E8:6D:11:0A:E8:
                37:02:4F:9D:61:92:FB:8B:34:D2:5F:19
filename:       /lib/modules/6.8.0-49-generic/kernel/drivers/net/geneve.ko.zst
alias:          rtnl-link-geneve
description:    Interface driver for GENEVE encapsulated traffic
author:         John W. Linville <[email protected]>
version:        0.6
license:        GPL
srcversion:     ACA21BA7AEB00B3332BAF1D
depends:        udp_tunnel,ip6_udp_tunnel
retpoline:      Y
intree:         Y
name:           geneve
vermagic:       6.8.0-49-generic SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        46:BC:40:18:10:63:93:9D:F5:E0:B2:46:81:F0:4D:BE:63:1E:65:99
sig_hashalgo:   sha512
signature:      50:AA:13:FA:16:04:8E:47:87:66:40:8A:72:8C:05:26:93:FB:FA:94:
                EC:51:E5:89:81:1B:27:B7:BA:07:A5:BD:BF:2F:91:E0:52:85:72:6F:
                94:FB:9C:2C:93:D4:A8:B5:A1:12:89:CC:D1:54:4F:17:E8:93:07:9F:
                76:28:3C:7A:EC:92:15:4B:33:DE:6C:54:02:FE:94:C2:A7:8D:C9:05:
                FC:D3:67:BF:A3:44:86:9C:18:7D:C4:C8:B0:89:9D:3A:9E:F9:08:00:
                28:5B:32:49:B5:92:0F:89:CD:7F:B1:57:BB:93:6E:1C:6B:04:6B:94:
                FC:3C:AD:4C:C7:74:B9:2C:64:EF:F9:62:F4:23:23:EC:59:80:E3:D1:
                4E:CD:DB:CD:50:2D:A6:44:DE:7B:51:CD:7E:76:56:FF:FA:CD:0D:3D:
                5E:75:16:7E:D7:64:F5:53:C7:C2:EE:30:D3:DD:76:D6:BC:D7:D6:9D:
                94:0C:73:A4:9B:A4:AC:FF:AC:AB:71:1C:2B:E2:F4:90:FB:C5:F0:7E:
                54:CF:EF:90:0E:B2:EB:C9:8F:23:2C:C0:9A:0F:42:75:BF:1F:85:5D:
                08:2D:BF:B9:5A:DA:F2:CE:9D:30:8B:02:65:4A:52:86:B7:EA:BB:2A:
                59:3F:09:29:6D:D6:F1:78:9D:9C:9E:CC:A1:DE:19:12:E2:83:8D:F4:
                BE:27:DD:9D:20:E4:C0:14:B6:F8:0C:5E:6B:F1:57:50:05:DC:4A:57:
                6E:F5:BE:B9:BD:6D:A2:B9:0F:CE:F1:83:72:83:10:32:2C:78:7A:05:
                8E:62:50:12:57:1B:94:1A:2C:B8:BA:BF:B0:91:DE:CB:B1:FE:11:AA:
                16:FB:6B:D9:11:9C:89:CD:20:0C:C3:44:48:E5:10:48:E1:BC:25:BE:
                DD:CA:E0:E4:FE:01:E3:F6:DD:A8:0E:CD:B3:E1:59:39:30:9A:D0:43:
                A5:BB:FE:19:24:3A:79:43:B5:B8:9F:CF:45:19:5E:5F:91:91:31:76:
                B7:DC:A0:4D:44:82:FB:9B:24:5C:B1:DE:AD:88:B6:85:63:65:D3:58:
                3F:FC:A2:2C:B5:50:97:DA:DF:9B:EB:81:D2:68:E8:C4:94:68:D2:11:
                CD:A8:B2:67:89:EC:8F:84:12:DE:48:10:A7:9A:B0:74:02:AD:DD:4B:
                68:79:5A:F0:ED:09:AF:8D:42:B1:7B:6F:DC:9C:A4:50:45:0F:51:D5:
                CB:2A:2C:CD:C8:35:FD:B7:B7:16:57:BE:21:69:7F:0A:CC:2E:63:71:
                C1:20:D7:4B:E5:78:E1:98:08:FE:8E:98:CE:FA:23:67:24:5C:32:AA:
                DF:96:DC:56:89:40:31:A3:2E:0B:43:24
parm:           log_ecn_error:Log packets received with corrupted ECN (bool)
filename:       /lib/modules/6.8.0-49-generic/kernel/net/ipv4/netfilter/ip_tables.ko.zst
description:    IPv4 packet filter
author:         Netfilter Core Team <[email protected]>
license:        GPL
srcversion:     364B4DA3E6E0326A73915C4
depends:        x_tables
retpoline:      Y
intree:         Y
name:           ip_tables
vermagic:       6.8.0-49-generic SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Build time autogenerated kernel key
sig_key:        46:BC:40:18:10:63:93:9D:F5:E0:B2:46:81:F0:4D:BE:63:1E:65:99
sig_hashalgo:   sha512
signature:      67:C0:EF:2D:8B:06:FC:3D:A7:0F:8D:F8:12:D0:06:F0:E7:3D:89:AF:
                78:A8:5F:E4:21:13:AB:AC:E8:20:A8:A9:26:89:07:21:BB:7C:D2:B8:
                DB:2C:49:74:53:73:2B:26:BA:C3:E3:75:0E:7E:C9:59:D7:06:EF:83:
                08:75:34:AE:BB:FC:66:1A:AD:1F:2B:10:0D:38:C2:D1:0D:54:D8:9F:
                63:56:3C:8B:97:E4:AE:45:87:E3:66:05:FF:0F:52:2D:B2:F8:6B:7F:
                D8:0F:83:86:7F:17:E8:F2:37:FE:0A:EB:20:9F:7E:FB:EF:FE:28:51:
                69:04:82:AF:21:78:A6:4E:21:A8:78:E9:C3:0A:04:DF:C9:D5:7F:D1:
                CD:4F:03:29:8E:A6:22:63:7C:5C:FB:B6:53:38:C8:25:5B:5B:B4:33:
                39:98:E9:49:17:04:C6:9E:74:5C:6C:65:AE:3C:1F:9E:CE:4F:A6:5D:
                CA:33:31:D7:64:F9:A7:AF:7E:7B:A3:8E:A3:63:68:A5:0B:9D:E7:1A:
                AE:07:7B:99:82:64:59:6E:7E:AF:85:03:1B:73:C0:5E:99:80:7E:C6:
                FA:C2:C8:F4:92:E7:11:F8:60:72:B0:2F:F3:D5:D3:7A:15:F6:CE:87:
                DF:84:71:35:C8:4D:97:B7:DE:EB:9B:CA:73:E1:0C:86:17:08:14:72:
                3A:41:9D:B1:BE:B0:A2:29:96:63:45:BE:4E:14:08:F5:55:A0:75:B8:
                54:EE:33:07:28:3A:96:96:08:F3:38:34:3B:40:8F:86:62:1A:8A:56:
                CD:DC:DC:20:9B:48:13:1A:92:B6:C2:92:C8:91:19:00:6E:18:38:B1:
                75:03:77:08:79:C9:D4:50:05:73:E4:21:66:5A:94:11:38:8E:9E:A4:
                BA:1A:3A:B8:6D:B3:D9:69:15:78:FC:70:71:C0:1D:8D:B5:2A:69:FB:
                23:E9:19:BF:0C:05:70:B3:EB:E4:3E:F5:B6:AE:DE:11:9B:33:6D:0E:
                D5:0E:78:E0:7C:16:EE:EE:9D:FE:B2:8A:1C:10:20:04:02:B1:78:5F:
                B9:5D:7C:C2:C6:AD:B1:26:EF:F8:94:FE:04:E3:30:25:CD:00:D7:D6:
                FA:DD:8E:2D:DD:39:E1:A7:7F:45:EE:B6:9E:7B:13:9C:BB:71:02:24:
                60:50:A5:E2:97:FE:6F:8E:F8:6A:10:8F:47:27:1C:42:CC:65:6D:4D:
                DE:56:DF:25:04:09:D1:D0:F4:81:A1:A2:16:D7:8C:12:91:E5:42:C3:
                49:08:8F:D5:30:86:17:1E:FB:84:81:99:68:E6:DA:C1:D8:C1:45:32:
                89:9F:16:B0:DA:1E:9D:9C:D0:D7:7C:41
iptables v1.8.10 (nf_tables)
 * ovsdb-server is not running
ovsdb-server: /var/run/openvswitch/ovsdb-server.pid.tmp: create failed (Permission denied)
 * Starting ovsdb-server
++ kubectl -n kube-system get pod ovs-ovn-ffcmn -o 'jsonpath={.metadata.generateName}'
+ gen_name=ovs-ovn-
++ kubectl -n kube-system get pod ovs-ovn-ffcmn -o 'jsonpath={.metadata.labels.controller-revision-hash}'
+ revision_hash=65c8c8f8c5
++ kubectl -n kube-system get controllerrevision ovs-ovn-65c8c8f8c5 --ignore-not-found -o 'jsonpath={.revision}'
+ revision=1
+ ds_name=ovs-ovn
++ kubectl -n kube-system get controllerrevision --no-headers
++ awk '$2 == "daemonset.apps/ovs-ovn" {print $3}'
++ sort -nr
++ head -n1
+ latest_revision=1
+ '[' x1 = x -o x1 = x1 ']'
++ /usr/share/ovn/scripts/ovn-ctl status_controller
++ tail -n1
++ awk '{print $NF}'
++ printf '\n0'
+ pid=0
+ cgroup_match 0 self
+ '[' 0 -le 0 ']'
+ return 1
++ /usr/share/openvswitch/scripts/ovs-ctl status
++ grep ovsdb-server
++ tail -n1
++ awk '{print $NF}'
++ printf '\novsdb-server 0'
+ pid=0
+ cgroup_match 0 self
+ '[' 0 -le 0 ']'
+ return 1
++ /usr/share/openvswitch/scripts/ovs-ctl status
++ grep ovs-vswitchd
++ tail -n1
++ awk '{print $NF}'
++ printf '\novs-vswitchd 0'
+ pid=0
+ cgroup_match 0 self
+ '[' 0 -le 0 ']'
+ return 1
+ exit 0

Expected Behavior

ovs-ovn pods would start successfully.

[dosubot]

Hey there, @EdwardCooke! I'm here to help you with this issue. Let's work together to solve this! 🤖

To resolve the "Permission denied" error when the ovs-ovn daemonset tries to create a PID file in /var/run/openvswitch, ensure that the directory permissions are correctly set. The initContainers section of the DaemonSet includes a command to change ownership of the directory:

initContainers:
  - name: hostpath-init
    command:
      - sh
      - -xec
      - |
        chown -R nobody: /var/run/ovn /var/log/ovn /etc/openvswitch /var/run/openvswitch /var/log/openvswitch

This command changes the ownership of /var/run/openvswitch to nobody. Ensure that the user running the openvswitch container has the necessary permissions to write to this directory. You might need to adjust the runAsUser in the securityContext of the openvswitch container to match the ownership set by the initContainer ^[1].

If the issue persists, verify that the hostPath volume for /var/run/openvswitch is correctly mounted and accessible:

volumes:
  - name: host-run-ovs
    hostPath:
      path: /run/openvswitch

Ensure that the path /run/openvswitch on the host has the correct permissions and is not restricted by AppArmor or SELinux policies. Adjust the permissions or policies as necessary to allow the ovs-ovn daemonset to create files in this directory ^[1].

To continue talking to Dosu, mention @dosu.

[EdwardCooke]

@dosu the runasuser for the ovs-ovn daemonset is hardcoded to be nobody by default, it would only appropriately be set to root if I enable ipsec. I am unable to change it. The init pod doesn't seem to take affect in the filesystem itself since I checked the permissions on the directory on the host and it was still owned by root with 755 permissions and not by owned by nobody:nogroup.

[dosubot]

To change the runAsUser for the ovs-ovn daemonset from nobody to root, you can modify the securityContext of the openvswitch container in the DaemonSet specification. Set runAsUser to 0 (root). Here's how you can update the YAML configuration:

containers:
  - name: openvswitch
    securityContext:
      runAsUser: 0  # Set to 0 to run as root
      privileged: false
      capabilities:
        add:
          - NET_ADMIN
          - NET_BIND_SERVICE
          - NET_RAW
          - SYS_NICE
          - SYS_ADMIN

Ensure this change is applied to the containers section of the DaemonSet template. This will allow the openvswitch container to run as the root user, which should resolve the permission issues you're encountering ^[1].

To continue talking to Dosu, mention @dosu.

[EdwardCooke]

I'm trying to use the official helm chart and avoid managing resources manually as that become a big headache during upgrade/updates.