[BUG] Providernetwork Crashs the kube-ovn-cni container

[supu2]

Kube-OVN Version

v1.13.0

Kubernetes Version

v1.30.3

Operation-system/Kernel Version

6.6.52-talos

Description

Whenever I've created the provider network kubeovn goes to restart loop with below error.

Steps To Reproduce

1. Create provider network

apiVersion: kubeovn.io/v1
kind: ProviderNetwork
metadata:
  name: external
spec:
  defaultInterface: eth1

Current Behavior

Stack Trace Error:
ovsdb-server is running with pid 4252
ovs-vswitchd is running with pid 4431
iptables v1.8.7 (nf_tables)
I1128 11:58:24.449050 155910 cniserver.go:35]

Kube-OVN:
Version: v1.13.0
Build: 2024-10-21_20:52:29
Commit: git-
Go Version: go1.22.8
Arch: amd64

I1128 11:58:24.460983 155910 config.go:170] node name not specified in command line parameters, fall back to the environment variable
I1128 11:58:24.460999 155910 config.go:350] no --kubeconfig, use in-cluster kubernetes config
I1128 11:58:24.461349 155910 k8s.go:27] succeeded to dial apiserver "10.96.0.1:443"
I1128 11:58:24.470606 155910 config.go:190] daemon config: &{ eth0 br-phy 1400 1360 false mirror0 /run/openvswitch/kube-ovn-daemon.sock /run/openvswitch/db.sock 0xc000883dc0 0xc0003a92b0 talos-rnt-6ee 10.96.0.0/16 ovn-cluster join true false false 10665 geneve /etc/cni/net.d /kube-ovn/01-kube-ovn.conflist 10-kube-ovn.conflist provider kube-system external true true /var/lib/kubelet false 8100 8101 false 100}
I1128 11:58:24.473430 155910 ovs.go:111] nic mirror0 exist, configure it
I1128 11:58:24.480088 155910 cniserver.go:56] init node gw
I1128 11:58:24.492363 155910 ovs_linux.go:591] wait ovn0 gw ready
I1128 11:58:24.494409 155910 ovs.go:50] 100.64.0.4 network ready after 1 ping, gw 100.64.0.1
I1128 11:58:24.602201 155910 cniserver.go:83] start daemon controller
I1128 11:58:24.602648 155910 server.go:33] start listen on /run/openvswitch/kube-ovn-daemon.sock
I1128 11:58:24.606211 155910 controller.go:603] Started workers
I1128 11:58:24.623786 155910 init.go:129] config provider nic eth1 on bridge br-external
E1128 11:58:24.624372 155910 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 166 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x5601522bd220, 0x560153b0f680})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0003a8860?})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x5601522bd220?, 0x560153b0f680?})
runtime/panic.go:770 +0x132
github.com/kubeovn/kube-ovn/pkg/daemon.(*networkManagerSyncer).SetManaged(0x560153b42da0?, {0xc0003a38c0?, 0xc0000d6420?}, 0x0?)
github.com/kubeovn/kube-ovn/pkg/daemon/nm_linux.go:188 +0x25
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).transferAddrsAndRoutes(0xc000326580, {0xc0003a38c0, 0x4}, {0xc000ca6601, 0xb}, 0x0)
github.com/kubeovn/kube-ovn/pkg/daemon/ovs_linux.go:1138 +0x4ff
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).configProviderNic(0xc000326580, {0xc0003a38c0, 0x4}, {0xc000ca6601, 0xb})
github.com/kubeovn/kube-ovn/pkg/daemon/ovs_linux.go:1273 +0x1e6
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).ovsInitProviderNetwork(0xc000326580, {0xc0003a38b0, 0x8}, {0xc0003a38c0, 0x4}, 0x0, 0x0)
github.com/kubeovn/kube-ovn/pkg/daemon/init.go:130 +0x605
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).initProviderNetwork(0xc000326580, 0xc0006ff700, 0xc0006ff428)
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:281 +0x265
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).handleAddOrUpdateProviderNetwork(0xc000326580, {0xc0003a38b0, 0x8})
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:266 +0x505
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).processNextAddOrUpdateProviderNetworkWorkItem.func1(0xc000326580, {0x560152134940, 0xc0003a8860})
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:203 +0x185
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).processNextAddOrUpdateProviderNetworkWorkItem(0xc000326580)
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:208 +0x46
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).runAddOrUpdateProviderNetworkWorker(...)
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:179
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00032b180, {0x5601526d08a0, 0xc00075e7b0}, 0x1, 0xc00038c000)
k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00032b180, 0x3b9aca00, 0x0, 0x1, 0xc00038c000)
k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).Run in goroutine 272
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:606 +0x4b8
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5601512aada5]

goroutine 166 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0003a8860?})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
panic({0x5601522bd220?, 0x560153b0f680?})
runtime/panic.go:770 +0x132
github.com/kubeovn/kube-ovn/pkg/daemon.(*networkManagerSyncer).SetManaged(0x560153b42da0?, {0xc0003a38c0?, 0xc0000d6420?}, 0x0?)
github.com/kubeovn/kube-ovn/pkg/daemon/nm_linux.go:188 +0x25
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).transferAddrsAndRoutes(0xc000326580, {0xc0003a38c0, 0x4}, {0xc000ca6601, 0xb}, 0x0)
github.com/kubeovn/kube-ovn/pkg/daemon/ovs_linux.go:1138 +0x4ff
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).configProviderNic(0xc000326580, {0xc0003a38c0, 0x4}, {0xc000ca6601, 0xb})
github.com/kubeovn/kube-ovn/pkg/daemon/ovs_linux.go:1273 +0x1e6
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).ovsInitProviderNetwork(0xc000326580, {0xc0003a38b0, 0x8}, {0xc0003a38c0, 0x4}, 0x0, 0x0)
github.com/kubeovn/kube-ovn/pkg/daemon/init.go:130 +0x605
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).initProviderNetwork(0xc000326580, 0xc0006ff700, 0xc0006ff428)
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:281 +0x265
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).handleAddOrUpdateProviderNetwork(0xc000326580, {0xc0003a38b0, 0x8})
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:266 +0x505
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).processNextAddOrUpdateProviderNetworkWorkItem.func1(0xc000326580, {0x560152134940, 0xc0003a8860})
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:203 +0x185
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).processNextAddOrUpdateProviderNetworkWorkItem(0xc000326580)
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:208 +0x46
github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).runAddOrUpdateProviderNetworkWorker(...)
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:179
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00032b180, {0x5601526d08a0, 0xc00075e7b0}, 0x1, 0xc00038c000)
k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00032b180, 0x3b9aca00, 0x0, 0x1, 0xc00038c000)
k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
k8s.io/[email protected]/pkg/util/wait/backoff.go:161
created by github.com/kubeovn/kube-ovn/pkg/daemon.(*Controller).Run in goroutine 272
github.com/kubeovn/kube-ovn/pkg/daemon/controller.go:606 +0x4b8
Stream closed EOF for cozy-kubeovn/kube-ovn-cni-fqvmm (cni-server)

Expected Behavior

Works without error.

[dosubot]

Hey @supu2! I'm here to help you with the issue you're experiencing. Let's work together to resolve it!

I found a similar closed issue that might be relevant to the problem you're experiencing. The issue titled "ipam invalid memory address or nil pointer dereference" was resolved by a pull request that addressed the "invalid memory address or nil pointer dereference" error ^[1].

The error you're encountering seems to be related to the networkManagerSyncer.SetManaged function, which is responsible for setting the "managed" property of a network device in NetworkManager. This function can potentially cause a nil pointer dereference if certain conditions are not met, such as when the device or its properties are not correctly initialized or retrieved ^[2].

To address this issue, you might want to check the initialization and configuration of the network devices involved, ensuring that all necessary properties are correctly set and that the devices are properly managed by NetworkManager. Additionally, reviewing the changes made in the pull request that fixed the similar issue might provide insights into resolving your current problem.

To continue talking to Dosu, mention @dosu.