OpenShift 4.12 installation at dev branch with dual-stack failed

[rbo]

$ oc logs  -n openshift-ovn-kubernetes ovnkube-node-4n7q8 -c ovnkube-node
....
I0118 20:59:08.173363   19096 ovs.go:203] Exec(28): stdout: "1\n"
I0118 20:59:08.173446   19096 ovs.go:204] Exec(28): stderr: ""
I0118 20:59:08.173565   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: 127.0.0.1
I0118 20:59:08.173586   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: 169.254.169.2
I0118 20:59:08.173600   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: 10.130.0.2
I0118 20:59:08.173614   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: ::1
I0118 20:59:08.173630   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: fe80::4615:f33b:48f0:11d
I0118 20:59:08.173643   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: fd00:10:128:2::2
I0118 20:59:08.173656   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: fe80::34ee:cdff:fe68:ad1b
I0118 20:59:08.173676   19096 node_ip_handler_linux.go:260] Skipping non-useable IP address for host: fe80::ce5:43ff:fea2:cdd4
I0118 20:59:08.173688   19096 node_ip_handler_linux.go:268] Node address annotation being set to: map[192.168.61.11:{} 2a01:4f9:6a:46a5:3d::1001:{} fd69::2:{}] addrChanged: true
I0118 20:59:08.173722   19096 kube.go:97] Setting annotations map[k8s.ovn.org/host-addresses:["192.168.61.11","2a01:4f9:6a:46a5:3d::1001","fd69::2"]] on node master-1
I0118 20:59:08.177552   19096 gateway_shared_intf.go:1736] Setting OVN Masquerade route with source: 192.168.61.11
F0118 20:59:08.177634   19096 ovnkube.go:133] error waiting for node readiness: failed to set the node masque
``

[rbo]

this is a timeing issue:

$ oc get nodes/master-0 -o jsonpath="{.status.addresses}" | jq
[
  {
    "address": "192.168.61.10",
    "type": "InternalIP"
  },
  {
    "address": "master-0",
    "type": "Hostname"
  }
]

At the time when the Kubelet start the first time, the node don't have an IPv6 address.

solution: Add ipv6.may-fail=no to connection.

[rbo]

added ipv6.may-fail=no to /etc/NetworkManager/conf.d/01-ipv6.conf don't solve the problem. I have to add ipv6.may-fail=no direct to the connection.

      storage:
        files:
          - path: /etc/NetworkManager/conf.d/01-ipv6.conf
            contents:
              source: data:,%5Bconnection%5D%0Aipv6.dhcp-iaid%3Dmac%0Aipv6.dhcp-duid%3Dll%0Aipv6.may-fail%3Dno%0A%0A
            mode: 420
            user:
              name: root
            overwrite: true

Because, may-fail is not a "global default setting": https://developer-old.gnome.org/NetworkManager/stable/NetworkManager.conf.html#id-1.2.3.11

[rbo]

Related to: openshift/installer#6706

[debovema]

Is it still an issue ?
I did an installation with Dual Stack enabled :

ip_families:
  - IPv4
  - IPv6

on an IPv6 only server.
Since there is no IPv4, I disabled the DNS and /etc/hosts entries and it went fine.
Other than that it's OK and the nodes get their IPv4 in the machine network.

[rbo]

@debovema Oh good to know I have to check again. Maybe something changed meanwhile.

[knumskull]

I re-installed a fresh OCP-4.14 cluster, where before I was able to upgrade from 4.11 to 4.14.
It seems all went fine. I'm unsure if this issue still exist. How can I double-check?

$ oc get all -n openshift-ovn-kubernetes
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                                         READY   STATUS    RESTARTS     AGE
pod/ovnkube-control-plane-655db48666-25c56   2/2     Running   0            3h11m
pod/ovnkube-control-plane-655db48666-v44lm   2/2     Running   2 (3h ago)   3h11m
pod/ovnkube-control-plane-655db48666-v5sls   2/2     Running   0            3h11m
pod/ovnkube-node-2zw9b                       8/8     Running   0            3h2m
pod/ovnkube-node-cjswh                       8/8     Running   0            3h11m
pod/ovnkube-node-pncl7                       8/8     Running   0            3h2m
pod/ovnkube-node-wsnn5                       8/8     Running   0            3h10m
pod/ovnkube-node-xgjbv                       8/8     Running   0            3h11m

NAME                                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/ovn-kubernetes-control-plane   ClusterIP   None         <none>        9108/TCP            3h12m
service/ovn-kubernetes-node            ClusterIP   None         <none>        9103/TCP,9105/TCP   3h12m

NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
daemonset.apps/ovnkube-node   5         5         5       5            5           beta.kubernetes.io/os=linux   3h11m

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ovnkube-control-plane   3/3     3            3           3h11m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ovnkube-control-plane-655db48666   3         3         3       3h11m