This document demonstrates how NGINX Ingress Controller can help scale, secure, and provide visibility into OpenShift in a multi-cluster world. The YouTube demo demonstrates OpenShift Multi-Cluster using F5 BIG-IP and Container Ingress Services (CIS) and NGINX IC. This document focuses on standalone deployment using ClusterIP. CIS is only deployed in OpenShift 4.11 cluster as shown in the diagram. However CIS can also be deployed in multi-cluster using active/standby or active/active. This will be covered in another demo
YouTube Demo
My environment has two cluster as shown in the diagram above:
- OpenShift 4.11
- OpenShift 4.13
- Similar Pods deployed in both cluster load-balanced by NGINX Ingress Controller
- Unique Pod Networks using OVNKubernetes
OpenShift-4-11 Cluster
Spec:
Cluster Network:
Cidr: 10.128.0.0/14
Host Prefix: 23
External IP:
Policy:
Network Type: OVNKubernetes
Service Network:
172.30.0.0/16
OpenShift-4-13 Cluster
Spec:
Cluster Network:
Cidr: 10.148.0.0/14
Host Prefix: 23
External IP:
Policy:
Network Type: OVNKubernetes
Service Network:
172.30.0.0/16
I would like to distribute traffic between the two cluster. While evaluating OpenShift 4.13 before upgrading/rebuilding OpenShift 4.11. Steps used to configure OpenShift multi-cluster using CIS standalone deployment.
Copy the KubeConfig from OpenShift-4-13 to OpenShift-4-11 so that CIS can monitor the services in remote clusters. CIS uses KubeClient
OpenShift-4-11 Cluster
View the copy of KubeConfig for both clusters
[root@ocp-installer auth]# ls
kubeadmin-password kubeconfig kubeconfig-openshift-4-13
[root@ocp-installer auth]#
Create secrets using the following command
oc create secret generic openshift-4-13 --from-file=kubeconfig=/openshift/ipi/auth/kubeconfig-openshift-4-13
secret/openshift-4-13 created
# oc get secret
NAME TYPE DATA AGE
openshift-4-11 Opaque 1 43h
openshift-4-13 Opaque 1 35h
[root@ocp-installer route]#
Note: Since CIS is only deployed in OpenShift-4-11, no need to share KubeConfig with OpenShift-4-13
Deploy CIS and RBAC for OpenShift 4-11 and OpenShift 4-13
OpenShift-4-11 Cluster
# oc create -f bigip-ctlr-clusterrole.yaml
# oc create -f f5-bigip-ctlr-deployment.yaml
deployment.apps/k8s-bigip-ctlr-deployment created
CIS deployment repo
OpenShift-4-13 Cluster
Create RBAC for CIS. This RBAC is created in OpenShift-4-13
# oc create -f external-cluster-rabc.yaml
RBAC repo
Deploy the NGINX Ingress Controller in OpenShift-4-11 for ClusterIP
Getting Started repo
View Service in OpenShift-4-11 for NGINX-Ingress
# oc get service -n nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-operator-controller-manager-metrics-service ClusterIP 172.30.209.54 <none> 8443/TCP 4d23h
nginxingress-sample-nginx-ingress-controller ClusterIP 172.30.69.213 <none> 80/TCP,443/TCP 4d22h
Deploy the NGINX Ingress Controller in OpenShift-4-13 for ClusterIP
Getting Started repo
View Service in OpenShift-4-13 for NGINX-Ingress
# oc get service -n nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress-operator-controller-manager-metrics-service ClusterIP 172.30.209.54 <none> 8443/TCP 4d23h
nginxingress-sample-nginx-ingress-controller ClusterIP 172.30.69.213 <none> 80/TCP,443/TCP 4d22h
Create routes in OpenShift-4-11
[root@ocp-installer openshift-4-11]# oc get route -n nginx-ingress
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
nginx-ingress-route-443 cafe.example.com ... 1 more nginxingress-sample-nginx-ingress-controller 443 passthrough/Redirect None
Cafe Routes repo
BIG-IP Pools members show the Pods from both OpenShift-4-11 and OpenShift-4-13