Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The inconsistent cluster network card name causes the ingress-nginx pod communication to fail. #4497

Closed
wsxedcer opened this issue Aug 28, 2019 · 3 comments
Closed

The inconsistent cluster network card name causes the ingress-nginx pod communication to fail. #4497

wsxedcer opened this issue Aug 28, 2019 · 3 comments

Comments

@wsxedcer
Copy link

The test found that the inconsistency of the cluster network card name will cause the ingress-nginx pod communication to fail.

IP                Roles       Network device name     kubernetes.io/hostname
192.168.11.180    master        em1                        192-168-11-180
192.168.11.181    master        em1                        192-168-11-181    
192.168.11.182    master        em1                        192-168-11-182
192.168.11.184    edge          em1                        192-168-11-184
192.168.11.185    worker        eth0                       192-168-11-185
192.168.11.186    edge          eth0                       192-168-11-186
192.168.11.187    worker        em1                        192-168-11-187

deploy calico

curl https://docs.projectcalico.org/v3.8/manifests/calico-typha.yaml -O
vim calico-typha.yaml
  name: CALICO_IPV4POOL_IPIP
  value: "Off" 
 
  name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"

deploy ingress-nginx

wget -c https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

vim mandatory.yaml
apiVersion: apps/v1
# kind: Deployment
kind: DaemonSet         # change
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
#  replicas: 1          # change
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
# ------------ Add the following content ------------------
      hostNetwork: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "kubernetes.io/hostname"
                operator: In
                values:
                - ''192-168-11-184"
                - ''192-168-11-185"
                - ''192-168-11-186"
                - ''192-168-11-187"

[root@192-168-11-180 ~]# kubectl apply -f mandatory.yaml

[root@192-168-11-180 ~]# kubectl get pods -n ingress-nginx  -o wide                           
NAME                             READY   STATUS             RESTARTS   AGE     IP               NODE             NOMINATED NODE   READINESS GATES
nginx-ingress-controller-9rdwk   0/1     CrashLoopBackOff   5          5m50s   192.168.11.186   192-168-11-186   <none>           <none>
nginx-ingress-controller-lrnpt   1/1     Running            0          5m50s   192.168.11.187   192-168-11-187   <none>           <none>
nginx-ingress-controller-v5xl9   0/1     CrashLoopBackOff   5          5m50s   192.168.11.185   192-168-11-185   <none>           <none>
nginx-ingress-controller-z27fp   1/1     Running            0          5m50s   192.168.11.184   192-168-11-184   <none>           <none>

[root@192-168-11-180 ~]# kubectl get daemonsets -n ingress-nginx 
NAME                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
nginx-ingress-controller   4         4         2       4            2           <none>          3m53s

describe 192.168.11.186 ，This is the problematic pod

[root@192-168-11-180 ~]# kubectl describe -n ingress-nginx pod nginx-ingress-controller-9rdwk 
Name:               nginx-ingress-controller-9rdwk
Namespace:          ingress-nginx
Priority:           0
PriorityClassName:  <none>
Node:               192-168-11-186/192.168.11.186
Start Time:         Wed, 28 Aug 2019 10:20:12 +0800
Labels:             app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    controller-revision-hash=6b54bc68fd
                    pod-template-generation=1
Annotations:        prometheus.io/port: 10254
                    prometheus.io/scrape: true
Status:             Running
IP:                 192.168.11.186
Controlled By:      DaemonSet/nginx-ingress-controller
Containers:
  nginx-ingress-controller:
    Container ID:  docker://bf083abe05f621a994073c1844c7c6b35b62c85da2cf7e68bb90d8e9e6fc3b0a
    Image:         registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0
    Image ID:      docker-pullable://registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller@sha256:9ea00cd445203f925d9dabfde2aa06b67dc1edc1099c0262723f3384d4245e54
    Ports:         80/TCP, 443/TCP
    Host Ports:    80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --publish-service=$(POD_NAMESPACE)/ingress-nginx
      --annotations-prefix=nginx.ingress.kubernetes.io
    State:          Running
      Started:      Wed, 28 Aug 2019 10:26:31 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 28 Aug 2019 10:24:17 +0800
      Finished:     Wed, 28 Aug 2019 10:25:06 +0800
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:10254/healthz delay=10s timeout=10s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-9rdwk (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-45hq6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nginx-ingress-serviceaccount-token-45hq6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-serviceaccount-token-45hq6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                    From                     Message
  ----     ------     ----                   ----                     -------
  Normal   Scheduled  6m21s                  default-scheduler        Successfully assigned ingress-nginx/nginx-ingress-controller-9rdwk to 192-168-11-186
  Normal   Started    5m36s (x2 over 6m20s)  kubelet, 192-168-11-186  Started container nginx-ingress-controller
  Warning  Unhealthy  4m57s (x6 over 6m7s)   kubelet, 192-168-11-186  Liveness probe failed: HTTP probe failed with statuscode: 500
  Normal   Killing    4m57s (x2 over 5m47s)  kubelet, 192-168-11-186  Container nginx-ingress-controller failed liveness probe, will be restarted
  Warning  Unhealthy  4m53s (x9 over 6m13s)  kubelet, 192-168-11-186  Readiness probe failed: HTTP probe failed with statuscode: 500
  Normal   Pulled     4m47s (x3 over 6m20s)  kubelet, 192-168-11-186  Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0" already present on machine
  Normal   Created    4m46s (x3 over 6m20s)  kubelet, 192-168-11-186  Created container nginx-ingress-controller
  Warning  BackOff    73s (x3 over 87s)      kubelet, 192-168-11-186  Back-off restarting failed container

describe 192.168.11.185， This is the problematic pod

[root@192-168-11-180 ~]# kubectl describe -n ingress-nginx pod nginx-ingress-controller-v5xl9 
Name:               nginx-ingress-controller-v5xl9
Namespace:          ingress-nginx
Priority:           0
PriorityClassName:  <none>
Node:               192-168-11-185/192.168.11.185
Start Time:         Wed, 28 Aug 2019 10:20:12 +0800
Labels:             app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    controller-revision-hash=6b54bc68fd
                    pod-template-generation=1
Annotations:        prometheus.io/port: 10254
                    prometheus.io/scrape: true
Status:             Running
IP:                 192.168.11.185
Controlled By:      DaemonSet/nginx-ingress-controller
Containers:
  nginx-ingress-controller:
    Container ID:  docker://6b8547290e3a7fcfde57887d47e3702502d717f37beb8ea9584980fe3e66b242
    Image:         registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0
    Image ID:      docker-pullable://registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller@sha256:9ea00cd445203f925d9dabfde2aa06b67dc1edc1099c0262723f3384d4245e54
    Ports:         80/TCP, 443/TCP
    Host Ports:    80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --publish-service=$(POD_NAMESPACE)/ingress-nginx
      --annotations-prefix=nginx.ingress.kubernetes.io
    State:          Running
      Started:      Wed, 28 Aug 2019 10:27:05 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 28 Aug 2019 10:24:43 +0800
      Finished:     Wed, 28 Aug 2019 10:25:32 +0800
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:10254/healthz delay=10s timeout=10s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-v5xl9 (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-45hq6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  nginx-ingress-serviceaccount-token-45hq6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-serviceaccount-token-45hq6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                     From                     Message
  ----     ------     ----                    ----                     -------
  Normal   Scheduled  7m24s                   default-scheduler        Successfully assigned ingress-nginx/nginx-ingress-controller-v5xl9 to 192-168-11-185
  Normal   Pulling    7m23s                   kubelet, 192-168-11-185  Pulling image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0"
  Normal   Pulled     6m54s                   kubelet, 192-168-11-185  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0"
  Normal   Created    6m13s (x2 over 6m54s)   kubelet, 192-168-11-185  Created container nginx-ingress-controller
  Normal   Started    6m13s (x2 over 6m54s)   kubelet, 192-168-11-185  Started container nginx-ingress-controller
  Warning  Unhealthy  5m34s (x6 over 6m44s)   kubelet, 192-168-11-185  Liveness probe failed: HTTP probe failed with statuscode: 500
  Normal   Killing    5m34s (x2 over 6m24s)   kubelet, 192-168-11-185  Container nginx-ingress-controller failed liveness probe, will be restarted
  Normal   Pulled     5m23s (x2 over 6m13s)   kubelet, 192-168-11-185  Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0" already present on machine
  Warning  Unhealthy  2m16s (x28 over 6m46s)  kubelet, 192-168-11-185  Readiness probe failed: HTTP probe failed with statuscode: 500

describe 192.168.11.184 and 192.168.11.187 ， this is OK

[root@192-168-11-180 ~]# kubectl describe -n ingress-nginx pod nginx-ingress-controller-z27fp 
Name:               nginx-ingress-controller-z27fp
Namespace:          ingress-nginx
Priority:           0
PriorityClassName:  <none>
Node:               192-168-11-184/192.168.11.184
Start Time:         Wed, 28 Aug 2019 10:20:12 +0800
Labels:             app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    controller-revision-hash=6b54bc68fd
                    pod-template-generation=1
Annotations:        prometheus.io/port: 10254
                    prometheus.io/scrape: true
Status:             Running
IP:                 192.168.11.184
Controlled By:      DaemonSet/nginx-ingress-controller
Containers:
  nginx-ingress-controller:
    Container ID:  docker://5639aa4699b71971fdaeec643d47dc01df4ea67bb44d5f903a683fdfb9f80189
    Image:         registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0
    Image ID:      docker-pullable://quay.io/kubernetes-ingress-controller/nginx-ingress-controller@sha256:464db4880861bd9d1e74e67a4a9c975a6e74c1e9968776d8d4cc73492a56dfa5
    Ports:         80/TCP, 443/TCP
    Host Ports:    80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --publish-service=$(POD_NAMESPACE)/ingress-nginx
      --annotations-prefix=nginx.ingress.kubernetes.io
    State:          Running
      Started:      Wed, 28 Aug 2019 10:20:13 +0800
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:10254/healthz delay=10s timeout=10s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-z27fp (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-45hq6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  nginx-ingress-serviceaccount-token-45hq6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-serviceaccount-token-45hq6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Scheduled  8m27s  default-scheduler        Successfully assigned ingress-nginx/nginx-ingress-controller-z27fp to 192-168-11-184
  Normal  Pulled     8m27s  kubelet, 192-168-11-184  Container image "registry.cn-hangzhou.aliyuncs.com/google_containers/nginx-ingress-controller:0.25.0" already present on machine
  Normal  Created    8m27s  kubelet, 192-168-11-184  Created container nginx-ingress-controller
  Normal  Started    8m26s  kubelet, 192-168-11-184  Started container nginx-ingress-controller

192.168.11.185 docker log

[root@192-168-11-180 ~]# kubectl logs -n ingress-nginx nginx-ingress-controller-v5xl9 
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.25.0
  Build:      git-1387f7b7e
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

W0828 02:57:53.703254       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0828 02:57:53.709750       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0828 02:57:53.710066       6 main.go:183] Creating API client for https://10.96.0.1:443
I0828 02:57:53.770963       6 main.go:227] Running in Kubernetes cluster version v1.14 (v1.14.6) - git (clean) commit 96fac5cd13a5dc064f7d9f4f23030a6aeface6cc - platform linux/amd64
I0828 02:57:54.611012       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
E0828 02:57:54.612473       6 main.go:131] v1.14.6
I0828 02:57:54.660198       6 nginx.go:275] Starting NGINX Ingress controller
I0828 02:57:54.671210       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"5d128c48-c93a-11e9-abf0-782bcb14a072", APIVersion:"v1", ResourceVersion:"182934", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I0828 02:57:54.677502       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"5d15b756-c93a-11e9-abf0-782bcb14a072", APIVersion:"v1", ResourceVersion:"182935", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0828 02:57:54.677593       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"5d1840c1-c93a-11e9-abf0-782bcb14a072", APIVersion:"v1", ResourceVersion:"182936", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0828 02:57:55.860652       6 nginx.go:319] Starting NGINX process
I0828 02:57:55.860716       6 leaderelection.go:235] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I0828 02:57:55.861634       6 controller.go:133] Configuration changes detected, backend reload required.
I0828 02:57:55.864951       6 status.go:86] new leader elected: nginx-ingress-controller-z27fp
E0828 02:57:55.881493       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:57:55.881540       6 queue.go:130] requeuing initial-sync, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I0828 02:57:59.195100       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:57:59.215419       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:57:59.215458       6 queue.go:130] requeuing kube-system/calico-typha, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:00.970369       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:02.528412       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:02.546815       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:02.546853       6 queue.go:130] requeuing kube-system/kube-controller-manager, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I0828 02:58:05.861692       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:05.880958       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:05.880997       6 queue.go:130] requeuing kube-system/kube-dns, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I0828 02:58:09.195038       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:09.214351       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:09.214386       6 queue.go:130] requeuing kube-system/kube-scheduler, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:10.970279       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:12.528389       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:12.548226       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:12.548268       6 queue.go:130] requeuing kube-system/metrics-server, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:12.799856       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:15.861737       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:15.880952       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:15.880993       6 queue.go:130] requeuing kubernetes-dashboard/dashboard-metrics-scraper, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I0828 02:58:19.195060       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:19.213835       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:19.213873       6 queue.go:130] requeuing kubernetes-dashboard/kubernetes-dashboard, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:20.970315       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:22.528322       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:22.547082       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:22.547124       6 queue.go:130] requeuing default/kubernetes, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:22.800355       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:25.861753       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:25.882284       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:25.882322       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I0828 02:58:29.195073       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:29.213679       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:29.213706       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:30.970197       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:32.528392       6 controller.go:133] Configuration changes detected, backend reload required.
E0828 02:58:32.547766       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W0828 02:58:32.547803       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E0828 02:58:32.799813       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:32.853127       6 main.go:154] Received SIGTERM, shutting down
I0828 02:58:32.853195       6 nginx.go:402] Shutting down controller queues
I0828 02:58:32.853237       6 status.go:117] updating status of Ingress rules (remove)
E0828 02:58:32.858733       6 status.go:121] error obtaining running IPs: []
I0828 02:58:32.858763       6 nginx.go:418] Stopping NGINX process
I0828 02:58:32.864718       6 main.go:158] Error during shutdown: signal: illegal instruction (core dumped)
I0828 02:58:32.864754       6 main.go:162] Handled quit, awaiting Pod deletion
E0828 02:58:40.970374       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0828 02:58:42.864889       6 main.go:165] Exiting with 1

NGINX Ingress controller version:
nginx-ingress-controller:0.25.0

Kubernetes version : v1.14.6

Environment:

  • OS (e.g. from /etc/os-release): centos 7
  • Kernel (e.g. uname -a): 4.4.190-1.el7.elrepo.x86_64
  • Others:
[root@192-168-11-180 ~]# curl localhost:10249/proxyMode
ipvs

If the NIC name of the master server is emX, this error occurs if the ingress-nginx pod is dispatched to the node whose NIC name is ethX. This problem does not occur on the node that is assigned to the NIC name emX.
@aledbf
Copy link
Member

aledbf commented Aug 28, 2019

@wsxedcer it seems this is related to #4311
What hardware is running the k8s node?

To check if this is the error, please use an older version (0.24.1)

@wsxedcer
Copy link
Author

@aledbf Switch to 0.24.1 and the fault is solved.
This is an abnormal node CPU architecture

[root@192-168-11-185 ~]# lscpu             
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 23
Model name:            Intel(R) Xeon(R) CPU           E5405  @ 2.00GHz
Stepping:              10
CPU MHz:               1995.158
BogoMIPS:              3990.31
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              6144K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor ds_cpl vmx tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm kaiser tpr_shadow vnmi flexpriority dtherm

The lack of the sse4_2 instruction set here. The failure of the operation is the reason for the new version of nginx support architecture.

thanks.

@aledbf
Copy link
Member

aledbf commented Aug 28, 2019

Closing. Duplicated issue #4311

@aledbf aledbf closed this as completed Aug 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aledbf @wsxedcer