Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

nginx Ingress Controller on CrashLoopBackOff Release:nginx-0.25.1-rancher1 #4751

Closed
arbermejo opened this issue Nov 13, 2019 · 1 comment
Closed

nginx Ingress Controller on CrashLoopBackOff Release:nginx-0.25.1-rancher1 #4751

arbermejo opened this issue Nov 13, 2019 · 1 comment

Comments

@arbermejo
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

NGINX Ingress controller version: nginx-0.25.1-rancher1

Kubernetes version (use kubectl version): v1.15.2

Environment:

  • Cloud provider or hardware configuration:
    Rancher Custom Deployment
  • OS (e.g. from /etc/os-release): Ubuntu 18.04
  • Kernel (e.g. uname -a): 4.15.0-66-generic
  • Install tools: Rancher
  • Others:

What happened:
nginx-ingress container is permanently on CrashLoopBackOff status after creating a new cluster

This is one of the nginx-ingress containers log

kubectl logs -n ingress-nginx nginx-ingress-controller-g8hll 
E1113 15:55:47.987155       6 main.go:165] unexpected error obtaining NGINX version: signal: illegal instruction (core dumped)
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       nginx-0.25.1-rancher1
  Build:         
  Repository:    https://github.com/rancher/ingress-nginx.git
  N/A
-------------------------------------------------------------------------------

W1113 15:55:47.987908       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W1113 15:55:48.200592       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1113 15:55:48.200880       6 main.go:183] Creating API client for https://10.43.0.1:443
I1113 15:55:48.212334       6 main.go:227] Running in Kubernetes cluster version v1.15 (v1.15.5) - git (clean) commit 20c265fef0741dd71a66480e35bd69f18351daea - platform linux/amd64
I1113 15:55:48.215949       6 main.go:91] Validated ingress-nginx/default-http-backend as the default backend.
I1113 15:55:48.894786       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
I1113 15:55:48.931734       6 nginx.go:274] Starting NGINX Ingress controller
I1113 15:55:48.960753       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"1b33c0ef-0e42-4d5b-aec2-6127d4ad72f7", APIVersion:"v1", ResourceVersion:"542", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I1113 15:55:48.964448       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"01406293-14b4-430d-b42a-7cf5b23a73b6", APIVersion:"v1", ResourceVersion:"545", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I1113 15:55:48.964508       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"3b6e5029-1f98-4dff-a499-5d4acdbbe70a", APIVersion:"v1", ResourceVersion:"546", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I1113 15:55:50.132155       6 nginx.go:318] Starting NGINX process
I1113 15:55:50.132183       6 leaderelection.go:235] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
I1113 15:55:50.132952       6 controller.go:133] Configuration changes detected, backend reload required.
I1113 15:55:50.138605       6 leaderelection.go:245] successfully acquired lease ingress-nginx/ingress-controller-leader-nginx
I1113 15:55:50.138713       6 status.go:86] new leader elected: nginx-ingress-controller-g8hll
E1113 15:55:50.363819       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:55:50.363864       6 queue.go:130] requeuing initial-sync, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:55:53.466414       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:55:53.689303       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:55:53.689329       6 queue.go:130] requeuing kube-system/metrics-server, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:55:56.799713       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:55:57.022714       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:55:57.022751       6 queue.go:130] requeuing default/kubernetes, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:55:57.071861       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:00.133039       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:00.356490       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:00.356521       6 queue.go:130] requeuing ingress-nginx/default-http-backend, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:56:03.466347       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:03.690128       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:03.690157       6 queue.go:130] requeuing kube-system/kube-controller-manager, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:05.285981       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:06.799663       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:07.023039       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:07.023069       6 queue.go:130] requeuing kube-system/kube-dns, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:07.072056       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:10.133111       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:10.355413       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:10.355452       6 queue.go:130] requeuing kube-system/kube-scheduler, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:56:13.466330       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:13.690602       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:13.690634       6 queue.go:130] requeuing ingress-nginx/nginx-ingress-controller-pghqs, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:15.286079       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:16.799667       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:17.026462       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:17.026551       6 queue.go:130] requeuing ingress-nginx/nginx-ingress-controller-g8hll, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:17.071928       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:20.133066       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:20.356144       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:20.356181       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
I1113 15:56:23.466368       6 controller.go:133] Configuration changes detected, backend reload required.
E1113 15:56:23.690407       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
W1113 15:56:23.690438       6 queue.go:130] requeuing configmap-change, err 
-------------------------------------------------------------------------------
Error: signal: illegal instruction (core dumped)

-------------------------------------------------------------------------------
E1113 15:56:25.286204       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I1113 15:56:25.324233       6 main.go:154] Received SIGTERM, shutting down
I1113 15:56:25.324265       6 nginx.go:401] Shutting down controller queues
I1113 15:56:25.324296       6 status.go:117] updating status of Ingress rules (remove)
I1113 15:56:25.442537       6 status.go:127] leaving status update for next leader (2)
I1113 15:56:25.442569       6 nginx.go:417] Stopping NGINX process
I1113 15:56:25.660041       6 main.go:158] Error during shutdown: signal: illegal instruction (core dumped)
I1113 15:56:25.660085       6 main.go:162] Handled quit, awaiting Pod deletion
E1113 15:56:27.071711       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory

What you expected to happen: nginx-ingress containers status to be Running and working properly

How to reproduce it (as minimally and precisely as possible):
Deploy a Custom Kubernetes cluster using Rancher v2.3.2
This is my cluster configuration file

# 
# Cluster Config
# 
answers: {}
docker_root_dir: /var/lib/docker
enable_cluster_alerting: false
enable_cluster_monitoring: true
enable_network_policy: false
local_cluster_auth_endpoint:
  enabled: true
name: cujae
# 
# Rancher Config
# 
rancher_kubernetes_engine_config:
  addon_job_timeout: 30
  authentication:
    strategy: x509|webhook
  authorization: {}
  bastion_host:
    ssh_agent_auth: false
  cloud_provider: {}
  ignore_docker_version: true
# 
# # Currently only nginx ingress provider is supported.
# # To disable ingress controller, set `provider: none`
# # To enable ingress on specific nodes, use the node_selector, eg:
#    provider: nginx
#    node_selector:
#      app: ingress
# 
  ingress:
    provider: nginx
  kubernetes_version: v1.15.5-rancher1-2
  monitoring:
    provider: metrics-server
# 
#   If you are using calico on AWS
# 
#    network:
#      plugin: calico
#      calico_network_provider:
#        cloud_provider: aws
# 
# # To specify flannel interface
# 
#    network:
#      plugin: flannel
#      flannel_network_provider:
#      iface: eth1
# 
# # To specify flannel interface for canal plugin
# 
#    network:
#      plugin: canal
#      canal_network_provider:
#        iface: eth1
# 
  network:
    options:
      flannel_backend_type: vxlan
    plugin: calico
  restore:
    restore: false
# 
#    services:
#      kube-api:
#        service_cluster_ip_range: 10.43.0.0/16
#      kube-controller:
#        cluster_cidr: 10.42.0.0/16
#        service_cluster_ip_range: 10.43.0.0/16
#      kubelet:
#        cluster_domain: cluster.local
#        cluster_dns_server: 10.43.0.10
# 
  services:
    etcd:
      backup_config:
        enabled: true
        interval_hours: 12
        retention: 6
        safe_timestamp: false
      creation: 12h
      extra_args:
        election-timeout: '5000'
        heartbeat-interval: '500'
      gid: 0
      retention: 72h
      snapshot: false
      uid: 0
    kube-api:
      always_pull_images: false
      pod_security_policy: false
      service_node_port_range: 30000-32767
    kube-controller: {}
    kubelet:
      fail_swap_on: false
    kubeproxy: {}
    scheduler: {}
  ssh_agent_auth: false

Anything else we need to know:
I'm running Docker behind a proxy
@aledbf
Copy link
Member

aledbf commented Nov 13, 2019

Closing. Please check #4311

@aledbf aledbf closed this as completed Nov 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@aledbf @arbermejo