[solved] Deploy Che-multiuser behind Nginx proxy with https using only FQDN

[Mbd06b]

I've been trying to install Eclipse Che on and off for about a year and a half now on a private Ubuntu(18-LTS) Server that I can access with a domain name using a Dynamic DNS record. Before I jump down this rabbit hole again. With the release of Eclipse Che 7, is it reasonable to:

Strategy 1. Deploy Che-multiuser behind a (non-ingress) Nginx proxy with https termination using a wildcard FQDN.

Strategy 2. Deploy Che-mulituser alongside a (non-ingress) Nginx with https termination occurring at the Che server on minikube.

Strategy 3. Deploy an accessible Che-single user with https and some other password protection/login strategy(non-VPN)??

Strategy 1 makes the most logical sense to me at this point, if anyone has a working sample Nginx config for this, and what arguments they are using, I'd appreciate you sharing it, before I light into this next release attempt.

Edit: Noticed that docker install is being deprecated from eclipse 7. I'm narrowing my focus to minikube. (Update: Minikube seems tailored for Windows/Mac users, I ultimately focused on Microk8s, because it doesn't require the VM overhead, and it's snap package is directly supported by Canonical, seemed to be a better choice for my target Ubuntu environment).

The purpose of this is to have a https/login (keycloak) secured private instance of Eclipse Che that will continuously run on a single server(node) that I can access anywhere and share with others. I develop heavily on local IDEs, but would like migrate more traditional developer workspaces/workflows to a self-hosted cloud, while increasing the overall utilization efficiency of my own running bare-metal server I use to DevOps those projects. The idea being purchasing PaaS or IaaS resources soley to run Che would theoretically be redundant and wasteful to this goal.

Relevant information

[tolusha]

/cc @skabashnyuk @sleshchenko

[Mbd06b]

Side note, ingress nginx > 0.25.0 will fail to run on CPUs that don't support SSE4 instructions due to a dependency,

See: kubernetes/ingress-nginx#4311

I was able to get mine to run on my Xeon E5310 by editing the deployed image version manually to 0.24.1
kubectl edit -n ingress daemonset.apps/nginx-ingress-microk8s-controller

[Mbd06b]

I also had to follow this workaround to get chectl to work with microk8s context #14455 (comment)

[Mbd06b]

My nginx logs indicate that I have a redirect loop in my config. I need some way of proxying back to my ingresses using a non-standard localhost or port. Listing the ingresses from my install,

HOW TO CHANGE THESE IP ADDRESSES OR PORTS?
kubectl get ingress --all-namespaces
che che-ingress che-che.code.example.com 127.0.0.1 80 2d11h
che devfile-registry devfile-registry-che.code.example.com 127.0.0.1 80 2d11h
che keycloak-ingress keycloak-che.code.example.com 127.0.0.1 80 2d11h
che plugin-registry plugin-registry-che.code.example.com 127.0.0.1 80 2d11h

[tolusha]

/cc @sleshchenko

[Mbd06b]

Solution

This single-host multiuser deployment on microk8s seems to be mostly working for Theia-based editor workspaces.
I'm still getting errors trying to launch an Eclipse IDE workspaces.

Any recommendations on cleaning up this nginx config would be appreciated. Best practices for how to proxy headers is a bit of a crap-shoot.

### Here's the solution/strategy...
omnibus Nginx Proxy config using microk8s snap on Ubuntu 18 LTS

sudo vim /etc/nginx/sites-enabled/code.example.conf

upstream chekubeingress {
server 127.0.0.2:8880; #ingress daemonset.app http port arg
}
upstream sslcheingress{
server 127.0.0.2:4443; #ingress daemonset.app https-port port arg
}
server {

 listen 80;
 server_name  *.code.example.com;

 location / {

   proxy_pass http://chekubeingress;
   proxy_set_header Upgrade $http_upgrade;
   proxy_set_header Connection "upgrade";
   proxy_http_version 1.1;
   proxy_connect_timeout 7d;
   proxy_send_timeout 7d;
   proxy_read_timeout 7d;
   proxy_set_header Host              $host;
 proxy_pass http://chekubeingress;

 }

}
server {

listen 443 ssl; # managed by Certbot
server_name  *.code.example.com;

ssl on;

#manually generated wildcard domain, this should be a valid cert for the root domain as well
#sudo certbot --server https://acme-v02.api.letsencrypt.org/directory -d code.example.com -d *.code.example.com --manual --preferred-challenges dns-01 certonly

ssl_certificate /etc/letsencrypt/live/code.example.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/code.example.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

location / {

    **#For wss:// websocket connections
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_connect_timeout 7d;
      proxy_send_timeout 7d;
      proxy_read_timeout 7d;**

    #for upstream ingress routing in kubernetes
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host              $host;
      proxy_pass https://sslcheingress;
}

}

The trick was to edit the ingress controller deployment args to listen on a non-standard ip address and ports to prevent listening conflict with nginx already listening on port 80 and 443.

- --publish-status-address=127.0.0.2 --http-port=8880 --https-port=4443

The ingresses in microk8s, with the following args still listen on port 80 and 443, but they are on a non-standard unique localhost IP address 127.0.0.2 ports :80 :443
kubectl edit daemonset.apps/nginx-ingress-microk8s-controller -n ingress

apiVersion: apps/v1
kind: DaemonSet
metadata:
annotations:
deprecated.daemonset.template.generation: "3"
generation: 3
labels:
microk8s-application: nginx-ingress-microk8s
name: nginx-ingress-microk8s-controller
namespace: ingress
resourceVersion: "19117"
selfLink: /apis/apps/v1/namespaces/ingress/daemonsets/nginx-ingress-microk8s-controller
uid: ead1ea64-a5e5-4a5f-ac52-0a1a419c0bbc
spec:
revisionHistoryLimit: 10
selector:
matchLabels:
name: nginx-ingress-microk8s
template:
metadata:
creationTimestamp: null
labels:
name: nginx-ingress-microk8s
spec:
containers:

• args:
• /nginx-ingress-controller
• --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf
  **- --publish-status-address=127.0.0.2
• --http-port=8880
• --https-port=4443**
  env:
• name: POD_NAME
  valueFrom:
  fieldRef:

Add the certbot generated wildcard certificates to kubernetes "che" namespace...
kubectl create secret tls che-tls --namespace='che' --key=/etc/letsencrypt/live/code.example.com/privkey.pem --cert=/etc/letsencrypt/live/code.example.com/fullchain.pem

I configured microk8s where it could be accessed with standard commands (in lieu of prefixed versions microk8s.kubectl ...) this seems to be required for chectl to find and use the Kubernetes API
$ kubectl
$ helm

Launch Eclipse Che
sudo chectl server:start --platform=microk8s --multiuser --installer=helm --chenamespace=che --domain=code.example.com --tls

$ kubectl get ingress --all-namespaces
NAMESPACE NAME HOSTS ADDRESS PORTS
che che-ingress che-che.code.example.com 127.0.0.2 80, 443
che devfile-registry devfile-registry-che.code.example.com 127.0.0.2 80, 443
che keycloak-ingress keycloak-che.code.example.com 127.0.0.2 80, 443
che plugin-registry plugin-registry-che.code.example.com 127.0.0.2 80, 443

[tolusha]

@Mbd06b
Thank you for detailed instruction.