Better starting flow with default certificate

[azlux]

(Bug)? description

During the start, the default certificate is present for all domains (certificate named "letsencrypt-nginx-proxy-companion")

I currently have around 30 domains, it's take time, so I receive alerts saying my domain isn't good on my monitoring tool, nextcloud clients, smartphone....
Is that possible to say to the acme-companion to start with existing certificate without verify them one by one. Or making an improvement to do it ?
The script can check them in a second step.

I suppose I can "avoid" the issue on the main domain with a mount point to have a file at /etc/nginx/certs/default.cr :
https://github.com/nginx-proxy/acme-companion/blob/main/app/entrypoint.sh#L110-L117
But I don't think it's enough.

What's your advice ?

docker image version : 1a7f5b3bc6cf
acme-companion version : v2.2.1

---

nginx-proxy's Docker configuration

Click to expand!

    nginx:
        image: nginxproxy/nginx-proxy
        container_name: nginx
        hostname: nginx
        restart: on-failure
        ports:
            - "80:80"
            - "443:443"
        environment:
            ENABLE_IPV6: "true"
            TZ: "Europe/*****"
        volumes:
            - /var/run/docker.sock:/tmp/docker.sock:ro
            - ../data/nginx/certs:/etc/nginx/certs:ro
            - ../data/nginx/conf.d:/etc/nginx/conf.d
            - ../data/nginx/htpasswd:/etc/nginx/htpasswd
            - ../data/nginx/vhost.d:/etc/nginx/vhost.d
            - ../data/nginx/dhparam:/etc/nginx/dhparam
            - ../data/nginx/logs:/var/log/nginx
            - ../data/nginx/html:/usr/share/nginx/html
        networks:
            default:
                ipv4_address: 172.18.0.100

    letsencrypt:
        image: nginxproxy/acme-companion
        container_name: letsencrypt
        hostname: letsencrypt
        environment:
            DEFAULT_EMAIL: ********
            NGINX_PROXY_CONTAINER: nginx
        volumes_from:
            - nginx
        volumes:
            - ../data/nginx/certs:/etc/nginx/certs:rw
            - ../data/letsencrypt/acme:/etc/acme.sh
            - /var/run/docker.sock:/var/run/docker.sock:ro
        depends_on:
          - nginx

Containers logs

Click to expand!

letsencrypt         | Info: running acme-companion version v2.2.1
letsencrypt         | Info: A custom dhparam.pem file was provided. Best practice is to use standardized RFC7919 Diffie-Hellman groups instead.
letsencrypt         | Reloading nginx proxy (nginx)...
letsencrypt         | 2022/04/25 16:30:14 Generated '/etc/nginx/conf.d/default.conf' from 47 containers
letsencrypt         | Warning: /app/letsencrypt_service_data not found, skipping data from containers.
letsencrypt         | 2022/04/25 14:30:19 Generated '/app/letsencrypt_service_data' from 47 containers
letsencrypt         | 2022/04/25 14:30:19 Running '/app/signal_le_service'
letsencrypt         | 2022/04/25 14:30:19 Watching docker events
letsencrypt         | 2022/04/25 14:30:19 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service'
letsencrypt         | Reloading nginx proxy (nginx)...
letsencrypt         | 2022/04/25 16:30:20 Generated '/etc/nginx/conf.d/default.conf' from 47 containers
letsencrypt         | Reloading nginx proxy (nginx)...
letsencrypt         | 2022/04/25 16:30:20 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
letsencrypt         | Creating/renewal azlux.fr certificates... (azlux.fr z.azlux.fr)
letsencrypt         | [Mon Apr 25 14:30:28 UTC 2022] Domains not changed.
letsencrypt         | [Mon Apr 25 14:30:28 UTC 2022] Skip, Next renewal time is: Thu May 26 16:09:38 UTC 2022
letsencrypt         | [Mon Apr 25 14:30:28 UTC 2022] Add '--force' to force to renew.
letsencrypt         | Creating/renewal a.azlux.fr certificates... (a.azlux.fr)
letsencrypt         | [Mon Apr 25 14:30:29 UTC 2022] Domains not changed.
letsencrypt         | [Mon Apr 25 14:30:29 UTC 2022] Skip, Next renewal time is: Thu May 26 16:10:00 UTC 2022
letsencrypt         | [Mon Apr 25 14:30:29 UTC 2022] Add '--force' to force to renew.
letsencrypt         | [Mon Apr 25 14:30:29 UTC 2022] Add '--force' to force to renew.
letsencrypt         | Reloading nginx proxy (nginx)...
letsencrypt         | 2022/04/25 16:30:30 Generated '/etc/nginx/conf.d/default.conf' from 47 containers
letsencrypt         | Creating/renewal b.azlux.fr certificates... (b.azlux.fr)
letsencrypt         | [Mon Apr 25 14:30:33 UTC 2022] Domains not changed.
letsencrypt         | [Mon Apr 25 14:30:33 UTC 2022] Skip, Next renewal time is: Thu May 26 16:10:16 UTC 2022
letsencrypt         | [Mon Apr 25 14:30:33 UTC 2022] Add '--force' to force to renew.
letsencrypt         | Reloading nginx proxy (nginx)...
letsencrypt         | 2022/04/25 16:30:33 Generated '/etc/nginx/conf.d/default.conf' from 47 containers
letsencrypt         | Creating/renewal c.azlux.fr certificates... (c.azlux.fr)
letsencrypt         | [Mon Apr 25 14:30:34 UTC 2022] Domains not changed.
letsencrypt         | [Mon Apr 25 14:30:34 UTC 2022] Skip, Next renewal time is: Thu May 26 16:10:34 UTC 2022
letsencrypt         | [Mon Apr 25 14:30:34 UTC 2022] Add '--force' to force to renew.
letsencrypt         | Creating/renewal d.azlux.fr certificates... (d.azlux.fr)
letsencrypt         | [Mon Apr 25 14:30:37 UTC 2022] Domains not changed.
letsencrypt         | [Mon Apr 25 14:30:37 UTC 2022] Skip, Next renewal time is: Thu May 26 16:10:47 UTC 2022
letsencrypt         | [Mon Apr 25 14:30:37 UTC 2022] Add '--force' to force to renew.
letsencrypt         | Creating/renewal e.azlux.fr certificates... (e.azlux.fr)
letsencrypt         | 2022/04/25 14:30:38 Received event start for container f97316d1521a
[...]

Docker host

• OS: Debian 11.3
• Docker version: Docker version 20.10.5+dfsg1, build 55c4c88

[fschrempf]

I have the same problem. As soon as a container is down (e.g. due to maintenance or running a backup) clients start to show a certificate error as they see the letsencrypt-nginx-proxy-companion default cert. Instead I would like the clients to see a valid certificate and get an error from the proxy that the target is not available.

[buchdag]

@azlux

Is that possible to say to the acme-companion to start with existing certificate without verify them one by one. The script can check them in a second step.

I think that's a good idea, I'll look into it.

@fschrempf that's a different issue, but also related to default certificate mechanism.

When a container is down and you send an HTTPS request to its former VIRTUAL_HOST, a few scenarios are possibles:

• you have no default certificate (ie not using acme-companion) : you'll get a 444 (missing cert) error.
• you have the default certificate provided by acme-companion : you'll get a certificate error, then a 503 error if ignored.
• you provided your own default certificate that cover the VIRTUAL_HOST you used in the request : you'll get a 503 error and no certificate error.

(There are a few others when using nginx-proxy's DEFAULT_HOST, but they're not covered here).

[fschrempf]

Ok, thanks @buchdag! So I can provide a default (wildcard) certificate that covers all my virtual hosts as a fallback to avoid certificate errors, right? But this certificate can't be managed (fetch, renew, etc.) by the acme-companion?

[buchdag]

So I can provide a default (wildcard) certificate that covers all my virtual hosts as a fallback to avoid certificate errors, right ?

Yep, you'll just have to mount both certificate and private key file to /etc/nginx/certs in the nginx-proxy container.

But this certificate can't be managed (fetch, renew, etc.) by the acme-companion ?

Unfortunately no(t yet) , I know this is a feature that's been requested for a long time. 😞

[azlux]

Why not just save the config/state in case of restart ?
So have a faster working config, even if you will check the full config after one container by one ?

[buchdag]

Re-reading your initial message, I'm not certain of the exact issue you're having anymore

Let say :

• I start my nginx-proxy + acme-companion containers
• I start 30 service containers and obtain a certificate for each one of them
• I stop my nginx-proxy + acme-companion containers then wait one hour (without stoping the other service containers)
• I start my nginx-proxy + acme-companion containers again

Unless I'm completely misremembering/misreading the code or you're losing data at some point, every 30 services should almost instantaneously become up and serve the correct certificate.

When you shut down the acme-companion containers, active certificate symlinks should not be removed from the /etc/nginx/certs volume. Are they ? If they are, that's not intended.

Could you provide a more precise sequence of what containers (including proxied services) you start, stop and restart and when ? 🤔

[azlux]

Hi,
Restarting the acme-companion to nothing, restarting the nginx-proxy is good too.
I think it's during an update, when I pull a new version so recreate the running image of the acme, I see the issue.

step:

• I start my nginx-proxy + acme-companion containers
• I start 30 service containers and obtain a certificate for each one of them
• I stop my acme-companion containers and remove it.
• I start my acme-companion containers again so it create a new one.

I can see on the acme-companion logs check all my domains one after the other:

letsencrypt | 2024/01/19 01:21:07 Generated '/etc/nginx/conf.d/default.conf' from 48 containers
letsencrypt | Warning: /app/letsencrypt_service_data not found, skipping data from containers.
letsencrypt | 2024/01/19 00:21:08 Generated '/app/letsencrypt_service_data' from 48 containers
letsencrypt | 2024/01/19 00:21:08 Running '/app/signal_le_service'
letsencrypt | 2024/01/19 00:21:08 Watching docker events
letsencrypt | 2024/01/19 00:21:08 Contents of /app/letsencrypt_service_data did not change. Skipping notification '/app/signal_le_service'
letsencrypt | Reloading nginx proxy (nginx)...
letsencrypt | 2024/01/19 01:21:08 Generated '/etc/nginx/conf.d/default.conf' from 48 containers
letsencrypt | Reloading nginx proxy (nginx)...
letsencrypt | 2024/01/19 01:21:09 Contents of /etc/nginx/conf.d/default.conf did not change. Skipping notification ''
letsencrypt | Creating/renewal azlux.fr certificates... (azlux.fr mumble.azlux.fr)
letsencrypt | [Fri Jan 19 00:21:10 UTC 2024] Domains not changed.
letsencrypt | [Fri Jan 19 00:21:10 UTC 2024] Skip, Next renewal time is: Tue Mar 5 21:30:18 UTC 2024
letsencrypt | [Fri Jan 19 00:21:10 UTC 2024] Add '--force' to force to renew.
letsencrypt | Reloading nginx proxy (nginx)...
letsencrypt | 2024/01/19 01:21:11 Generated '/etc/nginx/conf.d/default.conf' from 48 containers
letsencrypt | Creating/renewal ci.azlux.fr certificates... (ci.azlux.fr)
letsencrypt | [Fri Jan 19 00:21:16 UTC 2024] Domains not changed.
[....]

Then certificate will be good one after one. After the acme check for renewal.
Before the acme-companion have check the validity of my domain, I have the self-signed issuer: CN=letsencrypt-nginx-proxy-companion certificate. During that time, my nextcloud sync and phone stop because of the certificate.

[azlux]

Have you successfully reproduce the issue ? Do you consider it's one real issue or a normal behavior ?