Letsencrypt auto-renewal appears to have broken my proxy-ports settings.

[cooperaj]

Description of problem

At midnight last night I got a notification that my service was down. I investigated as was seeing these errors in the log:

2023/01/13 00:00:41 [error] 1988130#1988130: *2599409 SSL_do_handshake() failed (SSL: error:0A00010B:SSL routines::wrong version number) while SSL handshaking to upstream, client: 31.132.6.98, server: social.n8e.dev, request: "POST /inbox HTTP/2.0", upstream: "https://172.17.0.14:80/inbox", host: "social.n8e.dev"

This pointed me at my port maps being incorrect. True enough they were now set to http:80:80 https:443:80 instead of the functioning http:80:80 https:443:443 that they were originally set to.

I have not got auto updates enabled on the server so it took me a while to find out what happens at midnight. Turns out its the letsencrypt auto-renew cronjob that the plugin uses. It also happened to renew this domain at midnight last night.

=====> Enabling letsencrypt for social.n8e.dev
-----> Enabling ACME proxy for social.n8e.dev...
-----> Getting letsencrypt certificate for social.n8e.dev...
        - Domain 'social.n8e.dev'
       2023/01/13 00:00:25 [INFO] [social.n8e.dev] acme: Obtaining bundled SAN certificate
       2023/01/13 00:00:26 [INFO] [social.n8e.dev] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/194708765317
       2023/01/13 00:00:26 [INFO] [social.n8e.dev] acme: Could not find solver for: tls-alpn-01
       2023/01/13 00:00:26 [INFO] [social.n8e.dev] acme: use http-01 solver
       2023/01/13 00:00:26 [INFO] [social.n8e.dev] acme: Trying to solve HTTP-01
       2023/01/13 00:00:27 [INFO] [social.n8e.dev] Served key authentication
       2023/01/13 00:00:27 [INFO] [social.n8e.dev] Served key authentication
       2023/01/13 00:00:27 [INFO] [social.n8e.dev] Served key authentication
       2023/01/13 00:00:32 [INFO] [social.n8e.dev] The server validated our request
       2023/01/13 00:00:32 [INFO] [social.n8e.dev] acme: Validations succeeded; requesting certificates
       2023/01/13 00:00:33 [INFO] [social.n8e.dev] Server responded with a certificate.
-----> Certificate retrieved successfully.
-----> Installing let's encrypt certificates
-----> Unsetting DOKKU_PROXY_PORT
-----> Unsetting DOKKU_PROXY_SSL_PORT
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:80
-----> Setting config vars
       DOKKU_PROXY_PORT_MAP:  http:80:80 https:443:80
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
       Enabling HSTS
       Reloading nginx
-----> Overriding default nginx.conf with detected nginx.conf.sigil
-----> Configuring social.n8e.dev...(using app-supplied template)
-----> Creating https nginx.conf
       Enabling HSTS
       Reloading nginx
-----> Disabling ACME proxy for social.n8e.dev...
-----> Done

I've got no idea why it does the port map setting, nor why it does it twice. I also don't know why it regens and reloads nginx twice. Either way the configurations it used both times were not correct.

Steps to reproduce

Run letsencrypt plugin and have it renew a domain that has the port map http:80:80 https:443:443

dokku report $APP_NAME

-----> uname: Linux n8e 5.15.0-56-generic dokku/dokku#62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory: 
                      total        used        free      shared  buff/cache   available
       Mem:            3923        2384         167         151        1372        1123
       Swap:           7999        1790        6209
-----> docker version: 
       Client: Docker Engine - Community
        Version:           20.10.22
        API version:       1.41
        Go version:        go1.18.9
        Git commit:        3a2c30b
        Built:             Thu Dec 15 22:28:04 2022
        OS/Arch:           linux/amd64
        Context:           default
        Experimental:      true
       
       Server: Docker Engine - Community
        Engine:
         Version:          20.10.22
         API version:      1.41 (minimum version 1.12)
         Go version:       go1.18.9
         Git commit:       42c8b31
         Built:            Thu Dec 15 22:25:49 2022
         OS/Arch:          linux/amd64
         Experimental:     false
        containerd:
         Version:          1.6.14
         GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
        runc:
         Version:          1.1.4
         GitCommit:        v1.1.4-0-g5fd4c4d
        docker-init:
         Version:          0.19.0
         GitCommit:        de40ad0
-----> docker daemon info: 
       Client:
        Context:    default
        Debug Mode: true
        Plugins:
         app: Docker App (Docker Inc., v0.9.1-beta3)
         buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
         compose: Docker Compose (Docker Inc., v2.14.1)
         scan: Docker Scan (Docker Inc., v0.23.0)
       
       Server:
        Containers: 23
         Running: 18
         Paused: 0
         Stopped: 5
        Images: 219
        Server Version: 20.10.22
        Storage Driver: overlay2
         Backing Filesystem: extfs
         Supports d_type: true
         Native Overlay Diff: true
         userxattr: false
        Logging Driver: json-file
        Cgroup Driver: systemd
        Cgroup Version: 2
        Plugins:
         Volume: local local-persist
         Network: bridge host ipvlan macvlan null overlay
         Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
        Swarm: inactive
        Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
        Default Runtime: runc
        Init Binary: docker-init
        containerd version: 9ba4b250366a5ddde94bb7c9d1def331423aa323
        runc version: v1.1.4-0-g5fd4c4d
        init version: de40ad0
        Security Options:
         apparmor
         seccomp
          Profile: default
         cgroupns
        Kernel Version: 5.15.0-56-generic
        Operating System: Ubuntu 22.04.1 LTS
        OSType: linux
        Architecture: x86_64
        CPUs: 2
        Total Memory: 3.832GiB
        Name: n8e
        ID: CIWV:QUKD:AS4D:VIBI:7UCX:ZGG7:CVGQ:LWOU:NMUR:MRTV:XJNX:SUG4
        Docker Root Dir: /var/lib/docker
        Debug Mode: false
        Registry: https://index.docker.io/v1/
        Labels:
        Experimental: false
        Insecure Registries:
         127.0.0.0/8
        Live Restore Enabled: false
       
-----> git version: git version 2.34.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version: 
       herokuish: v0.5.40
       buildpacks:
         heroku-buildpack-multi     v1.2.0
         heroku-buildpack-ruby      v244
         heroku-buildpack-nodejs    v202
         heroku-buildpack-clojure   v90
         heroku-buildpack-python    v223
         heroku-buildpack-java      v72
         heroku-buildpack-gradle    v38
         heroku-buildpack-scala     v96
         heroku-buildpack-play      v26
         heroku-buildpack-php       v227
         heroku-buildpack-go        v169
         heroku-buildpack-nginx     v22
         buildpack-null             v3
-----> dokku version: dokku version 0.29.3
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins: 
         00_dokku-standard    0.29.3 enabled    dokku core standard plugin
         20_events            0.29.3 enabled    dokku core events logging plugin
         app-json             0.29.3 enabled    dokku core app-json plugin
         apps                 0.29.3 enabled    dokku core apps plugin
         builder              0.29.3 enabled    dokku core builder plugin
         builder-dockerfile   0.29.3 enabled    dokku core builder-dockerfile plugin
         builder-herokuish    0.29.3 enabled    dokku core builder-herokuish plugin
         builder-lambda       0.29.3 enabled    dokku core builder-lambda plugin
         builder-null         0.29.3 enabled    dokku core builder-null plugin
         builder-pack         0.29.3 enabled    dokku core builder-pack plugin
         buildpacks           0.29.3 enabled    dokku core buildpacks plugin
         caddy-vhosts         0.29.3 enabled    dokku core caddy-vhosts plugin
         certs                0.29.3 enabled    dokku core certificate management plugin
         checks               0.29.3 enabled    dokku core checks plugin
         common               0.29.3 enabled    dokku core common plugin
         config               0.29.3 enabled    dokku core config plugin
         cron                 0.29.3 enabled    dokku core cron plugin
         docker-options       0.29.3 enabled    dokku core docker-options plugin
         domains              0.29.3 enabled    dokku core domains plugin
         enter                0.29.3 enabled    dokku core enter plugin
         git                  0.29.3 enabled    dokku core git plugin
         letsencrypt          0.18.1 enabled    Automated installation of let's encrypt TLS certificates
         logs                 0.29.3 enabled    dokku core logs plugin
         mongo                1.23.0 enabled    dokku mongo service plugin
         network              0.29.3 enabled    dokku core network plugin
         nginx-vhosts         0.29.3 enabled    dokku core nginx-vhosts plugin
         plugin               0.29.3 enabled    dokku core plugin plugin
         postgres             1.26.0 enabled    dokku postgres service plugin
         proxy                0.29.3 enabled    dokku core proxy plugin
         ps                   0.29.3 enabled    dokku core ps plugin
         redirect             0.7.1 enabled    Plugin for managing application redirects
         redis                1.26.2 enabled    dokku redis service plugin
         registry             0.29.3 enabled    dokku core registry plugin
         repo                 0.29.3 enabled    dokku core repo plugin
         resource             0.29.3 enabled    dokku core resource plugin
         run                  0.29.3 enabled    dokku core run plugin
         scheduler            0.29.3 enabled    dokku core scheduler plugin
         scheduler-docker-local 0.29.3 enabled    dokku core scheduler-docker-local plugin
         scheduler-null       0.29.3 enabled    dokku core scheduler-null plugin
         shell                0.29.3 enabled    dokku core shell plugin
         ssh-keys             0.29.3 enabled    dokku core ssh-keys plugin
         storage              0.29.3 enabled    dokku core storage plugin
         trace                0.29.3 enabled    dokku core trace plugin
         traefik-vhosts       0.29.3 enabled    dokku core traefik-vhosts plugin
=====> social.n8e.dev app-json information
       App json computed selected:    app.json
       App json global selected:      app.json
       App json selected:             
=====> social.n8e.dev app information
       App created at:                1670442967
       App deploy source:             social.n8e.dev
       App deploy source metadata:    social.n8e.dev
       App dir:                       /home/dokku/social.n8e.dev
       App locked:                    false
=====> social.n8e.dev builder information
       Builder build dir:             
       Builder computed build dir:    
       Builder computed selected:     
       Builder global build dir:      
       Builder global selected:       
       Builder selected:              
=====> social.n8e.dev builder-dockerfile information
       Builder dockerfile computed dockerfile path: Dockerfile               
       Builder dockerfile global dockerfile path: Dockerfile               
       Builder dockerfile dockerfile path:                          
=====> social.n8e.dev builder-lambda information
       Builder lambda computed lambdayml path: lambda.yml               
       Builder lambda global lambdayml path: lambda.yml               
       Builder lambda lambdayml path:                          
=====> social.n8e.dev builder-pack information
       Builder pack computed projecttoml path: project.toml             
       Builder pack global projecttoml path: project.toml             
       Builder pack projecttoml path:                          
=====> social.n8e.dev buildpacks information
       Buildpacks computed stack:     gliderlabs/herokuish:latest-20
       Buildpacks global stack:       
       Buildpacks list:               
       Buildpacks stack:              
=====> social.n8e.dev ssl information
       Ssl dir:                       /home/dokku/social.n8e.dev/tls
       Ssl enabled:                   true                     
       Ssl hostnames:                 social.n8e.dev           
       Ssl expires at:                Apr 12 23:00:31 2023 GMT 
       Ssl issuer:                    C = US, O = Let's Encrypt, CN = R3
       Ssl starts at:                 Jan 12 23:00:32 2023 GMT 
       Ssl subject:                   subject=CN = social.n8e.dev
       Ssl verified:                  self signed              
=====> social.n8e.dev checks information
       Checks disabled list:          none                     
       Checks skipped list:           none                     
       Checks computed wait to retire: 60                       
       Checks global wait to retire:  60                       
       Checks wait to retire:                                  
=====> social.n8e.dev cron information
       Cron task count:               0
=====> social.n8e.dev docker options information
       Docker options build:          --link dokku.postgres.socialn8edev:dokku-postgres-socialn8edev --link dokku.redis.socialn8edev:dokku-redis-socialn8edev 
       Docker options deploy:         --link dokku.postgres.socialn8edev:dokku-postgres-socialn8edev --link dokku.redis.socialn8edev:dokku-redis-socialn8edev --restart=on-failure:10 -v /var/lib/dokku/data/storage/mastodon:/config 
       Docker options run:            --link dokku.postgres.socialn8edev:dokku-postgres-socialn8edev --link dokku.redis.socialn8edev:dokku-redis-socialn8edev -v /var/lib/dokku/data/storage/mastodon:/config 
=====> social.n8e.dev domains information
       Domains app enabled:           true                     
       Domains app vhosts:            social.n8e.dev           
       Domains global enabled:        true                     
       Domains global vhosts:         n8e.dev                  
=====> social.n8e.dev git information
       Git deploy branch:             main                     
       Git global deploy branch:      master                   
       Git keep git dir:              false                    
       Git rev env var:               GIT_REV                  
       Git sha:                                                
       Git source image:                                       
       Git last updated at:           1671118869               
=====> social.n8e.dev letsencrypt information
       Letsencrypt active:            true                     
       Letsencrypt autorenew:         true                     
       Letsencrypt email:             [email protected]
       Letsencrypt expiration:        1681340431               
=====> social.n8e.dev logs information
       Logs computed max size:        10m
       Logs global max size:          10m
       Logs global vector sink:       
       Logs max size:                 
       Logs vector sink:              
=====> social.n8e.dev network information
       Network attach post create:           
       Network attach post deploy:           
       Network bind all interfaces:          false
       Network computed attach post create:  
       Network computed attach post deploy:  
       Network computed bind all interfaces: false
       Network computed initial network:     
       Network computed tld:                 
       Network global attach post create:    
       Network global attach post deploy:    
       Network global bind all interfaces:   false
       Network global initial network:       
       Network global tld:                   
       Network initial network:              
       Network static web listener:          
       Network tld:                          
       Network web listeners:                172.17.0.14:443
=====> social.n8e.dev nginx information
       Nginx access log format:                                
       Nginx access log path:         /var/log/nginx/social.n8e.dev-access.log
       Nginx bind address ipv4:                                
       Nginx bind address ipv6:       ::                       
       Nginx client max body size:    50m                      
       Nginx disable custom config:   false                    
       Nginx error log path:          /var/log/nginx/social.n8e.dev-error.log
       Nginx global hsts:             true                     
       Nginx computed hsts:           true                     
       Nginx hsts:                                             
       Nginx hsts include subdomains: true                     
       Nginx hsts max age:            15724800                 
       Nginx hsts preload:            false                    
       Nginx computed nginx conf sigil path: nginx.conf.sigil         
       Nginx global nginx conf sigil path: nginx.conf.sigil         
       Nginx nginx conf sigil path:                            
       Nginx proxy buffer size:       4096                     
       Nginx proxy buffering:         on                       
       Nginx proxy buffers:           8 4096                   
       Nginx proxy busy buffers size: 8192                     
       Nginx proxy read timeout:      60s                      
       Nginx last visited at:         1673626009               
       Nginx x forwarded for value:   $remote_addr             
       Nginx x forwarded port value:  $server_port             
       Nginx x forwarded proto value: $scheme                  
       Nginx x forwarded ssl:                                  
=====> social.n8e.dev proxy information
       Proxy enabled:                 true
       Proxy port map:                http:80:80 https:443:443
       Proxy type:                    nginx
=====> social.n8e.dev ps information
       Deployed:                      true
       Processes:                     1
       Ps can scale:                  true
       Ps computed procfile path:     Procfile
       Ps global procfile path:       Procfile
       Ps procfile path:              
       Ps restart policy:             on-failure:10
       Restore:                       true
       Running:                       true
       Status web 1:                  running (CID: c59b4620a82)
=====> social.n8e.dev registry information
       Registry computed image repo:      dokku/social.n8e.dev
       Registry computed push on release: false
       Registry computed server:          
       Registry global push on release:   
       Registry global server:            
       Registry image repo:               
       Registry push on release:          
       Registry server:                   
       Registry tag version:              
=====> social.n8e.dev resource information
=====> social.n8e.dev scheduler information
       Scheduler computed selected:   docker-local
       Scheduler global selected:     docker-local
       Scheduler selected:            
=====> social.n8e.dev scheduler-docker-local information
       Scheduler docker local disable chown:                          
       Scheduler docker local init process: true                     
       Scheduler docker local parallel schedule count:                          
=====> social.n8e.dev storage information
       Storage build mounts:                                   
       Storage deploy mounts:         -v /var/lib/dokku/data/storage/mastodon:/config 
       Storage run mounts:            -v /var/lib/dokku/data/storage/mastodon:/config 

Additional information

No response

Output of failing commands after running: dokku trace:off

No response

Output of failing commands after running: dokku trace:on

No response

[michaelyork]

See referenced issue - ran into the same myself. Happy to help sponsor!

[josegonzalez]

Dokku itself is what is remapping these ports when we add a certificate. It will take any port http:80 mappings and use the container port to make https:443 mappings.

I'd like to know a bit more about the use case here for not doing that. What sort of apps are ya'll running where 80 and 443 should be passed to different ports on the backend, and why can't they be the same port? I would at least need a test case to figure out how to replicate this.

[josegonzalez]

Okay its mastadon doing something weird.

I think maybe in this case, Dokku should probably just ignore adding any https ports if there are already https:443 port mappings to begin with. Dokku can respect that opinion and log a warning saying it isn't changing ports, which should be fine for cases like this.

[cooperaj]

To be clear, we still would like dokku to do SSL termination, we just need it to also communicate with the container over 443 - which it does when you initially configure it, but then letsencrypt runs and removes that mapping.

[josegonzalez]

So the initial mapping is what in your case?

[josegonzalez]

Ah you specified at the top.

Basically if we see an https:443 mapping, don't touch the port mappings. I think thats what I'll change it to.

[gouravkhunger]

Facing a similar issue here. Basically when you add a new domain and enable letsencrypt for an app that already has an existing letsencrypt ssl issued domain, the port mappings reset. Preserving the port 443 mapping would be ideal.