Custom port + SSL install seems broken in redirection and LetsEncrypt request

[michiels]

I just tested enabling SSL via LetsEncrypt and enabling custom port on my instance. It appears that when enabling both options there are two problems:

1. Normally, nginx is configured to redirect a http:// to https:// when SSL is enabled. It still does that but it redirects to the same http port. So for example http://michiel.intercity.io:8880 is redirected to https://michiel.intercity.io:8880 but that should be to port 8843.
2. The SSL certificate file under /shared/ssl/michiel.intercity.io.cer seems empty. Making nginx trigger the following error:

root@michiel:/shared/ssl# nginx -t
nginx: [emerg] PEM_read_bio_X509_AUX("/shared/ssl/michiel.intercity.io.cer") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expecting: TRUSTED CERTIFICATE)
nginx: configuration file /etc/nginx/nginx.conf test failed

Perhaps LetsEncrypt needs the server to be available on :80 or we need to set a custom port number when requesting the cert at:8880 or something?

[jvanbaarsen]

@michiels The reason it is empty, is because the cert was not issues yet. Also, how many times did you try with the michiel.intercity.io domain in the past week?

[michiels]

@jvanbaarsen Ok, I tried 4 times or so. Shall I try and reproduce with a new domain name?

[jvanbaarsen]

From: https://letsencrypt.org/docs/rate-limits/

We also have a Duplicate Certificate limit of 5 certificates per week. A certificate is considered a duplicate of an earlier certificate if they contain the exact same set of hostnames, ignoring capitalization and ordering of hostnames. For instance, if you requested a certificate for the names [www.example.com, example.com], you could request four more certificates for [www.example.com, example.com] during the week. If you changed the set of names by adding [blog.example.com], you would be able to request additional certificates.

[jvanbaarsen]

And yeah, their rate limiting is awful

[michiels]

@jvanbaarsen Great, thanks for the pointer. I'll try and reproduce with a new domain name. I think point 1: the redirection issue is still applicable tho.

[jvanbaarsen]

@michiels You can use a different subdomain :-) Also, is the domain reachable for Let's Encrypt? I'll look into the first point

[michiels]

@jvanbaarsen in the first test it was, because on the subsequent installs (without custom port setting) everything worked perfect. I'll now test with michiel2.intercity.io should also be reachable for LetsEncrypt in the DNS.

If it succeeds for me, I think the user that reported this issue initially, hit the rate limiting by LetsEncrypt too. For a future improvement: would it be an option to first see if we can request the cert via LetsEncrypt before we set up nginx to use it?

[jvanbaarsen]

@michiels There is no real way of checking that :( Every check will also count as a request.

[michiels]

@jvanbaarsen Okay, but how are you receiving/putting the certificate in the file at the moment? That can be the final verification right?

[jvanbaarsen]

We tell acme.sh where it needs to store the SSL cert, and acme.sh does all the magic under the good (https://github.com/Neilpang/acme.sh)

[michiels]

@jvanbaarsen ok, but then we could periodically poll for the contents of the expected certificate file, right?

[jvanbaarsen]

@michiels Where do you want to poll this? We have nothing running on the system to check this?

[michiels]

@jvanbaarsen and I had a call and I think we have concluded that custom ports + SSL are currently not supported and this bug should be turned into some kind of enhancement/feature request that takes a bit more work than just fixing a bug.

[jvanbaarsen]

@michiels Yes, sounds about right!

[jvanbaarsen]

@michiels I have been playing around with Docker / Let's encrypt a little, and came to the following idea. Let me know what you think:

We install Nginx on the host machine, rip out the nginx config out of the Intercity-docker container. That way we can easily manage restarts of the SSL etc on the host machine.
We create custom Let's Encrypt docker image (Already 70% done), and have that take care of the cert creation.

The benefit of having NGINX on the host machine, is that we can add a nginx.conf file to tell nginx to redirect traffic to http://domain.com/.well-known/acme/uniq-code (The path let's encrypt checks). So we can have cert issuing without down time.

[jvanbaarsen]

@michiels See: https://github.com/intercity/letsencrypt

[michiels]

@jvanbaarsen I think it sounds sensible. How will you combine the nginx on the host machine with what Dokku installs as host nginx? Is that the same thing? Eventually you want Dokku to manage the nginx after it being installed, or at least not clash with the custom letsencrypt setup that you will make. If they can live alongside 👍

[jvanbaarsen]

@michiels I just create a custom nginx config file (The same as Dokku does) for Intercity. We might need to build some plumbing around it to make it all click correctly. But I think this is the way to go :)

[michiels]

@jvanbaarsen great! If it's just a matter of adding a sites-available or a new file in conf.d and Dokku also does that they can perfectly run alongside! I also agree that having a main nginx router on the host makes lots of these things handy dandy

[ariejan]

There's a dokku plugin nowadays for Letsencrypt that works, with a few caveats. The most important issue is that you need to have your app deployed already before requesting a certificate.

I don't think it'd be all that difficult to add support for LE this way.

I guess you already tackled this in #48?

[jvanbaarsen]

@ariejan I wonder if there is a way we can check if the app is deployed or not. Because there is a very low number throttle on the amount of request you can do to LE (At least that used to be the case).

[ariejan]

There is, using ps:report <app>. Also, checking if DNS is working correctly for each domain should be possible. When all checks are green, you can push the 'Letsencrypt' button.

root@ubuntu-8gb-nbg1-1 ~ # dokku ps:report wordcraft-io
=====> wordcraft-io process information
       Processes:           0                        
       Deployed:            false                    
       Running:             false                    
       Restore:             true                     
       Restart policy:      on-failure:10 

[jvanbaarsen]

Oh nice! We also need to have some check in place to see if the domain is reachable, right?

So the following checks need to pass:

• Custom domain is added
• ps:report returns all green
• domain is actually reachable

If those pass, LE can be enabled for that specific app.

[ariejan]

A custom domain is not a requirement IMHO. You could use the (ugly) generated subdomain vhost.

[jvanbaarsen]

@ariejan Are you sure? I know in the past that didn't work (for reasons that I don't know anymore). I have the feeling a lot has changes with LE since the last time we tried this, so it might indeed be good to give the whole LE plugin another go :)