Zero downtime deploy with Dokku
TL;DR we hacked Dokku to check-list new deployments before switching traffic over, eliminating down time. Here's our fork.
Dokku is a wonderful tool for quickly deploying apps to production, but it's missing zero downtime deploys. Not anymore!
Even our svelte Node.js app needs a few seconds to fully load the main code, and open connections to back-ends servers, before it can start handling requests.
The thing with Dokku: it switches traffic over to the new container before the server loads complete, so each deploy has several seconds of downtime.
That's best case scenario. When we screw deployment (usually some configuration issue that's different in production), even the quickest hands in the west, you still get a few minutes of downtime.
And so it became necessary that I learn how to catch and handle exceptions in Bash (yay, traps!) and shift a few lines of code around.
Checks Your Engine
The main piece is a new plugin that checks the newly deployed server.
The plugin runs through a list of checks, each check tells it to request a given URL and verify its contents. If all checks pass, then it's safe to switch traffic to the new server.
Checks looks something like this:
# Run these checks against server to verify it works correctly # Each check takes the form of: # <pathname> <expected contents of page> # # For example: # /about Our Amazing Team / My Amazing App /stylesheets/index.css .body /scripts/index.js $(function() /images/logo.png
You don't have to fuss with the expected content, but it helps against cases where templates don't load properly, LESS styles fail to compile, etc.
Depending on your application, these checks can also warm up the cache, making sure the first live requests are served with dispatch.
You add the checks to a file called
CHECKS in the app's root directory. For example, if your app is called
web, you'll need to create the file
Eliminate The Dead Zone
Besides a plugin, there are also some changes to the main Dokku script. The reason for doing both: you can run a different deploy check plugin if you don't like mine.
The changes to Dokku work as follows:
- Create a new container, deploy the app, start the web server (not much changed here)
- Run the
check-deployhooks, which runs through the
- Run the
post-deployhooks, which update the Nginx configuration and route traffic to the new server
- Now get rid of the old container
If the checks fail, traffic is still going to the old container, so we just delete the new container and fail the push.
And that's all there is to it: Dokku with zero downtime deploy.
If you want to use our fork of Dokku, please note that we also changed the behavior for HTTPS traffic.
If you have an SSL certificate, it will accept HTTPS requests, but also allow HTTP requests. Dokku's current behavior is to redirect all HTTP requests to HTTPS.
IE's implementation of CORS (XDR), for reasons that defy logic or consistency with real web browsers, won't access HTTPS resources from pages loaded over HTTP.
git clone email@example.com:broadly/dokku.git sudo make install
Add some checks to the file
Worry less, sleep better, look younger.