Zero-downtime deploys on a single small server

Zero-downtime deployment is usually described in terms of load balancers and rolling updates, which is unhelpful if you have one server. On one machine the gap has a simpler cause: the old process stops before the new one is ready to answer. Close that and you are done. Three things close it - start the new process on a second port and switch traffic only once it responds, teach the old process to finish the requests it already has instead of dropping them, and write migrations that both versions of the code can live with. None of them needs a second machine, and the first two are about twenty lines of work each.

The third one is where the real difficulty lives, and it is the reason most "zero-downtime" deploys still have a thirty-second hole in them.

Where the gap actually comes from#

Before fixing it, measure it. A deploy on a single server is a sequence with a measurable hole in the middle:

1. The deploy tool tells the old process to stop.
2. The process exits. If it dies instantly, every request in flight is dropped.
3. The port is released.
4. The new process starts, loads config, opens a database connection pool, and binds the port.
5. It begins answering.

Downtime is the distance between 2 and 5, and step 4 is nearly all of it. That number varies wildly and you should know yours:

Time it once by hand: stop the service, start it, and watch when a request first succeeds. If the answer is under a second, honestly, a plain restart at 04:00 is a reasonable engineering decision and the rest of this post is optional. If the answer is eight seconds and you deploy three times a day, it is not.

There is a second, less obvious contributor: the requests that were in progress when the old process exited. Those do not appear in your downtime measurement at all, because the endpoint came back up quickly, but somebody's upload failed and somebody's payment webhook got a connection reset. Dropped in-flight requests are usually the more damaging half.

Start the new process before you stop the old#

The core trick is that the thing listening on the public port should not be your application. Put a reverse proxy in front, run the application on an internal port, and a deploy becomes: start the new copy on a different internal port, wait for it to be healthy, point the proxy at it, then stop the old one. The public port never closes. What a reverse proxy does covers the general case; this is the specific one that pays for it.

With nginx, the switch is a two-line edit and a reload:

upstream app {    server 127.0.0.1:3001;}server {    listen 80;    location / {        proxy_pass http://app;        proxy_http_version 1.1;        proxy_set_header Upgrade $http_upgrade;        proxy_set_header Connection "upgrade";        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;        proxy_read_timeout 60s;    }}

$ nginx -t && nginx -s reload

nginx -s reload is graceful by design: the master process starts new workers with the new config and lets the old workers finish the requests they are handling before exiting. Always run nginx -t first, because a reload with a broken config is refused, and a restart with a broken config is an outage.

Two process managers do this for you and are worth knowing about.

gunicorn can replace itself without dropping the socket. Send the master USR2 and it re-executes with the new code while the old workers keep serving; once the new master is up, send the old one WINCH to retire its workers gracefully, then QUIT. For a pure code change with no config change, plain HUP is usually enough - it restarts the workers one at a time, which on a multi-worker setup means there is always somebody answering. Note that --preload defeats this, because the code is loaded in the master rather than the workers.

PM2 in cluster mode does the same thing for Node: pm2 reload app restarts workers one by one rather than all at once, and if you add wait_ready: true and call process.send("ready") after the server is actually listening, it waits for each worker to say so before killing the next. That is a genuine rolling deploy on a single box. PM2 vs a hosting panel is worth reading before you add it, though, because if the panel is already supervising the process you may be solving a problem you do not have.

Graceful shutdown is about ten lines#

The second half of the trick is the old process behaving properly on the way out. A normal stop sends SIGTERM; a kill does not send anything at all. Handle SIGTERM, stop accepting new connections, finish what you have, then exit - with a hard deadline so a stuck request cannot hold the deploy open forever.

const server = app.listen(process.env.PORT || 3000);let shuttingDown = false;process.on("SIGTERM", () => {    if (shuttingDown) return;    shuttingDown = true;    server.close(() => {        pool.end().then(() => process.exit(0));    });    // Keep-alive sockets will otherwise hold server.close open.    if (server.closeIdleConnections) server.closeIdleConnections();    setTimeout(() => process.exit(1), 15000).unref();});

The closeIdleConnections line matters more than it looks. server.close() stops the listener but waits for existing connections to end, and HTTP keep-alive connections do not end on their own - without closing the idle ones you will hit the 15-second timeout on every single deploy. The method exists on recent Node versions; check yours.

For a WSGI or ASGI app, the equivalent is configuration rather than code: gunicorn --graceful-timeout 30 gives workers half a minute to finish, and uvicorn handles SIGTERM by refusing new connections and draining. Under systemd, make the deadline explicit so the unit does not get SIGKILLed mid-drain:

[Service]ExecStart=/home/app/.venv/bin/gunicorn -c gunicorn.conf.py app:appExecReload=/bin/kill -HUP $MAINPIDKillSignal=SIGTERMTimeoutStopSec=45Restart=on-failureRestartSec=2

Whatever the stack, the pattern is identical and the failure is identical: a process that ignores SIGTERM gets killed after the timeout with work still in its hands. Graceful shutdown and health checks goes through the per-framework details.

Health checks that mean something#

"Wait until it responds" needs a definition of responds, and the common mistake is to check the wrong thing.

Use two endpoints with different jobs:

• Liveness answers "is this process wedged?" It should check nothing but itself and return 200 as long as the event loop is turning. If liveness checks the database, a thirty-second database blip restarts a perfectly healthy application, which turns a small problem into an outage.
• Readiness answers "can this process serve traffic?" It checks the things a request needs: a database ping, the cache, whether migrations have run, whether the warm-up is finished. This is the one the deploy script waits on.

app.get("/livez", (req, res) => res.status(200).send("ok"));app.get("/readyz", async (req, res) => {    if (shuttingDown) return res.status(503).send("draining");    try {        await pool.query("SELECT 1");        res.status(200).send("ready");    } catch {        res.status(503).send("not ready");    }});

Returning 503 from readiness while draining is the small detail that makes everything else work: the proxy stops sending new requests to that copy the moment it starts shutting down, rather than at the moment it finishes. The deploy script then becomes a loop:

$ for i in $(seq 1 60); do>   curl -fsS http://127.0.0.1:3001/readyz && break>   sleep 1> done

Sixty attempts, one second apart, then give up and do not switch. Never switch on a timer.

Migrations are the hard part#

Adding a column is safe. Renaming one is not, because for a few seconds both versions are running: the old code writes to full_name and the new code reads display_name, and whichever request lands on the wrong side gets a null. The fix is to never do a rename as one deploy. Add, deploy, backfill, switch reads, then remove in a later release - three or four boring deploys instead of one exciting one. This is usually called expand and contract, and it is the whole of the discipline.

One more safeguard that costs nothing and prevents the worst version of this. A DDL statement that cannot get its lock will wait, and every query arriving behind it waits too, so a single blocked ALTER TABLE can stall the entire application while appearing to do nothing. Cap it:

SET lock_timeout = '3s';SET statement_timeout = '60s';ALTER TABLE orders ADD COLUMN display_name text;

If it cannot get the lock in three seconds it fails, you retry in a quieter minute, and nobody notices. Backfills go in batches with a commit between them:

UPDATE orders SET display_name = full_nameWHERE display_name IS NULL AND id IN (    SELECT id FROM orders WHERE display_name IS NULL LIMIT 5000);

Migrations without downtime works through the full expand-and-contract sequence for the common cases.

What you cannot make zero-downtime#

The honest section, because half of the people reading this run game servers rather than web apps.

A game server restart is downtime. There is no proxy trick that keeps a Minecraft or Valheim world loaded while the process it lives in is replaced, because the world state is in that process's memory and only one process can own it. What you can do is make the downtime predictable and short: schedule it when nobody is on, warn in chat with a countdown, and make sure the stop is clean so the save completes. Restart schedules that help covers picking the hour.

A proxy network changes the shape of the problem without removing it. With Velocity in front of several Minecraft backends, restarting the minigames server does not disconnect the people on survival, and you can move players to the lobby first so they stay on the network while their server comes back. That is genuinely useful, and it is not zero downtime for the server being restarted. Minecraft Velocity proxy networks has the setup.

Long-lived connections always break. WebSockets, game sockets and server-sent events cannot be handed from the old process to the new one. The answer is client-side: reconnect with exponential backoff and re-subscribe on reconnect, so the interruption is a blip rather than a broken page. If the client cannot reconnect cleanly, no amount of server work will help. WebSockets behind a reverse proxy covers the proxy configuration that keeps them alive the rest of the time.

Anything with a single writer. A migration that rewrites a table, a database engine upgrade, a single-instance queue worker holding a lock. Some of these can be made brief. None of them can be made invisible on one machine.

Restarting the database itself. If the application and the database are on the same small server and you restart the database, the application is down whatever its deploy process looks like.

Doing this on a panel with one container#

On a panel-based host each server is one container with one process tree, and that shapes what is possible.

The two-port pattern still works inside it, because nothing stops you running the new copy on a second internal port - RE:NODE lets you add and remove ports on the Network tab, so a second allocation is available if the switch needs to be visible from outside. Where the proxy sits in front, app and web plans include a reverse-proxy slot: point an A record at the address shown and the certificate is issued and renewed automatically, with the real client address arriving in X-Forwarded-For, which your application needs to be told to trust.

The constraint to plan around is memory. During an overlap you are running two copies of the application inside one memory limit, so peak usage is roughly double. That matters here more than on most hosts, because at the memory limit the kernel stops the container and it restarts clean rather than swapping - which is exactly the hard stop you were trying to avoid. If the app uses 700 MB and the plan is 1 GB, the overlap is what kills you, not the deploy. Either size the plan for the overlap or use a worker-by-worker reload, which only ever has one extra worker in memory. Node memory limits explained covers setting the heap so the runtime fails before the container does.

The deploy mechanism itself is Git: RE:NODE's app lines pull from GitHub through a GitHub App with short-lived tokens, with switches for pull-on-start and deploy-on-push, and deploy-on-push restarts only a server that was already running. That restart is the gap this post is about, so the graceful shutdown handler is the part worth writing first - it is the piece that turns the restart from dropped connections into a pause. Deploy a Node app from GitHub has the setup end to end.

Rolling back without a second outage#

A deploy process that cannot be reversed quickly is not finished. Three rules keep rollback cheap:

1. Keep the previous release on disk. A directory per release with a symlink to current is the classic arrangement, and rolling back is re-pointing the symlink and reloading. Deleting old releases after the fifth one is a one-line cron job.
2. Never roll the database back. Forward-only migrations, written so the previous version of the code still works. If you obeyed the expand-and-contract rule, the old code runs fine against the new schema, and rollback is only the application.
3. Take the backup before, not after. Every time, including the deploys you are sure about. On a panel that is one button or one scheduled task.

And rehearse the whole thing somewhere that is not production. A deploy script has the same property as a backup: until it has been run under the conditions it was written for, it is a hypothesis. Staging and production on one account is where to run it.

FAQ#

Do I need a load balancer for zero-downtime deploys?

No. On a single server the job is done by a reverse proxy in front of two internal ports, or by a process manager that reloads workers one at a time. The load balancer in the usual diagrams is solving a different problem - surviving the loss of a whole machine - which one server cannot do regardless.

How long should a graceful shutdown wait?

Long enough for your slowest normal request, plus a margin: 15 to 45 seconds covers most web applications. Put a hard exit after the deadline so one stuck request cannot hold the deploy open, and make sure the supervisor's own kill timeout is longer than yours, or it will kill the process mid-drain.

Can I get zero downtime for a Minecraft or game server?

Not for the server being restarted. The world lives in that process's memory. You can shorten and schedule the gap, warn players, keep a proxy network so that only one backend is affected at a time, and make sure the stop is clean so nothing is lost. Calling that zero downtime would be dishonest.

What is the single biggest cause of dropped requests during a deploy?

A process that exits immediately on SIGTERM without draining. It is invisible in uptime monitoring because the endpoint comes back quickly, and it shows up as random failed uploads, failed webhooks and 502s in the log for a few seconds after every release.

Do I have to change my database migrations?

If you overlap two versions of the code, yes - that is the entire point of expand and contract. If you accept a short stop instead, you can run destructive migrations normally, which is a perfectly reasonable trade for a small service with a quiet hour. Decide deliberately rather than discovering it during a release.

Is a health check endpoint worth it on a small app?

Yes, and mainly for the deploy rather than for monitoring. It is the difference between switching traffic when the new process is ready and switching it after an arbitrary sleep, which is the most common reason a "zero-downtime" deploy still serves errors for four seconds.