Testing a restore before you need it: a 30-minute drill

Everyone has backups. Rather fewer have restores. The difference only becomes visible on the day it matters, which is the worst possible moment to find out that the archive contains the wrong folder, that the database was never in it, or that the one person who knows the procedure is asleep. The fix is a drill: take a backup you have not touched, put it back somewhere harmless, and time yourself doing it.

Half an hour, once a quarter. That is the whole commitment, and it converts a folder full of files into a number you can quote and a procedure somebody else can follow. This post is the drill itself - what to restore, how to verify it genuinely worked, what usually fails, and how to write down the result so the next one takes fifteen minutes instead of thirty. What belongs in the backup in the first place is backups that actually restore; the database-specific commands are in database backups and restores.

What the drill is measuring#

Three numbers come out of a restore test, and only one of them is the one people expect.

• Recovery time. How long from "we need the backup" to "it is verified and people are back". Not the restore button's progress bar - the whole thing, including finding the right archive, remembering the startup variables and checking that the world is the right one.
• Recovery point. How much work the restore threw away. If the backup ran at 04:00 and the incident was at 21:00, that is seventeen hours, and the drill is where you find out whether seventeen hours of your particular server is an inconvenience or a community-ending event.
• Bus factor. How much of the procedure lives only in one person's head. This is the number the drill is really for. A restore that takes twenty minutes when you do it and is impossible for anyone else is not a recovery plan.

Guesses at all three are systematically optimistic. In practice a restore takes two to three times the estimate, because the estimate covers the copying and not the deciding.

The thirty-minute drill#

Pick a quiet hour. You are not touching production, so there is no risk beyond the cost of a scratch server, but you do want to be undisturbed.

1. Choose the backup you would actually use, not the newest one. The most recent archive is the one you are most confident in, which makes it the least useful test. Take the oldest one still inside your retention, because that is the one that tests whether retention works and whether an archive from three weeks ago still matches the current server version.
2. Do not look at the server first. Start from the backup list and nothing else. If you find yourself needing to check something on the live server to proceed, write that down - that is a dependency your plan has on a machine that may not exist during a real incident.
3. Restore onto something that is not production. A second server on the cheapest plan, a development server you already keep, or your own machine. Restoring over the live server to see whether it works is not a test, it is an outage with extra steps.
4. Start a stopwatch and follow the written procedure. If there is no written procedure, the drill for this quarter is to write one while you do it, in the order you actually did things.
5. Bring it up. Apply the startup variables, allocate the ports, start it, and watch the console rather than the spinner. The log tells you what the status dot does not.
6. Verify, properly. See the next section. "It booted" is not verification; a Minecraft server with a missing world/ folder boots happily and generates a new one.
7. Write down the four timings and stop. Locate, restore, start, verify. Then delete the scratch server, or keep it if you would rather have staging - see staging and production on one account.

On RE:NODE the scratch server for step 3 is genuinely cheap: a game server is created when payment clears, usually inside a minute, restoring a backup is a button rather than a ticket, and backups are downloadable if you would rather do the whole thing on your own machine. That is the part that makes the drill repeatable, and a drill you will not repeat is a one-off that ages out in a month.

Verifying that the restore is real#

The failure you are hunting is the restore that appears to succeed. Every workload has a specific thing to look at, and the general rule is: check the newest data, not the oldest. Anything from three months ago is in every copy you own. The last change before the backup ran is the one that proves you got the right archive.

The cheapest way to make verification unambiguous is a canary: leave a dated marker in the data before the backup runs. A sign in-game with the date on it, a row in a backup_marker table, a file called 2026-09-21.txt in the server root. After a restore, the marker either says the date you expected or it does not, and you know in five seconds which archive you are looking at. It removes the entire class of mistake where somebody restores the right backup from the wrong server.

The six things that usually fail#

Across the restores people ask for help with, the same six causes come round.

• The world is there and the configuration is not. The archive holds world/ but not server.properties, bukkit.yml or the plugin configs, so the server starts as a stranger: default MOTD, wrong difficulty, whitelist off, nobody an operator.
• Plugin or mod data lives outside the folder somebody assumed was everything. Permissions in one place, economy in another, a claims database in a third. LuckPerms in database mode is the classic: the plugin restores, its data does not.
• The database was never in the backup at all, because it is a different service with its own backup mechanism. This is the single biggest cause on app and web workloads.
• Versions do not match. The archive was taken under one game build, mod set or runtime, and the server you restored onto is running another. If a mod update is in the mix, what to do when a mod update breaks is the companion piece.
• The settings that are not files are missing. Startup variables, port allocations, schedules and subuser grants live in the panel, not in the archive. A restored server has the right world and the wrong start command until you put them back.
• Access. The person doing the restore does not have the permission, the credential or the account. Grant a second person the backup and file permissions ahead of time - subusers and least privilege covers doing that without handing over billing.

Notice that four of the six are not about the archive being corrupt. Corrupt archives are rare. Incomplete ones are the norm, and the drill is how you find out which category yours is in.

Write the runbook while you do it#

The output of the drill is not a warm feeling, it is a file. Keep it somewhere that is not the server - a note, a pinned message, a text file in the repository - and update it every time the answer changes.

SERVER: survival.example.net  (Paper 1.21.4, Java 21)BACKUPS: panel slots x2, nightly 04:15; weekly download to NAS /backups/mcDATABASE: Postgres on db plan, dump written to /home/container/dumps at 04:05RESTORE 1. Stop the server. Download the current world folder first (evidence). 2. Panel > Backups > pick archive > Restore. ~6 min for 3.2 GB. 3. Startup tab: SERVER_JARFILE=paper-1.21.4.jar, MEMORY=6G, port 25565 + 25575. 4. psql: createdb mc_restore; pg_restore --no-owner -d mc_restore dumps/latest.dump 5. Start. Watch console for "Done". Check /seed, ops.json, a LuckPerms group. 6. Canary: sign at spawn shows the backup date.MEASURED 2026-09-21: locate 2m, restore 6m, start 4m, verify 7m. Total 19m.WHO CAN DO THIS: me, and <name> (has Backups + Files on the subuser role).

That is about as long as a runbook for one server ever needs to be. The version numbers, the port numbers and the measured total are the parts that make it worth having; everything else you would have worked out anyway.

How often, and what should trigger an unscheduled one#

Quarterly is the baseline for a game server or a small app. Monthly if the data is money or if the server is the reason a community exists. Beyond that you are into continuous verification, which is a real thing but not a thing two people running a server should build by hand.

Run an extra drill, off schedule, whenever the answer to "what is in the backup?" has changed:

• You added a plugin, mod or feature that stores data somewhere new - especially anything with its own database.
• You changed the backup schedule, the retention, or which folders it covers.
• You upgraded the game, the runtime or a major dependency.
• You moved host, plan or machine. Moving a server without losing players is essentially a restore drill with an audience.
• A backup failed, or produced an archive noticeably smaller than the last one. Size drift is the earliest warning you get, and it only helps if somebody is looking - see monitoring that tells you something.
• Somebody new joined the team who would have to do it. Let them run the drill; that is the bus factor number improving in real time.

And one more, the unglamorous one: after any security incident. A restore in that situation has a different goal - you are trying to get back to a point before the intrusion, not before the crash - and the timeline matters more than the speed. What to do when your server is hacked covers the ordering.

Doing the drill when you have no second server#

You can still test without paying for anything extra, with two caveats about what each method actually proves.

Restore locally. Download the archive, unpack it on your own machine, and start the server there. This proves the archive is complete and readable, and it proves the world loads. It does not prove that the panel restore path works or that the startup variables are right, so treat it as a partial pass and do a full one when you can.

Restore into the same server's folder under a different name. For file-level data you can upload a second copy of the world beside the live one and point a test configuration at it. Fine for Valheim or Minecraft worlds, useless for anything where the data is a database, and it competes for the same CPU and disk as the live server.

Use the staging server you should have anyway. If you already keep a second small server for testing plugin updates, it is also your restore target, and the drill costs nothing but the half hour. This is the arrangement worth aiming for.

What does not count: restoring over production and seeing whether anybody complains. That is a live fire exercise where the fire is real.

An untested backup is a hypothesis. A tested one is a plan, with a number attached.

FAQ#

How long should a restore take?

Whatever your last drill measured. For a 500 MB game server on a panel, five to fifteen minutes end to end is normal. A 40 GB modded server with a database behind it is an evening. The useful answer is your own measured total, because that is the number you can promise to the people waiting.

Which backup should I test?

The oldest one still in retention, restored onto something that is not production. Testing the newest archive tells you the backup ran last night, which you already knew. Testing the oldest tells you whether your retention window is real and whether an archive that age still matches your current server.

Do I have to test every server?

No. Test the one whose loss would hurt most, then the one whose backup is most complicated. A dozen servers with the same configuration share a procedure, so proving it once proves it for all of them, as long as their data really is the same shape.

What if the restore fails?

That is the drill working. Fix the cause before you do anything else, take a fresh backup, and run the drill again on that one. A failed drill on a Tuesday afternoon costs half an hour; the same failure discovered during an incident costs the server.

Can somebody else on my team do the restore?

Only if you have given them the permission and they have done it once. Create a role with the backup and file permissions but no billing access, add them to it, and let them run the next drill. If the answer is currently no, that is the highest-value thing on your list.

Does restoring lose the changes made since the backup?

Yes, and that gap is the point of measuring the recovery point. Before restoring, download the current state so those changes still exist somewhere even though they are not in the restored server - occasionally you can copy one specific thing across afterwards, and you cannot do that if you overwrote it.