RE:NODE

restic restore and check: getting your data back from S3

By RE:NODE Team
16 min read


To get data back out of restic, you find the snapshot with restic snapshots, look inside it with restic ls, and restore into an empty directory with restic restore <id> --target /tmp/restore --include <path>. A single file or a database dump comes out faster with restic dump, and restic mount lets you browse every snapshot as a folder when you do not yet know what you are looking for. restic check --read-data-subset proves, ahead of time, that the data you will need is still readable. None of it works without the repository password, and nobody - not restic, not the storage provider - can give it back to you if it is gone.

Setting up the repository, the backup script and retention is covered in restic backups to S3. This post is the other half: the day you need the data, and the drills that make that day boring. Everything here assumes restic 0.17 or newer; where a flag is newer than that, or older versions behave differently, it says so.

Getting back into the repository#

On the server that made the backups, the environment file from the setup post still works: source it and every command below runs. The interesting case is the one where that server is gone, and you are on a fresh machine with nothing but what you wrote down.

You need exactly three things: the repository location (endpoint, bucket and prefix), an access key pair that can read the bucket, and the repository password. With those, any machine with restic installed can restore.

/root/.restic-env
export AWS_ACCESS_KEY_ID=RNAKEXAMPLE123export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYexport AWS_DEFAULT_REGION=us-east-1export RESTIC_REPOSITORY=s3:https://s3.example.com/my-bucket/restic/web-01export RESTIC_PASSWORD_FILE=/root/.restic-password
bash
$ . /root/.restic-env$ restic cat config$ restic snapshots

restic cat config is the smallest possible test: it reads one object and decrypts it. If it succeeds, the endpoint, the keys and the password are all right, and anything that fails afterwards is a problem with what you are asking for rather than with access. The first command on a new machine is slower than usual because restic builds its local metadata cache in ~/.cache/restic; that is normal and only happens once.

The access keys are the replaceable part. If they were stored only on the dead server, generate a new pair from the storage provider's control panel and carry on - the repository does not care which key read it. On RE:NODE S3 storage, the keys are generated per storage server and shown in the panel, so a lost secret key is a trip to the panel, not a disaster. The password is the irreplaceable part, and it has its own section further down.

If you are restoring to a different machine, set --host explicitly when you mean the old one. A new machine has a different hostname, and filters that default to "this host" will find nothing.

Finding the right snapshot#

Restoring the wrong snapshot is more common than restoring a broken one. Spend the extra minute here.

bash
$ restic snapshots --host web-01 --tag nightly$ restic snapshots --latest 3 --compact$ restic snapshots --path /srv/minecraft

The filters combine: --host, --tag and --path narrow the list, and --latest n shows only the n newest snapshots in each group. --json gives you something to feed a script.

The word latest deserves a warning. Without filters, latest means the newest snapshot in the whole repository, whatever it contains. If your nightly script backs up /srv and then stores a mysqldump through --stdin-from-command, the newest snapshot is the database dump, and restic restore latest --target /tmp/restore gives you one SQL file instead of your server. Always pin latest to something:

bash
$ restic restore latest --tag nightly --host web-01 --target /tmp/restore

Once you have a candidate, look inside before restoring anything:

bash
$ restic ls a81d0b37 /srv/minecraft/world$ restic ls --long latest --tag nightly /etc/nginx$ restic find --host web-01 'server.properties'$ restic diff 4f2a1c9e a81d0b37

Each of the three does one job:

  • restic ls lists the files of one snapshot, optionally limited to a directory. --long adds size, mode and modification time, which is what tells you whether a file is the one from before or after the bad change.
  • restic find searches every snapshot for a name or a pattern and tells you which snapshots contain it. It is the fastest way to answer "when did this file last exist?" after somebody deleted it three weeks ago.
  • restic diff compares two snapshots and lists added (+), removed (-) and modified (M) files. When you know something broke between Tuesday and Wednesday, diffing those two snapshots usually shows you what.
CommandAnswers
restic snapshotsWhich backups exist, from where, when
restic ls <id> <dir>What is inside one snapshot
restic find <pattern>Which snapshots hold a given file
restic diff <a> <b>What changed between two backups

restic restore in detail#

restore is the workhorse. It reads the snapshot, downloads only the packs it needs, and writes files with their permissions, timestamps and, when run as root, their original ownership. Run as an ordinary user, the restored files belong to that user.

bash
# Whole snapshot into an empty directory$ restic restore a81d0b37 --target /tmp/restore# One directory$ restic restore a81d0b37 --target /tmp/restore --include /srv/minecraft/world# Everything except the noise$ restic restore a81d0b37 --target /tmp/restore \    --exclude /srv/minecraft/logs --exclude '*.tmp'# Contents of one directory, without the leading path$ restic restore a81d0b37:/srv/minecraft --target /tmp/mc

Paths are kept as they were. Restoring /srv/minecraft/world into /tmp/restore produces /tmp/restore/srv/minecraft/world, not /tmp/restore/world - which surprises people looking for their files in the top of the target. The snapshot:subfolder form in the last example strips that prefix: it restores the contents of /srv/minecraft directly into /tmp/mc.

The include and exclude options mirror the backup side. --include and --exclude take patterns, --iinclude and --iexclude are case-insensitive, and from restic 0.17 --include-file and --exclude-file read patterns from a file, one per line. Without any --include, everything is restored.

Options added in restic 0.17

Version 0.17 made restore considerably more useful for restoring into a directory that already has data in it:

OptionWhat it does
--dry-runShows what would be restored, writes nothing
--overwritealways (default), if-changed, if-newer or never
--deleteRemoves files in the target that are not in the snapshot
--verifyReads the restored files back and checks their content

--overwrite if-changed with --delete turns restore into something close to rsync from a snapshot: unchanged files are left alone, changed ones are rewritten, and extra files are removed, so the target ends up an exact copy of the snapshot. That is the fastest way to roll back a large directory where only a little changed, and also the most dangerous command in this post. Always run it with --dry-run -vv first and read the list of files it would delete.

--verify costs a second read of everything restored, from local disk rather than the network, and is cheap insurance on a restore you are about to put into production.

How long it takes

Restore speed is mostly download speed. restic fetches packs in parallel over several connections (-o s3.connections=5 is the default for S3; raising it can help on a fast link and a fast endpoint), decrypts and decompresses them, and writes the files. A repository on a remote endpoint over a home connection is limited by that connection, not by restic. Measure it once on real data, as the drill section below describes, rather than assuming.

restic dump: one file, one database, one archive#

When you want a single file, a full restore is overkill. dump writes the content of one file from a snapshot to standard output:

bash
$ restic dump a81d0b37 /etc/nginx/nginx.conf > /tmp/nginx.conf.old$ diff /tmp/nginx.conf.old /etc/nginx/nginx.conf

This is the natural way to restore a database dump stored with --stdin-from-command, because the dump never needs to touch the disk:

bash
$ restic snapshots --tag mysql --latest 1$ restic dump --tag mysql latest /app.sql | mysql app_restore$ restic dump --tag postgres latest /app.sql | psql -d app_restore

The path is the --stdin-filename you gave at backup time, shown in the snapshot's path column. Restore into a new, empty database first (app_restore above), compare it with what you expected, and only then point the application at it or swap the names. The engine-specific restore steps - users, grants, --no-owner for Postgres custom-format dumps - are in database backups and restores and mysqldump backup and restore.

Given a directory instead of a file, dump produces a tar archive of it, or a zip with --archive zip:

bash
$ restic dump latest --tag nightly /srv/minecraft/world > world.tar$ restic dump --archive zip latest --tag nightly /srv/minecraft/world > world.zip

That is handy when the data has to end up somewhere restic is not installed - for instance, an archive you upload through a game panel's file manager or over SFTP, where the server itself runs in a container and has no restic of its own.

restic mount: browsing every snapshot#

restic mount presents the repository as a read-only filesystem through FUSE:

bash
$ mkdir -p /mnt/restic$ restic mount /mnt/resticNow serving the repository at /mnt/resticUse another terminal or tool to browse the contents of this folder.When finished, quit with Ctrl-c here or umount the mountpoint.

In a second terminal, the repository looks like this:

code
/mnt/restic/  ids/        one directory per snapshot ID  snapshots/  one directory per snapshot time, plus 'latest'  hosts/      snapshots grouped by host name  tags/       snapshots grouped by tag

You can cd into snapshots/2026-10-07T04:30:09+02:00/srv/minecraft, run ls, grep, less or diff -r against the live tree, and copy single files out with cp. It is the best tool when you do not yet know what you need - "the config from before somebody edited it, sometime last week" - because ordinary tools work on every snapshot at once.

The limits:

  • It needs FUSE: Linux, macOS with macFUSE, or FreeBSD. It does not work on Windows, and it usually does not work inside a container, which seldom has /dev/fuse or the privilege to use it. Install fuse3 (or fuse) on Debian and Ubuntu if the mount fails with a missing fusermount.
  • Only the user who ran the mount can see it, unless you add --allow-other and the system allows it in /etc/fuse.conf.
  • It is slow for bulk copies. Every read fetches and decrypts packs on demand; cp -r of a 20 GB directory from a mount is much slower than restic restore of the same directory, which plans its downloads. Browse with mount, restore with restore.

Stop it with Ctrl-C in the terminal that started it, or umount /mnt/restic (fusermount -u on some systems).

Checking the repository before you need it#

restic check is how you find out that the repository is damaged on a Tuesday afternoon instead of during an outage. It has two levels.

bash
$ restic check$ restic check --read-data-subset=5%$ restic check --read-data-subset=3/12$ restic check --read-data

Plain check verifies structure: that the index is consistent, that every snapshot's tree is complete and that every chunk referenced exists in some pack. It reads metadata only and finishes quickly. It does not open the packs, so it cannot notice a pack whose bytes have rotted.

--read-data downloads every pack and verifies every chunk against its hash. That is the only full proof, and on a large repository it is a full download, with the time and egress that implies. --read-data-subset reads part of it:

  • --read-data-subset=5% picks a random 5% of packs each run. Over many runs it covers most of the repository, but with no guarantee about any particular pack.
  • --read-data-subset=n/t splits the packs into t groups and reads group n. Running 1/12 in January, 2/12 in February and so on reads the whole repository once a year, every pack exactly once.

A monthly cron job with the month number as n is a sensible default for repositories too large to read in full:

bash
$ restic check --read-data-subset="$(date +%-m)/12"

A clean run ends with no errors were found. Anything else is worth stopping for. check exits non-zero on errors, so a scheduled run can alert you through the same path as a failed backup - see monitoring that tells you something.

When check finds errors

Do not run prune or forget on a repository that fails check; you will make the damage harder to reason about. Work through it in order:

  1. Run restic check again with a fresh cache (--no-cache, or move ~/.cache/restic aside). A corrupted local cache can produce errors the repository does not have.
  2. If the index is the problem, restic repair index (restic 0.16 and later; older versions call it rebuild-index) rebuilds it from the packs actually present.
  3. If packs are missing or damaged, run a new backup from the source while the source still exists. Chunks that are still on disk get uploaded again, which heals every snapshot that shares them.
  4. Only then consider restic repair snapshots, which rewrites snapshots to drop references to data that is truly gone. It loses those files from history; it does not bring them back.

Repository damage on healthy storage is rare. When it happens, the cause is usually outside restic: storage that silently lost objects, a lifecycle rule on the bucket that deleted old files, or two tools writing to the same prefix.

The lost password reality#

Every object in a restic repository is encrypted with a master key, and the master key is stored in the keys/ directory encrypted with your password. There is no backdoor, no reset link and no support ticket that helps. If every password for a repository is gone, the data is unrecoverable, by you, by the storage provider and by restic's developers. That is the design working as intended.

What you can do is make sure that never happens:

bash
$ restic key list$ restic key add          # asks for a new password$ restic key passwd       # changes the password of the current key$ restic key remove 3e1b0c9d

Three habits keep it that way:

  • Have more than one key. restic key add adds a second password that opens the same repository. Keep one on the server in /root/.restic-password and another in a password manager or a sealed envelope that does not live with the server. Each key opens everything; they are equals.
  • Test the copy, not just the original. A password in a password manager with a typo in it is not a backup of the password. Open the repository with the stored copy once: restic -r ... --password-file /tmp/pw-from-vault snapshots.
  • Know what changing a password does not do. key passwd and key remove change who can unlock the master key; they do not re-encrypt the data. Anyone who once had a working password and a copy of the key file can still decrypt the repository. If a password leaks, the only real fix is a new repository: restic init a fresh one and restic copy the snapshots you want into it.

The access keys and the password are separate secrets. A leaked S3 secret key lets someone delete or download your encrypted packs, but not read them; rotate it in the provider's panel. A leaked repository password without the S3 keys gets them nothing until they also reach the bucket. Store the two apart.

A restic restore drill#

A backup you have never restored is a hypothesis. The general method - what to measure, what usually fails, how to write the runbook - is in testing a restore before you need it. For restic specifically, the drill is short enough to run every quarter:

credentials onlysnapshots, restorePassword managerpassword, endpointScratch machinerestic installedS3 bucketrestic repository/tmp/restorescratch directoryVerificationcanary, diff, app starts
Restore drill from a clean machine
  1. Start from a machine that is not the server. A throwaway VM, a second small server, your laptop. Install restic. The point is to prove you can restore with nothing from the original machine.
  2. Use only what is stored outside. Endpoint, bucket and prefix, an access key, the password - from the password manager, not copied off the server. If you need to look at the server for anything, write that down: it is a dependency your recovery has on the thing that failed.
  3. Pick an old snapshot. Not latest: the oldest one still inside retention tests that retention worked and that old data is still intact.
  4. Restore with a timer running. time restic restore <id> --target /tmp/restore --verify. Write down the size and the time. That number divided into your largest server is your real recovery time over that connection.
  5. Restore the database too. restic dump ... | mysql app_restore or the equivalent, then count rows in the tables that matter and find the newest timestamp.
  6. Verify the content. Look for a canary - a file or a database row with the date of the backup - and compare a few key files against their source with diff. Start the application against the restored data if you can.
  7. Run a data check while you are there. restic check --read-data-subset=1/12 adds a twelfth of the repository to the evidence.
  8. Write down what was missing. A config file outside the backed-up paths, a startup flag, a secret that lived only in an environment variable. Fix the backup, not just the runbook.

For game servers, the same drill confirms the world restores and loads; the save-flushing rules that make the world consistent in the first place are in game server backups to S3.

Errors you will meet during a restore#

Message (excerpt)Cause
wrong password or no key foundWrong password, or the right one for another repository
Is there a repository at the following location?Wrong bucket, prefix or endpoint
repository is already lockedA running or crashed backup or prune holds a lock
no matching ID foundSnapshot ID typo, or a filter that hides it
SignatureDoesNotMatchWrong secret key, or region or clock mismatch

Some notes on each:

  • Wrong password. Check that RESTIC_PASSWORD_FILE points to the file you think, and that the file holds the password and nothing else - no leading space, no second line. From 0.17 restic exits with code 12 on a wrong password, which scripts can test for.
  • No repository. The prefix is part of the address. s3:https://s3.example.com/my-bucket/restic/web-01 and .../my-bucket/restic/web01 are different repositories, and one of them does not exist. List the bucket with any S3 client to see what is actually there. If the endpoint needs path-style addressing, add -o s3.bucket-lookup=path, as S3 path-style vs virtual-hosted explains.
  • Locked. restore, ls, dump and mount take a non-exclusive lock and run alongside backups, but not alongside prune. If no other process is running, restic unlock removes stale locks safely. For a restore that must not wait, --no-lock skips locking on read-only commands, at the risk of reading while a prune deletes data under you.
  • Signature errors. The secret key is wrong, or the machine's clock is far enough off that the request signature is rejected. Fix the time with NTP before chasing anything else.

FAQ#

Can I restore a restic backup without restic?

No. The repository is encrypted, compressed and split into chunks; nothing else can read it. restic is a single static binary for Linux, macOS, Windows and the BSDs, so keep the download link in your runbook. If you need a plain archive for something that cannot run restic, produce it with restic dump on a machine that can.

Can I restore on Windows from a Linux backup?

Yes, restic restore works across operating systems. Unix ownership and some permission bits will not map onto NTFS, and mount is not available on Windows, but file contents come back intact.

How do I restore just one deleted file?

Find the last snapshot that had it with restic find --host web-01 'filename', then restic dump <id> /full/path/to/file > file. That writes only that file and does not touch the rest of the directory.

Does restore overwrite newer files in the target?

By default, yes: --overwrite always replaces existing files. From restic 0.17 you can choose if-newer to leave files that are newer than the snapshot, or never to touch only missing files. Restoring into an empty directory avoids the question entirely.

How often should I run check with read-data?

Plain check weekly, and a rotating --read-data-subset=n/12 monthly, reads every pack once a year without a full download each time. For a small repository, a full --read-data once a month is cheap enough to just do.

Is a restic repository on one S3 server enough?

As one copy, yes. RE:NODE S3 storage, for example, keeps one copy on NVMe in one location, not replicated - a good off-machine copy, not your only one. restic copy to a second repository elsewhere covers the rest, as S3 storage sizing and the 3-2-1 rule works through.