restic forget and prune: retention policies on S3

A restic retention policy is two commands with different jobs and very different costs. restic forget is cheap: it reads your policy, decides which snapshot records to delete, and deletes them - a few small objects. restic prune is expensive: it works out which data no remaining snapshot needs, then deletes whole pack files and rewrites the partly used ones, which on S3 means downloading and uploading real data while holding an exclusive lock on the repository. The sensible arrangement for most servers is forget after every backup, prune once a week with --max-unused left at its 5% default or raised, and both run by systemd timers with --retry-lock so they wait for each other instead of failing. The rest of this post is the detail that makes that arrangement work: what the --keep-* flags actually count, why snapshots survive a policy you thought would remove them, what prune costs, and what to do when the repository says it is locked.

This is the deep end of one topic. Setting up a repository, backing up files and databases, checking and restoring are in restic backups to S3; how much space the history will take is in S3 storage sizing and the 3-2-1 rule.

What forget actually counts#

The --keep-* flags are easy to read and easy to misread. The rule behind all of them is the same: restic sorts snapshots newest first, divides time into buckets of the given size (an hour, a day, a week, a month, a year), and keeps the newest snapshot in each of the n most recent buckets that contain a snapshot.

Three consequences follow, and each one surprises somebody.

• It counts buckets with snapshots, not calendar units. --keep-daily 7 keeps one snapshot from each of the last seven days on which a backup ran. If the server was off for a fortnight, the seven kept days stretch back three weeks. This is deliberate: a backup job that silently stops does not slowly delete your remaining history.
• Only the newest snapshot in a bucket counts. With backups every six hours and --keep-daily 7, you keep the last snapshot of each day, not the first, and the other three from each day are removed unless another rule keeps them.
• Rules are a union. A snapshot is kept if any rule keeps it. The newest snapshot is usually the daily, the weekly and the monthly at once, so --keep-daily 7 --keep-weekly 4 --keep-monthly 6 keeps around fifteen snapshots, not seventeen.

Buckets are based on the snapshot's own timestamp, in the local time zone of the machine running forget. A server in UTC that backs up at 23:30 and a laptop in UTC+4 reading the same repository will draw the day lines in different places. For the same reason, run forget from the machine that runs the backups, not from your workstation.

keep-within and its variants

--keep-within 30d keeps every snapshot newer than the given duration. Durations combine years, months, days and hours: 1y6m, 14d, 2d12h. The trap is the reference point: the duration is measured from the newest snapshot in the group, not from now. If backups stopped three months ago, --keep-within 30d still keeps the thirty days before the last backup - the same protective behaviour as the bucket counts.

restic 0.13 added --keep-within-hourly, --keep-within-daily, --keep-within-weekly, --keep-within-monthly and --keep-within-yearly. They combine both ideas: one snapshot per bucket, for every bucket inside the duration. --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y reads more naturally than counting buckets and behaves better when the backup schedule changes, because a duration does not care how many runs happened inside it.

Choosing a policy#

There is no correct policy, only a trade between how far back you can go and how much you store. Two questions settle it. How long might a problem go unnoticed? A corrupted world file or a deleted table is often discovered days later, so a daily history shorter than a week is optimistic. And how far back would you ever restore? Almost nobody restores a game server from eleven months ago, but somebody might need last quarter's database for an accounting question.

The storage cost of a longer policy is smaller than people expect, because each extra snapshot only adds the chunks that changed since its neighbour. A monthly snapshot that has been unique for half a year costs whatever changed in that month, not a full copy. The sizing post works through the arithmetic; the short version is that going from 7 daily snapshots to 7 daily plus 6 monthly rarely doubles a repository and often adds a third.

The cost that does grow with policy length is time. Every snapshot is metadata that forget, prune and check have to read, and a policy of --keep-hourly 720 on a busy server makes every maintenance run slower for very little benefit. Thin the history as it ages: hourly for a day, daily for weeks, monthly beyond that.

Pinning snapshots with tags

--keep-tag is the escape hatch. Tag a snapshot and no time-based rule will remove it:

$ restic tag --add keep-forever --tag nightly a81d0b37$ restic forget --keep-daily 7 --keep-weekly 4 --keep-tag keep-forever --prune

restic tag also takes --remove and --set. Tagging rewrites the snapshot record, so the snapshot gets a new ID; the old one disappears from restic snapshots. Use this before a major game update, a database migration or anything else you might want to undo next month, and remove the tag when you are sure you will not.

Remember that --keep-tag must appear on every forget run. A one-off command without it, typed in a hurry, removes the pinned snapshots like any other.

Groups: why some snapshots never go away#

forget does not apply your policy to the whole repository at once. It first splits snapshots into groups and applies the policy to each group separately. The default is --group-by host,paths: snapshots from the same hostname with the same list of backed-up paths form one group.

That default is right most of the time - the database dumps and the file backups of one server each keep their own seven days - but it causes the most common retention complaint: snapshots that never expire. Each of these starts a new group, and the old group keeps its last snapshots forever, because they are always the newest in their group:

• You changed the backup paths. Adding /etc to a job that backed up /srv creates a new group. The old /srv-only group keeps its last 7 daily, 4 weekly and 6 monthly snapshots indefinitely.
• The hostname changed. Containers get a new random hostname on every rebuild, and a reinstalled machine may not get the old name back. Every backup from a new hostname is a new group. Fix it at backup time with restic backup --host web-01, which records a fixed name regardless of the real one.
• Paths are written differently. restic stores absolute paths, so /srv/app and /srv/app/ normalise to the same thing, but a symlinked path and its target do not.

You can see it happening in the dry-run output, which prints one block per group:

Applying Policy: keep 7 daily, 4 weekly, 6 monthly snapshotssnapshots for (host [web-01], paths [/srv]):keep 6 snapshots:...snapshots for (host [web-01], paths [/etc /srv]):keep 14 snapshots:...

A group you no longer back up shows up there with its full history kept. Remove it deliberately by selecting it and applying a short policy:

$ restic forget --host web-01 --path /srv --keep-last 1 --dry-run

The alternative is to change the grouping. --group-by host treats all paths from one host as one history, which survives path changes but merges file and database snapshots into one timeline - with --keep-daily 7, only the newest of the two from each day would survive. --group-by host,tags is the cleanest choice when every job has its own tag (files, mysql, postgres), because the tag then defines the history and the paths can change freely. --group-by '' disables grouping altogether, which is almost never what you want on a repository with more than one job.

--host, --path and --tag on forget are filters, not groupings: they restrict which snapshots the command considers at all. restic forget --tag mysql --keep-daily 30 applies a thirty-day policy to database snapshots and leaves every other snapshot untouched, which lets one repository carry different policies per job.

Running forget safely#

Read the dry run the first time and every time you change a policy:

$ restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --dry-run

The output lists, per group, the snapshots to keep with the reason each is kept (daily snapshot, weekly snapshot, last snapshot and so on) and the snapshots to remove. If the remove list contains something you expected to keep, it is nearly always a grouping problem or a bucket boundary in a different time zone, and both show up in the reasons column.

restic 0.16 and later refuse to run a policy that would remove every snapshot in a group; that needs --unsafe-allow-remove-all. forget with no policy at all removes nothing. Both are guards against the shell script that builds the command from an empty variable.

forget also accepts snapshot IDs directly - restic forget a81d0b37 c5e9f210 - which is the tidy way to remove a known-bad snapshot, for example one taken while a database dump was failing, before it pollutes a monthly slot.

What prune does, and the options that control it#

forget only deletes snapshot records. The data stays until prune runs. Prune works in stages:

1. It reads every remaining snapshot and builds the set of chunks still in use.
2. It classifies every pack file: fully used, fully unused, or partly used.
3. Fully unused packs are deleted outright - cheap, one delete request each.
4. Partly used packs are repacked: downloaded, their live chunks written into new packs, the new packs uploaded, and the old ones deleted.
5. The index is rewritten to match.

Step 4 is where the time and traffic go, and it is what the options control:

--max-unused is the important one. It accepts a percentage, an absolute size such as 10G, or unlimited. At the 5% default, prune repacks partly used packs until the remaining waste is under 5% of the repository. At unlimited, it never repacks: it deletes fully unused packs and stops. That trades some wasted space - a pack where one chunk of fifty is still in use is kept whole - for a prune that downloads almost nothing.

--max-repack-size spreads the work out instead. A weekly prune with --max-repack-size 2G never moves more than about two gigabytes in a run, so a large backlog after a policy change is cleared over several weeks rather than in one night that overruns the next backup.

--repack-uncompressed matters only for repositories created before restic 0.14 and upgraded to format version 2 with restic migrate upgrade_repo_v2. Old data stays uncompressed until it is repacked, and this flag does that gradually. On a new repository it does nothing.

Always look at a dry run with -v before changing these settings. It prints how much would be deleted, how much repacked and how much unused space would remain, which is exactly the number you are trading.

Prune cost on S3#

On a local disk prune is just I/O. On S3 every step is a request, and on providers that bill for them the costs fall into four kinds:

• Egress. Repacking downloads pack files. On a provider that charges for outbound traffic, this is the largest cost and the one --max-unused controls.
• Requests. Listing the bucket, reading indexes, deleting packs. Cheap per request, but a repository of a few hundred gigabytes has tens of thousands of objects.
• Upload. New packs from repacking. Usually free inbound, but it takes time on a slow uplink.
• Early deletion. Infrequent-access classes have minimum storage durations - 30 days for Amazon's STANDARD_IA, 90 for Glacier Instant Retrieval - and deleting a pack sooner is billed as if it had stayed.

The last one is a reason to think twice before -o s3.storage-class=STANDARD_IA. Packs in a restic repository are deleted constantly as prune runs, so a cheaper storage class can cost more once early-deletion and retrieval charges are counted. Archive classes that need a restore request before an object can be read, such as Glacier Flexible Retrieval and Deep Archive, do not work for a restic repository at all, because restic expects to read any pack on demand.

The practical settings, by situation:

On storage where traffic is not billed per gigabyte, the cost of prune is time and the exclusive lock rather than money, and the default is fine. Pick the setting once, based on how your provider bills, and write it into the scheduled job rather than remembering it each time.

Lock errors and stale locks#

restic uses lock files, stored as objects under locks/ in the repository, so that two processes cannot damage each other. backup, restore and snapshots take a shared lock; several can run at once. forget and prune take an exclusive lock and need the repository to themselves. A process refreshes its lock every few minutes while it runs.

When an exclusive operation meets another lock, or a backup meets a running prune, it stops with something like this (wording varies a little between versions):

unable to create lock in backend: repository is already locked by PID 41234on web-01 by root (UID 0, GID 0)lock was created at 2026-10-12 04:30:02 (6h12m10s ago)storage ID 8c1d2e4fthe `unlock` command can be used to remove stale locks

Read the age and the host before doing anything. A lock a few minutes old belongs to a process that is still working: wait. A lock hours old from the same host, where ps shows no restic process with that PID, is stale - left behind by a run that was killed, an out-of-memory stop or a reboot.

• `restic unlock` removes stale locks only: locks whose process is gone on this host, or that have not been refreshed for over half an hour. It is safe to run at any time.
• `restic unlock --remove-all` removes every lock, including live ones. Run it only when you are certain no restic process anywhere is using the repository; removing a running prune's lock and starting a backup is how repositories get damaged.
• `--retry-lock 30m` (restic 0.16 and later) on any command makes it wait up to that long for a lock instead of failing at once. This is the real fix for scheduled jobs that occasionally overlap.

Lock age is judged against the clocks of the machines involved. A server whose clock is wrong by an hour can see a fresh lock as stale or a stale one as fresh, so keep NTP running on anything that backs up.

If prune complains about missing packs or an inconsistent index rather than a lock, stop and run restic check. restic repair index (called rebuild-index before 0.16) rebuilds the index from the packs that actually exist, and is the usual first step before prune will run again.

Scheduling with systemd timers#

A systemd timer beats cron for this job: missed runs can be caught up after a reboot, two runs of the same unit never overlap, output goes to the journal with timestamps, and a failure can trigger another unit. You need a machine where you control systemd - your own VPS or a dedicated server - and the general pattern is in systemd services for your apps.

systemd's EnvironmentFile wants plain KEY=value lines, so give it its own file rather than reusing a shell file full of export lines:

AWS_ACCESS_KEY_ID=RNAKEXAMPLE123AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYAWS_DEFAULT_REGION=us-east-1RESTIC_REPOSITORY=s3:https://s3.example.com/my-bucket/restic/web-01RESTIC_PASSWORD_FILE=/etc/restic/web-01.passwordRESTIC_CACHE_DIR=/var/cache/restic

The backup service runs the backup and the cheap forget:

[Unit]Description=restic backup and forgetWants=network-online.targetAfter=network-online.targetOnFailure=restic-alert@%n.service[Service]Type=oneshotEnvironmentFile=/etc/restic/web-01.envNice=10IOSchedulingClass=idleExecStart=/usr/bin/restic backup /srv /etc --host web-01 --tag files \    --one-file-system --retry-lock 30mExecStart=/usr/bin/restic forget --host web-01 --tag files \    --keep-daily 7 --keep-weekly 4 --keep-monthly 6 \    --keep-tag keep-forever --retry-lock 30m

[Unit]Description=Nightly restic backup[Timer]OnCalendar=*-*-* 04:30:00RandomizedDelaySec=15mPersistent=true[Install]WantedBy=timers.target

A oneshot service may have several ExecStart= lines; they run in order and the service fails at the first one that fails, so a failed backup skips the forget. Prune gets its own service and a weekly timer, so a slow prune never delays a backup:

[Unit]Description=restic prune and checkOnFailure=restic-alert@%n.service[Service]Type=oneshotEnvironmentFile=/etc/restic/web-01.envNice=15IOSchedulingClass=idleExecStart=/usr/bin/restic prune --max-unused 10% --retry-lock 2hExecStart=/usr/bin/restic check --read-data-subset=5% --retry-lock 2h

Its timer is the same as the backup's with OnCalendar=Sun *-*-* 06:00:00. Then:

$ systemctl daemon-reload$ systemctl enable --now restic-backup.timer restic-prune.timer$ systemctl list-timers 'restic-*'$ systemctl start restic-backup.service      # run once now, to test$ journalctl -u restic-backup.service -n 50

Persistent=true runs a missed job once at boot if the machine was off at the scheduled time. RandomizedDelaySec spreads several servers backing up to one storage endpoint so they do not all start in the same second. OnFailure= starts a template unit you write yourself - a short script that sends a message to a webhook or an email - with the failing unit's name as its argument. That last part is not optional: a timer that fails quietly every night looks exactly like a timer that works, which is the point monitoring that tells you something keeps returning to.

If you prefer cron, the same commands go in a wrapper script with --retry-lock, and cron expressions explained covers the schedule syntax. Add flock around the script so two runs never overlap, which systemd gives you for free.

Troubleshooting#

The repository is as big as before, after forget. You ran forget without --prune and never ran prune. Or prune ran with --max-unused unlimited and the space is held in partly used packs. A dry run with -v shows how much is unused.

Old snapshots never expire. A stale group - changed paths or a changed hostname. Look for the extra block in forget --dry-run and clear it with a --host/--path filter, then set --host on every backup.

forget removed snapshots it should have kept. A missing --keep-tag on that run, a --group-by that merged two jobs, or a day boundary in a different time zone. The reasons column in the dry run explains each kept snapshot; anything not listed there was removed by design.

Backups fail every Sunday morning. They collide with prune's exclusive lock. Add --retry-lock to the backup or move prune to an hour when nothing else runs.

Prune runs for hours and downloads a lot. A large policy change or the first prune in months. Cap it with --max-repack-size, or raise --max-unused until the backlog is gone.

The cache directory keeps growing. /var/cache/restic holds metadata for every repository this machine has opened. restic cache --cleanup removes caches for repositories not used recently.

Whatever policy you choose, restore something from the oldest snapshot it keeps at least once. That is the only proof the policy keeps what you think, and testing a restore before you need it shows how to make it a routine.

FAQ#

Should I run prune after every backup?

No. forget after every backup is cheap and keeps the snapshot list tidy. prune costs time, traffic and an exclusive lock, and the space it frees between two consecutive nights is tiny. Weekly is plenty for most servers; monthly is fine on a slow link or when egress is billed.

What is the difference between --keep-last and --keep-daily?

--keep-last 7 keeps the seven newest snapshots, whenever they were taken - with hourly backups that is seven hours of history. --keep-daily 7 keeps the newest snapshot from each of the last seven days that have one, which is seven days of history whatever the backup frequency.

Is it safe to run forget while a backup is running?

restic will not let it happen. A backup holds a shared lock and forget needs an exclusive one, so forget either fails with a lock error or, with --retry-lock, waits for the backup to finish. The same applies to prune.

Can different jobs in one repository have different retention?

Yes. Tag each job at backup time and run one forget per tag with its own policy, for example --tag mysql --keep-daily 30 and --tag files --keep-daily 7. The --tag filter makes each run touch only its own snapshots.

Does a cheaper S3 storage class save money with restic?

Rarely. Infrequent-access classes charge for retrieval and for deleting objects before a minimum age, and prune deletes and rewrites packs constantly. Archive classes that need a restore before reading do not work with restic at all. Standard storage is the right default.

How do I keep one snapshot forever?

Tag it - restic tag --add keep-forever <id> - and include --keep-tag keep-forever in every forget command. Tagging changes the snapshot's ID, so look it up again with restic snapshots afterwards.