rttf

rttf · recipe pack · version 1.2.0

Backups you have actually restored

Nightly MariaDB and S3 backups, Postgres to any minute, and restore tests that run every week and email you when a backup would not come back.

  • Deployed from zero
  • x86_64 and arm64
  • Ready for AI coding agents
Test output of four recovery drills: a dropped MariaDB database restored with 500 rows, a deleted file restored, Postgres recovered with 1500 rows, and Postgres recovered to a chosen minute with 1000 rows
4/4recovery drills passed
Weeklyautomatic restore tests
Any minutePostgres point in time
8alerts when something stops

The problem

A backup you've never restored isn't a backup. It's a rumour.

Dumps get corrupt, jobs stop silently, and you find out on the day you need them. On a home lab, nobody is on call to notice. Except you, eventually, at the worst possible moment.

The usual backup job

  • Never restored
  • Stops without anyone noticing
  • Restore steps live in your head
  • You find out on the worst day

This recipe

  • Restored every week, automatically
  • An email when a job or a test fails
  • One tool per restore, and a runbook
  • Every restore was run for real

What you get

Backups, restore tests and the way back

Flux manifests for the jobs, restore tools and a runbook. The tests restore into throwaway instances, so they never touch your live data.

Nightly MariaDB dumps

One file per database, kept for 7 days.

Nightly S3 mirror

Every bucket mirrored; changed and deleted files kept for 14 days.

Postgres to any minute

CloudNativePG with weekly base backups and every WAL segment archived to your S3.

Weekly restore tests

Each restores the latest backup into a throwaway instance, compares it with live data, and fails if it cannot.

Restore tools and runbook

One script each: a MariaDB database, bucket files, Postgres into a new cluster, or (optional) the whole cluster's state from a nightly k3s datastore copy.

Eight alerts

A job stopped, a restore test failing or missing, a Postgres backup stale, WAL archiving failing.

Made for AI coding agents

Hand it to your agent. It never restores without your yes.

The recipe ships with an AGENTS.md and a CLAUDE.md written as an operating procedure. A restore can overwrite live data, so the agent must have you name the exact target and confirm.

Illustration of the steps in AGENTS.md, not a recorded session.

Tested, not promised

Deployed on an empty VM with real data, then broken on purpose

500 MariaDB rows, 1,000 Postgres rows and 2 files. The backups ran, the restore tests passed, then five recovery drills, each compared with the source. On x86_64 and again on arm64.

  • Backups ran, and all three restore tests passed
  • Drill: a dropped MariaDB database restored (500 rows)
  • Drill: a deleted file restored
  • Drill: Postgres recovered at its latest state (1,500 rows)
  • Drill: Postgres recovered to a chosen minute (1,000 rows; the 500 written after it never happened)
  • Drill: the whole cluster rolled back to last night's datastore copy (a deleted object came back)
  • Same result on x86_64 and on arm64
  • Upgrade from 1.1 on a running install: 26 s without Postgres writes, then every check again
Test output: Postgres healthy, CronJobs created, data seeded, backups ran, and the MariaDB, Garage and Postgres restore tests passed
The test run, word for word.
File tree of the download: START-HERE.md and recipes/backups with AGENTS.md, README.md, Flux manifests for the jobs and restore tests, and three restore tools
Exactly what is in the zip.

Is it for you?

For data you'd hate to lose

Yes, if you run MariaDB, Postgres or S3 on a small k3s cluster and want to know, every week, that you could get it back.

No, if you need Neo4j, Valkey or an offsite copy to a second S3: not in this version.

What you need

  • k3s v1.35 or newer, Flux v2.10 or newer, and kubectl
  • An S3 bucket for the Postgres archive (the S3 recipe makes one; any S3 works)
  • A MariaDB, if you want those dumps
  • A backup volume on another machine, such as NFS

Pinned versions

  • k3s v1.37.2 · Flux v2.10.0
  • CloudNativePG chart 0.29.1 (operator 1.30.1)
  • Barman Cloud plugin v0.15.1 · cert-manager v1.21.3
  • PostgreSQL 18.6
  • MariaDB client 12.3.3 · rclone 1.76.0

Questions

Before you buy

Where are the backups stored?

On a volume you choose. The default is a volume on one node's disk, for trying it out; the README and the agent both tell you to use NFS or another machine for real, so a dead disk does not take the backups with it.

What does a restore test actually do?

Each week it restores the latest backup into a throwaway instance, compares it with your live data (databases present, rows there, files present), and fails if it cannot. A failing or missing test raises an alert.

Do I need the S3 recipe?

No. Postgres needs an S3 bucket for its archive; the S3 recipe provides one, but any S3 works.

What about the CloudNativePG plugin change?

Done in 1.2. Postgres backs up through CloudNativePG's Barman Cloud plugin, so 1.31 dropping the built-in backup doesn't touch you. Coming from 1.1? The README has an upgrade, tested on a running install: 26 seconds without Postgres writes, and your old backups stay restorable.

Does it run on a Raspberry Pi?

It runs on 64-bit ARM: it was tested from zero on an arm64 VM. It was not tested on a Pi itself, and 32-bit ARM is not supported.

Know your backups work before you need them.

Nightly backups, weekly proof, and a restore you have already rehearsed. Tested from zero on x86_64 and arm64.

Get the recipe €29