Backups for a small k3s cluster that prove themselves every week: nightly MariaDB and S3 backups, Postgres to any minute, and restore tests that email you when a backup wouldn't come back. Made for people who host something they'd hate to lose (a WordPress or two, an app on Postgres, files in S3) and have never once restored from their backups. I was one of them for years. Why I made it Before the cluster, my data lived on a few small servers, and one of them ran a nightly backup script from cron. I wrote it, I read it, I saw it in the crontab. I never once opened the folder it wrote to. When I finally did, during the move to Kubernetes, the folder was empty, thanks to one redirect in the cron line. So the new backups got restore tests. Then the first restore test I wrote kept passing when it shouldn't have, which was a whole new kind of humbling. This recipe has both lessons in it: backups that run every night, and a weekly restore test that has to reach the newest timeline in your Postgres archive before it is allowed to pass. What you get Nightly MariaDB dumps. One file per database, 7 days kept. A nightly mirror of your S3 buckets. Changed and deleted files are kept for 14 days. Postgres to any minute. CloudNativePG with weekly base backups and every WAL segment archived to your S3. Weekly restore tests. Each one restores the latest backup into a throwaway instance, compares it with your live data, and fails and alerts if it can't. Restore tools and a runbook. One script each for a MariaDB database, bucket files, or Postgres into a new cluster (latest state or a chosen minute), and one that rolls the whole cluster's state back. The cluster's own state (optional, new in 1.1). A nightly, integrity-checked copy of the k3s datastore: every Deployment and hand-made Secret, on top of your data. For k3s on sqlite or embedded etcd, one server or several (etcd new in 1.2). Eight alerts (optional). A nightly job stopped, a restore test failing or missing, Postgres backup stale or failed, WAL archiving failing. Made for AI coding agents It ships with an AGENTS.md and a CLAUDE.md written as an operating procedure: what to ask you, what to check first, the exact steps, how to verify, and where to stop for your approval. Above all, it never runs a restore, which can overwrite live data, until you've named the target and said yes. It's a very polite agent. How it was tested I deployed it on an empty VM with real data in it (500 MariaDB rows, 1,000 Postgres rows and 2 files), followed the README as written, and then broke things on purpose: ✅ Backups ran, and all three restore tests passed ✅ Drill: a dropped MariaDB database restored (500 rows) ✅ Drill: a deleted file restored ✅ Drill: Postgres recovered into a new cluster at its latest state (1,500 rows) ✅ Drill: Postgres recovered to a chosen minute (1,000 rows; the 500 written after it never happened. Very Back to the Future, minus the DeLorean) ✅ Drill: the whole cluster rolled back to its datastore copy (a deleted object came back; node Ready) ✅ Same result on x86_64 and on arm64 Is it for you? Yes, if you run MariaDB, Postgres or S3 on a small k3s cluster and want to know, every week, that you could get it back. No, if you need Neo4j, Valkey or an offsite copy to a second S3: they're not in this version. And keep the backups off the disk that holds your data, because a backup on the same disk goes down with it. The default volume is for trying it out, and the README shows NFS. What you need k3s v1.35 or newer, Flux v2.10 or newer, and kubectl An S3 with a bucket for the Postgres archive (the Garage recipe provides one; any S3 works) A MariaDB, if you want those dumps A backup volume on another machine Runs on x86_64 and arm64 (64-bit). Not tested on a Raspberry Pi itself. Good to know Postgres backs up through CloudNativePG's Barman Cloud plugin (new in 1.2), so CloudNativePG 1.31 dropping the built-in backup doesn't touch you. Coming from 1.1? The README has an upgrade, tested on a running install: 26 seconds without Postgres writes, and your old backups stay restorable. Pinned versions This recipe: version 1.2.0 k3s v1.37.2 and Flux v2.10.0 CloudNativePG chart 0.29.1 (operator 1.30.1), Barman Cloud plugin v0.15.1, cert-manager v1.21.3, PostgreSQL 18.6 MariaDB client 12.3.3 and rclone 1.76.0