The full stack, Flux-ready
Prometheus, Alertmanager, Grafana, Loki with fluent-bit, and blackbox probes for your sites.
rttf · recipe pack · version 1.1.0
A complete monitoring stack for a small k3s cluster. It emails you when a site goes down, fits beside your apps instead of eating their RAM, and deploys with one Git push.
The problem
Install kube-prometheus-stack with its defaults and you get hundreds of thousands of series and rules evaluated every few seconds. And when a site goes down, it says nothing. Your first alert is a friend asking "is your site down?"
What you get
Flux manifests, a small dashboard generator and a runbook. Copy the folder into your Flux repository, fill in one settings file, push.
Prometheus, Alertmanager, Grafana, Loki with fluent-bit, and blackbox probes for your sites.
About 1.5 GiB of RAM and 40 millicores of CPU on a 3-node cluster. The README explains every trim, so you can undo any of it.
Latency, traffic, errors and saturation in 19 panels, plus a generator for cloudflared, Garage and nginx panels.
Site down, target down, node not ready, crash looping, disk or volume filling up, CPU steal.
Deploy, verify, update and tear down, with every setting in one table.
3 days of metrics and 7 of logs by default; each is one setting. The README shows the disk it costs and how to measure yours.
Made for AI coding agents
The recipe ships with an AGENTS.md and a CLAUDE.md written as an operating procedure: what to ask you, what to check first, the exact steps, how to verify, and where to stop for your approval.
# following recipes/observability/AGENTS.md
? Where should alerts go, and which sites should I probe?
✓ preflight: nodes Ready, Flux healthy, local-path present
✓ settings filled in, secrets created (values not shown)
? Ready to commit and push to your Flux repo. Proceed? [y/N]
✓ Kustomizations Ready, targets up, test alert emailed
Illustration of the steps in AGENTS.md, not a recorded session.
Tested, not promised
On x86_64 and again on arm64. Every check below passed, and each run started from VM creation.
Is it for you?
Yes, if you run k3s on one to a few small nodes, host real sites on it, and want to know when something breaks.
No, if you want high availability on one or two nodes: by default everything runs once, on one node's disk, and if that node dies your monitoring dies with it. With three nodes and an S3 bucket, the included HA option keeps alerting, metrics and logs going when a node is lost.
local-path storage classkubectl
Questions
It runs on 64-bit ARM: it was tested from zero on an arm64 VM, where you set one documented value for the operator image. It was not tested on a Pi itself, and 32-bit ARM is not supported.
Yes. Retention is one setting for metrics and one for logs. Longer retention mainly needs disk: about 65 MiB a day for metrics and 10 MiB a day for logs on a 3-node home cluster. The README shows how to measure yours and how to cap the size.
Not by default: the 1.5 GiB setup runs everything once. Since version 1.1.0 the download includes an HA option for three nodes: two Prometheus, two clustered Alertmanagers and three Lokis with the logs in S3 (the S3 recipe works), plus two Grafanas on Postgres if you want them. Tested from zero with a node stopped mid-run: the alert about it was emailed, one alert through two Prometheus and two Alertmanagers arrived once, and no log line was lost. It costs about 0.7 GiB more on a 3-node home cluster.
No. The README is a complete runbook for doing it by hand. The AGENTS.md is there if you would rather have your coding agent follow the same steps, asking you before each change.
A zip with Flux manifests, a settings example, a dashboard generator in Python, a README runbook, a VERSIONS.md listing what was tested, and the agent files. Nothing in it needs access to anyone else's infrastructure.
One folder, one settings file, one Git push. Tested from zero on x86_64 and arm64.
Get the recipe €24