Case study · Self-healing infrastructure
The pager still goes off at 06:58. The pull request is open by 07:02.
A Kubernetes estate pages a person when a workload breaks. This one also hires an agent. It reads the cluster, finds the commit at fault, and opens one small pull request it is unable to merge.
78
incidents handled by an agent
Fifteen days, one estate, one runbook, no rota.
4m 27s
from alert to open pull request
The incident below. The second took 4m 08s.
7.5 min
median incident, start to verdict
The longest ran 50 minutes.
0
cluster credentials the agent holds
No kubectl, no kubeconfig, no write path of its own.
Counted on one production estate over 11 to 25 August 2026.
Some breakage reconciliation cannot fix.
GitOps makes the cluster match the repository, and keeps doing it when the repository is wrong.
Drift
Green
The cluster is exactly what the repository says. Reconciliation has nothing left to do, and reports success.
Health
Red
The workload crashes anyway, because what the repository says is false: a dropped variable, a lost secret reference, a bad image pin. No apply fixes that. Somebody has to read the evidence and change the source.
That gap is where the pager lives, and reading evidence to write a small diff is exactly what a coding agent is good at. What was missing was never the diagnosis. It was a machine, a checkout, a push credential, and a human still allowed to say no.
Nine steps, one of them a person.
Everything but the agent was already in the estate. One webhook, one API call.
-
1
Prometheus notices
The estate's existing alert rules. Prometheus knows nothing about the agent.
KubePodCrashLooping · PodOOMKilled · CPUThrottlingHigh · PodRestartChurn -
2
Alertmanager forks the page
One copy to the operator's phone, as before. One to a webhook. The human is added to, not replaced.
continue: true -
3
A webhook hires an agent
Two hundred lines of Node, posting an agent id, a vault id and the alert text. It needs no cluster access.
POST /api/conversations -
4
The agent reads the estate
A read-only estate service answers with node health, drift from source, and controller conditions. Every other method returns 405.
Authorization: Bearer … (GET only) -
5
It finds the wrong fact in git
A bad cluster is a bad commit. The agent reads the history and names the one at fault before editing.
gh api · git log -S -
6
It opens one minimal pull request
Source and regenerated manifests in one commit, from an identity with push rights and nothing else. If no change to the repository can fix the alert, it opens nothing.
symptom · root cause · why this diff is the whole fix -
7
A human approves
The one gate. GitHub blocks self-approval, and the branch needs a review from another account.
422 on self-approval · 405 on self-merge -
8
Flux applies the merge
Git was always the apply path. There is no apply button to trust it with.
reconcile on webhook -
9
The agent verifies and reports
It watches the node come back, checks nothing else regressed, posts the summary, and stops.
poll until healthy
The gate is a mechanism, not a promise.
An agent that opens infrastructure pull requests is only as safe as the least it can do. Nothing rests on a good choice, a prompt holding, or a model behaving. Each refusal below is somebody else's 405.
It can
Read the whole estate graph, including controller conditions.
It cannot
Change anything. The token is refused on any method but GET.
It can
Clone the infrastructure repository and push a branch.
It cannot
Push to the default branch. It is protected, and needs a pull request.
It can
Open a pull request and answer its review comments.
It cannot
Approve or merge it. GitHub blocks self-approval, and the branch needs another account's review.
It can
Name a secret, so the fix restores the reference.
It cannot
Read a secret value. The material never leaves the cluster.
It can
Decide the repository cannot fix this, and stop.
It cannot
Reach past the repository and fix it anyway. The sandbox has no kubectl and no kubeconfig.
One morning, end to end.
25 August 2026, UTC. Nobody was awake for the first half.
-
06:58:15
A sidecar is killed for memory
PodOOMKilled on a container that syncs a checkout. The phone buzzes. So does a webhook, which hires the agent.
-
06:59:34
A second alert, same container
CPUThrottlingHigh, at 94.44%. Its own incident, its own agent.
-
07:02:42
The first pull request opens
Two files, nine lines added and four removed. The agent traced the kill to a code path the limit never covered, and to the commit that introduced it.
-
07:03:42
The second pull request opens
Six lines added, two removed. The same root cause seen from the other side, and the second agent said so rather than diagnose it again.
-
08:50:27
A human wakes up and reads two pull requests
The throttling fix is merged. Flux reconciles on the webhook, and the agent watches the container come back.
-
23:46:01
The memory fix follows
Nothing was on fire, so it waited for a proper read. That is the gate working.
Both modes callinstall_members()(make install→npm ciacross every member) whenever a fetched commit touches apackage-lock.json. Therepo-syncsidecar's limit was sized only for its cheap steady-state git-fetch loop and never accounted for this shared, memory-heavy reinstall path — a gap latent since the container was authored and invisible until a commit actually moved a lockfile.
What Managoat was for.
The alerting, the repository and the reconciler were already there. What was missing was somewhere for an agent to live between alerts.
Agent
Runtime, model, runbook and tools, written once as a file in a public repository. The runbook is the product: diagnose, fix by pull request, wait for the human, verify, report.
Environment
The machine the agent wakes up on, with the repository, the CLI and the read-only estate service's address. Described once, rebuilt per incident, never patched by hand.
Vault
The bot identity, kept apart from the environment. Swap it and the agent acts as a different GitHub account. The token is an environment variable, never in the prompt, the context or the transcript.
Conversation
One incident, one sandbox, one transcript, addressed by its alert. It survives an hour-long diagnosis, and a human opens it at 08:50 to see 07:02.
Nobody provisions a machine for an incident, or pays for one between them. A sandbox lasts as long as the incident. Read about the four primitives.
Point your own alerts at an agent.
Whatever pages you today can post a webhook. That is the whole handoff.
// Alertmanager POSTs here.
// This is the entire integration.
await fetch(`${FOUNTAIN_URL}/api/conversations`, {
method: "POST",
headers: {
Authorization: `Bearer ${FOUNTAIN_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
agent_id: agentId, // estate-medic
vault_id: vaultId, // the identity it acts as
prompt: [
`The estate is alerting.`,
alertLines,
`Diagnose it. If a change to the repo can`,
`fix it, open one minimal PR, wait for the`,
`human merge, verify healthy, and report.`,
`If nothing in the repo can fix it, do not`,
`open a PR. Say so and stop.`,
].join("\n\n"),
}),
});Write the agent down first. Runbook, tools and model belong in a file you review, not a webhook handler. Apply it once and the dispatcher needs only its id.
Put the identity in a Vault. The token that pushes should be its own account with the least rights that work, so changing identity is a field, not a redeploy.
Make the last step refuse you. Let the agent stop where a human already signs off. A gate it is unable to pass beats one it is asked not to.
Your on-call has a queue of small, obvious fixes.
Each starts with reading the evidence and ends with a diff somebody approves. Hire an agent for the middle.
The estate is a private Kubernetes cluster run by one person, and the numbers are its own. Twelve fix pull requests opened in the window and eight merged, four of them the same rehearsal fault raised on purpose. Most incidents end with no pull request, because the agent decides the repository cannot fix them. The agent's definition is public at jhgaylor/agent-specs.