Case study · Alert to pull request
The pager went off at 06:58. The pull request was open 4 minutes 27 seconds later.
Alertmanager paged the on-call engineer and started an agent at the same time. The agent read the cluster through a GET-only service, traced the failure through git, and opened one focused pull request. A human still had to approve it.
7.5 min
median alert-to-verdict time
The longest investigation ran 50 minutes.
78
alerts investigated by an agent
Fifteen days, one cluster, one runbook.
12
fix pull requests opened
Sixty-six alerts ended with no repository fix to propose.
8
fix pull requests merged
The agent could neither approve nor merge.
Counted in one Kubernetes cluster carrying live production workloads over 11 to 25 August 2026. Numbers reported by the cluster administrator (us).
One incident, by the clock.
25 August 2026, UTC. Both pull requests were waiting when the on-call engineer checked the alert.
-
06:58:15
The repo-sync sidecar runs out of memory
A PodOOMKilled alert fires for the container that keeps the checkout current. Alertmanager sends it to the on-call engineer and the webhook at the same time.
-
06:59:34
The same sidecar starts throttling
CPUThrottlingHigh reports 94.44%. Alertmanager sends that alert through the same route, starting a second investigation.
-
07:02:42
The OOM investigation opens a pull request
The agent traces the memory spike to a package install that the sidecar's limit never accounted for. The pull request changes two files: nine lines added and four removed. It also names the commit that exercised the path.
-
07:03:42
The CPU investigation opens a second pull request
The second run reaches the same root cause and references the first pull request instead of repeating the diagnosis. Its diff adds six lines and removes two.
-
08:50:27
The on-call engineer reviews both pull requests
The CPU fix is merged first. Flux reconciles the change, and the agent watches the sidecar become healthy again.
-
23:46:01
The OOM fix is merged after review
The OOM pull request stays open until that evening. The agent could open it, but it could not approve or merge it.
Both modes callinstall_members()(make install→npm ciacross every member) whenever a fetched commit touches apackage-lock.json. Therepo-syncsidecar's limit was sized only for its cheap steady-state git-fetch loop and never accounted for this shared, memory-heavy reinstall path — a gap latent since the container was authored and invisible until a commit actually moved a lockfile.
GitOps can reconcile the wrong state perfectly.
It makes the cluster match the repository. It keeps doing so when the repository is wrong.
Drift
Green
The cluster is exactly what the repository says. Reconciliation has nothing left to do, and reports success.
Health
Red
The workload crashes anyway, because what the repository says is false: a dropped variable, a lost secret reference, a bad image pin. No apply fixes that. Somebody has to read the evidence and change the source.
That gap is where the pager lives. Reading the evidence and writing a small diff is exactly what a coding agent is good at. The agent still needs a machine, a checkout, a push credential, and a human with the only approval.
Nine steps · one human decision
The pull request waited for the engineer.
The agent opened a proposed fix 4 minutes 27 seconds after the page. The on-call engineer reviewed it 1 hour 47 minutes 45 seconds later.
4m 27s
Alert to proposed fix
1h 47m 45s
Proposed fix to human review
By review time, the starting point was no longer an alert. It was a root-cause explanation, the commit that exposed the fault, and a 13-line diff.
-
Automatic
Detect → page and start agent → read cluster → trace git → open pull request
-
Human
Decide whether to approve and merge
-
Automatic
Flux applies the merge → agent verifies health
The person still owned the consequential step. The agent could propose a change; it could not approve or merge one.
The gate is a mechanism, not a promise.
The agent held zero cluster credentials and could not approve or merge. Nothing rested on a good choice, a prompt holding, or a model behaving. Every limit below was enforced outside the model.
It can
Read cluster state, including controller conditions.
It cannot
Change anything. The token is refused on any method but GET.
It can
Clone the infrastructure repository and push a branch.
It cannot
Push to the default branch. It is protected, and needs a pull request.
It can
Open a pull request and answer its review comments.
It cannot
Approve or merge it. GitHub blocks self-approval, and the branch needs another account's review.
It can
Name a secret, so the fix restores the reference.
It cannot
Read a secret value. The material never leaves the cluster.
It can
Decide the repository cannot fix this, and stop.
It cannot
Reach past the repository and fix it anyway. The sandbox has no kubectl and no kubeconfig.
What Managoat supplied.
The alerting, repository and reconciler already existed. Managoat supplied a ready machine for each run, reusable agent and credential configuration, and an addressable run the webhook could start and the engineer could reopen.
Agent
Runtime, model, runbook and tools, defined once in a file in a public repository. That reviewed file defines the job: diagnose, fix by pull request, wait for the human, verify, report.
Environment
The machine the agent wakes up on, with the repository, the CLI and a read-only service for cluster state. Described once, rebuilt per incident, never patched by hand.
Fountain Vault
Not a central secret store, a Fountain Vault is a small override layer. Here it supplied the bot identity separately from the environment. The token never entered the prompt, context or transcript.
Conversation
One incident, one sandbox, one transcript, addressed by its alert. The run and transcript stayed available while the human was away.
Managoat builds the sandbox for the run and parks it between prompts without discarding the checkout. Read about the four primitives.
Point your own alerts at an agent.
Send the same alert to a webhook. That is the whole handoff.
// Alertmanager POSTs here.
// This is the entire integration.
await fetch(`${FOUNTAIN_URL}/api/conversations`, {
method: "POST",
headers: {
Authorization: `Bearer ${FOUNTAIN_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
agent_id: agentId, // estate-medic
vault_id: vaultId, // the identity it acts as
prompt: [
`The cluster is alerting.`,
alertLines,
`Diagnose it. If a change to the repo can`,
`fix it, open one minimal PR, wait for the`,
`human merge, verify healthy, and report.`,
`If nothing in the repo can fix it, do not`,
`open a PR. Say so and stop.`,
].join("\n\n"),
}),
});Define the agent in a file. Runbook, tools and model belong in a file you review, not a webhook handler. Apply it once and the dispatcher needs only its id.
Put the identity in a Fountain Vault. The token that pushes should be its own account with the least rights that work, so changing identity is a field, not a redeploy.
Put the last gate outside the agent. Let the agent stop where a human already signs off. A gate it is unable to pass beats one it is asked not to.
Some alerts end in a small repository change.
The work starts with reading the evidence and ends with a diff somebody approves. An agent can do the middle. Ask Jake how this one is set up.
Most alerts end with no pull request, because the agent decides the repository cannot fix them. That share is a question of reach, not of the model. This agent could read one cluster and open pull requests against one repository. Each further system it can read, and each further repository it can open a pull request against, moves another class of alert inside the set it can fix. The agent's definition is public at jhgaylor/agent-specs.