Manual contents

Architecture

This page is the system view. It says what processes exist, what each one talks to, and what breaks when a dependency is down. For the domain objects, which are agents, environments, vaults and conversations, read The four primitives.


The runtime shape

Managoat is one OTP release, fountain_server. Everything below runs in a single BEAM instance. There is no separate worker deployment and there is no sidecar. It runs as one node: read One node.

Piece Job
Phoenix endpoint The one public listener. It serves the operator console, the REST API and the SSE streams, on one port.
Conversation server One process for each active conversation. It owns the sandbox. It provisions the sandbox, spawns a turn in it, streams the output back, and enforces the lifecycle bounds. It holds the decrypted account key in memory while it lives. Managoat registers it on the node, so exactly one exists for each conversation.
Rehydrator Runs once at boot. It finds the conversations that were live before the restart and starts their servers again. Those servers reattach to the sprite, which still runs, so a deploy kills no work.
Oban Background jobs, such as the account emails. It also runs the cron schedule below.
Metrics listener A second, private HTTP listener on METRICS_PORT, which is 9568 in production and off elsewhere. It serves /metrics and /health. It is deliberately apart from the public endpoint, so that no ingress rule can expose it by accident.

Here is the scheduled work. All times are UTC.

Schedule Job What it does
Each hour at :07 The sandbox reaper. Reconciles the sandbox rows against sprites.dev. It frees a row stuck mid-provision, expires an abandoned sandbox, destroys a sprite whose row is already terminal, and reports an untracked sprite.
04:23 daily The retention pruner. Deletes a row past its retention. Log events go after 90 days, audit events after 365, and usage events after 400. A revoked API key goes after 30.
04:29 daily The ChatGPT keepalive. Refreshes each linked ChatGPT subscription that nobody has refreshed for 6 days, one at a time. With none linked, it does nothing.

One node

Managoat runs as exactly one node. More than one replica is not supported.

  • The conversation registry and the machine registry are local to the node. Each conversation server and each sandbox owner registers there, and a lookup that finds nothing means that nothing is running.
  • PubSub fans a conversation's events out to the viewers connected to that node.
  • Postgres still serializes the work on a sandbox. A per-sandbox advisory lock and row locks order each operation, as they did before.
  • The scheduled jobs run in Oban, as before. The rehydration sweep runs once at boot.

A deploy stops the old node before it starts the new one. On Kubernetes that is the Recreate strategy. The API is unavailable for a short gap, and at boot the rehydrator starts a server again for each conversation that had one. A rolling deploy would run two nodes at the same time. They would be two schedulers over the same sandboxes, and a viewer connected to the other node would see no stream.


Where state lives

Postgres is the only durable store. Users, agent and environment configs, encrypted secrets, conversations, turns, log output, the audit trail, the job queue and uploaded images are all rows. A sandbox is disposable by design, and Managoat persists the output as it streams. So to destroy a sandbox loses nothing that mattered.

Two things are deliberately not durable.

  • The in-memory tables on the node. Those are the rate-limit counters and the log-redaction registry. Managoat rebuilds them empty at boot.
  • The memory of one conversation. The decrypted account key, the inference credentials, and the sandbox's callback token live in that conversation's process alone, and die with it.

A complete backup is therefore Postgres plus MASTER_SECRETS_KEY. Nothing else holds state. The secrets model below explains why the key is half of that sentence.


Dependencies and failure domains

Dependency Needed for When it is down
Postgres Everything. GET /health/ready returns 503, and a load balancer drains the instance. The app stays up and does not restart, because a restart does not fix Postgres. That is why the restart check deliberately checks nothing. Recovery happens on its own. An instance that boots against a database that is down fails instead, because migrations run at boot.
sprites.dev Conversations. A new provision and a wake both fail, after bounded retries with backoff. Managoat then marks the conversation failed, or leaves it resumable if it was a wake. Everything else still works, which is sign-in, dashboards, agent, environment and vault management, and past logs. Readiness deliberately leaves it out, because a third party's uptime does not belong on the request path.
Mail, over SMTP, and optional. Signup verification, password reset, and the account-deleted confirmation. Each email is an Oban job that retries, so a short outage delays mail rather than losing it. A longer one dead-ends verification and reset. The escape hatch is Fountain.Release.verify_email/1. Read Configure email. Mail off (no SMTP_HOST) is a different thing, because an account then self-verifies (ADR 0011).
GitHub OAuth The OAuth login button, which is optional. That button, and nothing else. Email and password auth still works.
Sentry Error reports, which are optional. It is inert without a DSN, and nothing depends on it.
Object storage Database backups, at the ops layer. The application never touches object storage. It exists in a deployment's backup pipeline alone.
Inference providers Turns. The credentials belong to one user. Managoat stores them, and the sandbox consumes them. An Anthropic or OpenAI outage fails a turn, and not Managoat.

The secrets model

This is envelope encryption, in two layers.

  1. Each account has a data encryption key, a DEK. Managoat encrypts each environment and vault secret with it, under AES-256-GCM.
  2. Managoat stores the DEK itself wrapped by MASTER_SECRETS_KEY. That is an environment variable, and Managoat never writes it to the database.

Three results follow from that layout.

  • A database backup cannot decrypt itself. To restore the secrets you need the same MASTER_SECRETS_KEY that was live when somebody took the backup. So keep it apart from the backups.
  • Lose the master key, or change it without re-wrapping, and you lose each stored secret. Nothing else goes, because accounts, configs and history are plain rows.
  • An attacker with database access alone holds ciphertext and wrapped keys. They hold no value.

At spawn, a conversation's server loads its account's DEK into memory. It decrypts the environment secrets, then the vault secrets, and the vault wins on a key collision. It resolves the ${VAR} references in the agent's MCP config. It registers each value with the log redactor before the first line of log. It then hands the values to the sandbox, as the process env, and as an env file with chmod 600.

The DEK never leaves that process, and Managoat discards it when the process stops. The API is write-only throughout, and it never returns a stored value.


A conversation's life

Managoat records progress as stage events. The stage names below are the exact strings that the UI, the SSE stream and the log rows show. Each carries a state of started, done, failed or interrupted.

  1. Gates. A prompt arrives, through POST /api/conversations or through a client on it. Three gates apply, in order. The agent must exist and belong to you, because Managoat scopes each query in the system to one account. The vault must be on the agent's allowlist. The fleet ceiling of the agent's sandbox provider applies: 20 live sandboxes on that provider across the whole deployment by default (SANDBOX_FLEET_CEILING_SPRITES, SANDBOX_FLEET_CEILING_E2B), refused with 503 fleet_full past it.

  2. provision. Managoat creates the sandbox and conversation rows, as pending, then starts a conversation server. That server creates the sprite, mounts the skills, and mints a scoped API key that expires. The sandbox calls back into Managoat with that key. It assembles the env, and writes it into the sprite.

    Then the provisioning pipeline runs, which is network, then packages, then clone, then setup, from scratch in every sandbox. The runtime's ACP adapter downloads beside packages, clone and setup, once network is applied, and runs on the image's Node rather than one a setup script installs. The sandbox is then ready. A step that fails destroys the sprite, and marks the sandbox and the conversation failed.

  3. turn. Managoat spawns the runtime, which is claude, codex or gemini, inside the sprite. Its output flows back to the server, which redacts it, persists it as log events, and broadcasts it. The LiveView and the SSE endpoint both subscribe to the same feed. That endpoint is GET /api/conversations/:id/stream, and Last-Event-ID resumes it.

  4. Idle. The turn exits and the conversation goes idle, with the sandbox still warm. A follow-up prompt starts the next turn at once.

  5. reattach. A deploy or a restart can take the server away while the sprite survives. The next prompt, or the rehydrator at boot, then reattaches. It verifies that the sprite still exists, rewrites its env, and picks a turn's session back up where it stopped.

    The runtime's session lives on the sprite's disk. So a reattach to the same sprite keeps the agent's memory. When the sprite has gone, the conversation provisions again from step 2. Managoat's transcript survives, and the agent's session does not (#649).

  6. sandbox, which is the lifetime bounds. Each server checks its bounds each minute, and the two bounds do different things.

    To cross the idle timeout suspends. The server stops, the sprite stays and scales itself to zero, and the sandbox parks in suspended. The next prompt wakes it with everything intact.

    The max lifetime is off by default. When an operator sets it, to cross it destroys the sprite and marks the sandbox terminated. The conversation goes back to idle, and it is resumable at the #649 price. The hourly reaper applies the same split to whatever a crashed server left behind.

  7. terminate. An explicit POST .../terminate marks the conversation terminated. That is one of the two terminal states, next to failed. It destroys the conversation's sprite after the response: the sandbox is marked for teardown first and takes no new work, and a destroy that does not finish is completed within five minutes. Read the Sandboxes section of the API reference.

Here are the two status vocabularies, side by side.

conversation: pending -> running -> idle -> running ... -> failed | terminated
sandbox: pending -> starting -> ready <-> suspended -> terminated | failed

A conversation outlives its sandboxes. idle with a suspended sandbox is the normal state at rest, and not an error. A max-lifetime reclaim leaves idle with a terminated sandbox.


Where to look

The point of this page is that a symptom must predict a component. The action level, which says what to run and what the output means, is Self-host Managoat and its guides.

Symptom Look at
A conversation is stuck or failed at startup. The stage events in its log view name the step that failed. A packages, clone or setup failure is usually the environment's own config. A provision that fails outright is sprites.dev, or the fleet ceiling.
The UI is down, and GET /health/ready returns 503. Postgres.
A container restarts in a loop at boot. Migrations cannot reach Postgres, so the boot fails before it opens a listener.
The stream is dead for some viewers and correct for others. The replica count. Managoat runs as one node, and a viewer on a second replica sees no stream.
A signup never completes. Mail delivery. Read Configure email.
A sandbox suspends between prompts, and the work resumes on the next one. That is the design. Read lifecycle bounds.