Architecture

This page is the system view. It says what processes exist, what each one talks to, and what breaks when a dependency is down. For the domain objects, which are agents, environments, vaults and conversations, read The four primitives.

The runtime shape

Fountain is one OTP release, fountain_server. Everything below runs in a single BEAM instance. There is no separate worker deployment and there is no sidecar. To scale out is to run more replicas of the same image.

Here is the scheduled work. All times are UTC.

Clustering

One replica needs none of this. With more than one, the replicas must form an Erlang cluster, and CLUSTER_DNS_QUERY must point at a headless service. Run more than one replica lists the env that wires it. Two things depend on it.

• The conversation registry places each conversation server on exactly one node, and finds it from any node.
• PubSub fans a conversation's events out to whichever replica holds the viewer's websocket or SSE connection.

Two replicas that never clustered form two islands. A conversation spawns without trouble, and the stream breaks without a sound for a viewer connected to the other replica.

The scheduled jobs elect one leader, whatever the replica count. The rehydration sweep at boot waits for cluster membership to settle, then runs on one node alone.

Where state lives

Postgres is the only durable store. Users, agent and environment configs, encrypted secrets, conversations, turns, log output, the audit trail, the job queue and uploaded images are all rows. A sandbox is disposable by design, and Fountain persists the output as it streams. So to destroy a sandbox loses nothing that mattered.

Two things are deliberately not durable.

• The in-memory tables on each node. Those are the rate-limit counters and the log-redaction registry. Fountain rebuilds them empty at boot.
• The memory of one conversation. The decrypted tenant key, the inference credentials, and the sandbox's callback token live in that conversation's process alone, and die with it.

A complete backup is therefore Postgres plus MASTER_SECRETS_KEY. Nothing else holds state. The secrets model below explains why the key is half of that sentence.

Dependencies and failure domains

The secrets model

This is envelope encryption, in two layers.

1. Each tenant has a data encryption key, a DEK. Fountain encrypts each environment and vault secret with it, under AES-256-GCM.
2. Fountain stores the DEK itself wrapped by MASTER_SECRETS_KEY. That is an environment variable, and Fountain never writes it to the database.

Three results follow from that layout.

• A database backup cannot decrypt itself. To restore the secrets you need the same MASTER_SECRETS_KEY that was live when somebody took the backup. So keep it apart from the backups.
• Lose the master key, or change it, and you lose each stored secret. Nothing else goes, because accounts, configs and history are plain rows.
• An attacker with database access alone holds ciphertext and wrapped keys. They hold no value.

At spawn, a conversation's server loads its tenant's DEK into memory. It decrypts the environment secrets, then the vault secrets, and the vault wins on a key collision. It resolves the ${VAR} references in the agent's MCP config. It registers each value with the log redactor before the first line of log. It then hands the values to the sandbox, as the process env, and as an env file with chmod 600.

The DEK never leaves that process, and Fountain discards it when the process stops. The API is write-only throughout, and it never returns a stored value.

A conversation's life

Fountain records progress as stage events. The stage names below are the exact strings that the UI, the SSE stream and the log rows show. Each carries a state of started, done, failed or interrupted.

1. Gates. A prompt arrives, through POST /api/conversations or through a client on it. Four gates apply, in order. The agent must exist and belong to you, because Fountain scopes each query in the system to one tenant. The vault must be on the agent's allowlist. The credit gate applies when CREDITS_ENABLED is on, and answers 402 at a zero balance. The quota for concurrent sandboxes applies. The balance funds it, between a floor of 2 and a ceiling of 20 for each user by default.

2. provision. Fountain creates the sandbox and conversation rows, as pending, then starts a conversation server. That server creates the sprite, mounts the skills, and mints a scoped API key that expires. The sandbox calls back into Fountain with that key. It assembles the env, and writes it into the sprite.

   Then one of two things happens. checkpoint_restore is a warm start from an environment checkpoint, and it skips the rest. Or the cold pipeline runs, which is packages → network → clone → setup. The sandbox is then ready. A step that fails destroys the sprite, and marks the sandbox and the conversation failed.

3. turn. Fountain spawns the runtime, which is claude, codex, gemini or opencode, inside the sprite. Its output flows back to the server, which redacts it, persists it as log events, and broadcasts it. The LiveView and the SSE endpoint both subscribe to the same feed. That endpoint is GET /api/conversations/:id/stream, and Last-Event-ID resumes it.

4. Idle. The turn exits and the conversation goes idle, with the sandbox still warm. A follow-up prompt starts the next turn at once.

5. reattach. A deploy or the loss of a node can take the server away while the sprite survives. The next prompt, or the rehydrator at boot, then reattaches. It verifies that the sprite still exists, rewrites its env, and picks a turn's session back up where it stopped.

   The runtime's session lives on the sprite's disk. So a reattach to the same sprite keeps the agent's memory. When the sprite has gone, the conversation provisions again from step 2. Fountain's transcript survives, and the agent's session does not (#649).

6. sandbox, which is the lifetime bounds. Each server checks its bounds each minute, and the two bounds do different things.

   To cross the idle timeout suspends. The server stops, the sprite stays and scales itself to zero, and the sandbox parks in suspended. The next prompt wakes it with everything intact.

   The max lifetime is off by default. When an operator sets it, to cross it destroys an ephemeral sprite and marks the sandbox terminated, or parks a persistent home. The conversation goes back to idle, and it is resumable at the #649 price. The hourly reaper applies the same split to whatever a crashed server left behind.

7. terminate. An explicit POST .../terminate marks the conversation terminated. That is one of the two terminal states, next to failed. It destroys an ephemeral sprite that no other conversation holds. A shared machine, or a persistent home, stays for the conversations that remain on it. DELETE /api/sandboxes/:id resets a persistent home. Read the Sandboxes section of the API reference.

Here are the two status vocabularies, side by side.

conversation:  pending -> running -> idle -> running ...     -> failed | terminated
sandbox:       pending -> starting -> ready <-> suspended    -> terminated | failed

A conversation outlives its sandboxes. idle with a suspended sandbox is the normal state at rest, and not an error. A max-lifetime reclaim leaves idle with a terminated sandbox.

Where to look

The point of this page is that a symptom must predict a component. The action level, which says what to run and what the output means, is Operations.