Three attempts at an internal agent

We wanted one internal agent for the whole team. You give it a job in chat. It does the work and replies in the same thread. That job might be session triage, an analytics question, a policy lookup, a scheduled report, or browser work.

We built it three times in three months. Condor v0 ran on Cloudflare, v1 on Docker, and v2 now runs on Kubernetes. Rebuilding it this often was painful, but each failed design made the next one less naive.

v0: agents as edge functions

1. IngressChat WebhookGoogle Chat and Telegram, signature checked before anything is routed.
2. signed event
   ControlHandler WorkerRoutes the event to its thread. Also the egress hook: every connection the sandbox opens is terminated back here.
3. turn
   Session stateThreadSession DOTurn log, budget caps and session id, in DO storage plus KV, so it survives a restart.
4. prior turns, budget left
   ExecutionSandbox ContainerClaude Code or opencode with bash and git, plus a headed Chromium for browser work. One per session.Outbound interceptionevery HTTPS connection is handed back to the Worker: deny list, then default-deny allowlist, then the real key swapped in for the placeholder. One audit row per decision.
   R2 Bundlesskills and knowledge, streamed in at boot

v0 ran entirely on Cloudflare. A chat webhook hits a Worker, which verifies the sender, normalizes the event, and routes it to a Durable Object keyed by chat thread. The Durable Object stores the turn log, budget counters, and session ID. It boots a Container for the session through the @cloudflare/containers Container class, posts the turn to a small server inside it, and streams text and tool calls back to the thread.

Inside the container, the harness was either Claude Code or OpenCode.

I would build the credential design again. Every credential environment variable inside the container contains the literal string credential-brokered. A boot scan checks every variable against secret-shaped regular expressions. If a real key slipped in, the container aborts before the agent starts. The real values stay in Secrets Store, bound to the parent Worker and never exposed to the container.

Outbound traffic interception enforces that boundary. The image trusts Cloudflare's certificate authority, and the Container class intercepts each request. The Worker checks a hard-deny list, then a default-deny allowlist. If the request passes, it replaces the placeholder with the real credential, forwards the request, and writes an audit row.

The rest of the state was scattered across Cloudflare products. D1 stored one audit row per egress decision. An R2 dead-letter bucket caught failed writes so we did not lose decisions. Workers KV held the session bearer token and budget counters.

R2 also stored the agent's knowledge and skills as markdown. We packed the files with reproducible tar flags into content-addressed bundles and streamed the selected bundle through the Worker at boot. A nightly Cron Trigger built and published a new bundle hash. We could update the agent's knowledge without rebuilding its image or deploying it again.

Three things killed it:

1. A tight allowlist made the agent safe and much less useful. It worked for a bounded task such as "investigate this session." It fell apart on SEO work, which might need product analytics, Search Console, the CDN, the website repository, and whichever competitor pages mattered that week. We had built an agent we could trust, then discovered it could not do the work we wanted to give it.

2. The harness was baked into the image, so changing it required a rebuild. A container crash killed the session. State lived in Durable Object storage, KV, D1, an in-memory map, and container disk. We could neither rewind a session nor replay it against another model.

3. Running all of this on Cloudflare cost too much.

v1: agents as an orchestrated swarm

1. ControlOrchestrator APIHost process. Holds the database and every credential. Spawn pipeline, reaper and audit log.
2. request
   CoordinationLead AgentTakes the request and delegates to role workers. Re-spawns an executor that dies, so a task is never orphaned.
3. delegate — 60–130 s cold start
   ExecutionWorker ContainerOne task, then discarded. Read-only rootfs, dropped capabilities, no secret anywhere inside. Identity is the bridge IP.MCP tool gatetools/call authorizes against the role's tool list and an argument policy. Fails closed.Per-spawn CONNECT proxyegress allowlist, auth injected on the wire, responses scrubbed on the way back. Its vault session is short-lived and minted for this spawn alone.
   Role Manifesttools, egress hosts and skills, pinned at boot

For the second attempt, we forked desplega-ai/agent-swarm and moved to Docker. A lead agent received each request and delegated it to workers assigned to code, observability, sales, research, or support. If a worker died, the lead could spawn a replacement and keep the task alive. That recovery model was the best part of v1.

The host process held the database and every credential. Workers ran in short-lived containers. The bridge IP identified the spawn, and the spawn identified its role. Each role had a YAML manifest listing its allowed tools and hosts, plus the knowledge and skills mounted at boot.

Tool access passed through two independent MCP checks. tools/list trimmed the catalogue to the worker's role. It failed open because a bug there only exposed a longer menu. tools/call checked the role's tool list and argument policy, and failed closed. Seeing a tool was harmless. Calling one was not.

An HTTPS CONNECT proxy sat between each worker and the internet. It held a short-lived vault session minted for that spawn, enforced the role's egress allowlist, injected authentication into outbound requests, and scrubbed the responses. The worker could use a credential without possessing it.

One container meant one worker and one task. Every task paid for a full cold boot, adding 60 to 130 seconds before useful work began. About seven serial configuration fetches took 30 to 50 seconds even when the API was healthy. Three more round trips fetched the vault session, GitHub token, and harness credentials.

We had accidentally started building a container orchestrator. Capacity gates, heartbeats, reapers, warm pools, delivery receipts. The work was real and occasionally interesting, but none of it made the agent better at its job.

v2: agents as a control plane

1. IngressChat and APIFour calls: session, messages, execute, events. Clients reconnect with after_event_id.
2. four calls
   ControlCentaur APIPostgres holds every turn. One execution per thread, enforced by a partial unique index, so it survives a restart.
3. claim
   SchedulingWarm PoolPre-booted pods claimed by one atomic statement. Idle pods pause to zero replicas instead of being killed, and resume on the next turn.
4. assigned pod — no cold start
   ExecutionSandbox PodA harness adapter for Claude Code, Codex or Amp. The workspace is a git clone --shared off a node-level cache, so it is a fresh branch with near-zero copying.Per-sandbox proxyplaceholders bound to a host and a request field, under a default-deny network policy. The pod gets no Kubernetes credentials and no third-party keys.
   Private Overlaytools, workflows and skills, pinned to a commit SHA

On the third attempt, we stopped writing our own control plane. We now run paradigmxyz/centaur on Kubernetes, with our changes kept in a private overlay repository.

Its scheduler has the model v1 needed. A session is separate from its sandbox, and the scheduler assigns a sandbox only when execution begins. It claims a pre-booted pod from a warm pool, removing the cold start from the request path. Idle pods scale to zero replicas instead of being destroyed, then resume on the next turn.

A partial unique index in Postgres allows one active execution per thread, even across restarts. A node-level repository cache fetches updates every 30 seconds and mounts read-only into each pod. The session workspace runs git clone --shared against that cache. Each session gets an isolated worktree and a fresh branch without downloading the repository or copying all of its objects.

The harness is now an adapter. Each implementation only needs to translate content into the form its runtime expects. It may save images and documents as files, pass Anthropic content blocks through unchanged, or extract plain text for a CLI that expects one prompt. The pod receives prompt files, a CLI command, an internal API URL, the proxy certificate authority, and proxy settings. It receives no Kubernetes credentials or third-party keys.

Credential brokering carries forward the best idea from v0. The sandbox holds a literal placeholder, and a proxy pod dedicated to that sandbox replaces it with the real value on the wire.

What actually generalizes

Start by brokering credentials at the network layer. Keep placeholders inside the sandbox. Inject real values on the wire, bound to an allowed host and request location. A compromised sandbox cannot leak a key that never entered its address space.

Cold starts change how people use an agent. When every task takes 60 to 130 seconds just to begin, people stop asking small questions. Claiming a warm pod and pausing it while idle improved the product more than another round of prompt tuning would have.

Put every event in an ordered session log. When state is split across five stores, you cannot rewind a session, replay it with another model, or reconstruct what happened after a crash.

Let the model choose its skills. We wrote two routers and regretted both. A substring matcher picked the wrong skill when words overlapped. A tool-call sniffer missed loads because the CLI injected the skill body as a system message without emitting a tool-call event.

Markdown skills packed into content-addressed bundles survived every version. Domain experts can edit them, and the agent can load a new bundle without a deployment. Most of the infrastructure changed in three months. Plain text did not.