Sandboxes: running an agent with no permission prompts

In the introduction I said the answer to unattended agents is to give them full permissions inside something where full permissions can’t hurt you. That’s probably easier said than done, and in this blog post I will explore the options for that.

Before we start: I don’t run Kubernetes for a living. I’m not a DevOps engineer or a cluster maintainer, I’m a developer experimenting on a small cluster of my own. So read this as what I tried and what I learned, not as a reference architecture, and check the security parts against your own setup before you rely on them.

Something where full permissions can’t hurt you comes down to two things: where it runs, and what it can reach. Where it runs is a pod or a VM, thrown away when the work is done, so the worst an agent can wreck is its own copy. What it can reach is a short list: nothing leaves unless I allowed it, and the model key isn’t in there to steal. I run mine as Kubernetes pods, so that’s where this starts.

The pod: the fields that matter

My sandboxes run on a small Kubernetes cluster that I’ve built from refurbished machines (4x HP EliteDesk 800 G6 Mini), mainly for learning and playing around, so that’s what I will also use to run my containers/ pods on. Each one a microVM under Kata Containers, with Cilium for network policy. You don’t need that exact stack.

Here’s the shape (trimmed), and the fields worth knowing about follow it:

spec:
  podTemplate:
    metadata:
      labels:
        sandbox: "true"            # the egress policy selects on this
    spec:
      runtimeClassName: kata       # admission refuses a sandbox without it
      automountServiceAccountToken: false
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        fsGroup: 65532        
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: agent
          image: your-registry/agent-sandbox@sha256:<digest>   # a digest, never a tag
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            limits: { memory: 6Gi, ephemeral-storage: 8Gi }
          volumeMounts:
            - { name: workspace, mountPath: /workspace }
            - { name: tmp, mountPath: /tmp }
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 2Gi         # dotnet, npm and Chromium all want a /tmp

One important part of this is the label. If we forget sandbox: "true", nothing errors: the pod runs fine without any network constraints, and our agents can take from the web, and send to it, what they want.

The fsGroup setting allows the agent to perform and create, for example, folders inside of the container in the workspace, what it needs to get nuget packages etc. And overall the container is using the runtimeclass kata, which gives each pod its own kernel inside a VM. An agent that breaks that kernel only ends up inside a VM with nothing else in it, and still has to get through the hypervisor to reach the host. That field is the boundary. The other security settings are good hygiene, but a kernel exploit can walk past them.

Why Kata? I started on gVisor, which handles a pod’s system calls in userspace instead of passing them to the host kernel. Kata gives each pod its own kernel in a small VM, and on real work it wasn’t slower: a warm npm ci took 1,775 ms against gVisor’s 2,085. It costs about a second more to start and about 30 MB more idle memory.

The downside is that each VM runs its own copy of Linux, and that copy needs security updates just like the machine it runs on. The cluster also refuses to start a sandbox that isn’t using Kata or gVisor, so one can’t end up without this protection by accident.

Of the two volumes in that spec, /workspace is the one that outlives the pod. Each phase gets its own fresh pod, but the phases of one attempt share that volume, so dependencies install once per attempt instead of once per phase.

It’s also the one place an agent can leave something behind. Anything a phase writes there, a tampered package, a git hook, a change to $HOME, is waiting for the next phase to run it. A clean volume per phase would close that, but every phase would clone the repository and install its dependencies again, and that’s most of the wall clock of a pipeline. So this is a balance between security and effort: I accept that risk within one attempt, and you should treat what’s on that volume as untrusted. That includes my check phases, which mount the same volume, so a check can run with a $HOME an earlier agent changed.

Because the volume outlives its pods, deleting a pod doesn’t clean it up, so give the volume a single owner: in my setup, one component in the factory creates it and cleans it up.

Egress: default-deny, and every allowance has a reason

The less an agent can reach, the less it can download or send somewhere it shouldn’t. So nothing gets out by default, and each host is added on purpose. The policy permits DNS, the model broker, the git broker and a short list of package hosts, almost all on port 443. That rules out git over SSH: it’s HTTPS with a token.

Keep that list in code, with a reason per host. I started with a markdown file and a policy applied by hand, and the two quietly drifted apart. Now the list lives in code and a configuration page renders it:

public static IReadOnlyList<EgressHost> Shared { get; } =
[
    new("registry.npmjs.org", "npm install, for a repository with a package manifest."),
    new("api.nuget.org", "dotnet restore."),

    // Cilium reads `*.nuget.org` as requiring a subdomain, so it does not cover the apex.
    new("*.nuget.org", "The CDN api.nuget.org redirects package downloads to."),
    // ...
];

If nobody knows why a host is on the list, nobody dares remove it. So my list says what each tool needs, not what the cluster happens to allow right now. Compare the two and you can see when they’ve drifted apart.

Three things to watch for when you build your own list:

• Blocked traffic looks like a slow network. A blocked request doesn’t fail straight away. The tool just waits until it times out.
• Some tools check a host before they start. The interactive Claude Code CLI won’t start unless it can reach api.anthropic.com, even if you point it at a different base URL. The non-interactive form doesn’t check, so it runs without that host on the list.
• Agents don’t know there’s no web. They waste turns finding out, seven in one of my runs, and the error doesn’t help: it names the site the agent asked for, when what really failed was a safety check the fetch tool makes first. So tell the agent up front that there’s no web, and where the documentation is on disk.

Testing the policy once isn’t enough. A CNI upgrade, a typo, a label that stops matching, or someone widening the list “just for a minute” all reopen the internet silently. So a CronJob checks it every thirty minutes: an allowed host must answer, an arbitrary host must not, and neither must a raw IP address, because a DNS-based policy says nothing about traffic that never asks for a name.

The job runs like a real sandbox, under the same runtime, on a sandbox node, with the same sandbox: "true" label, because a probe exempt from the policy passes for the wrong reason. If I did this again, I’d run the policy in audit mode first, which is a cheap way to see what it would block.

The model key stays outside

An agent can read its own environment, so any key inside the sandbox is a key it can send somewhere. It’s also a way around your budget: limit an agent to ten euros, and with the key it can start new work that limit never sees. So the sandbox doesn’t get a key. It gets the address of a broker, a small nginx proxy inside the cluster, and the broker adds the real key on the way out. The egress policy does the other half: requests for work can only get to a model through the broker.

The core of it is small:

# Resolved per request, not once at startup.
resolver kube-dns.kube-system.svc.cluster.local valid=30s ipv6=off;

location / {
  set $anthropic api.anthropic.com;
  proxy_pass https://$anthropic$request_uri;

  # Overwrite, never add: a sandbox can't send a key of its own
  # and can't read the one that's used.
  proxy_set_header authorization "Bearer ${ANTHROPIC_AUTH_TOKEN}";
  proxy_set_header x-api-key     "";

  # Buffering turns a stream into one long silence, which looks exactly like a hung run.
  proxy_buffering off;
}

Four things make it work:

• Overwrite the auth headers. That’s the whole mechanism. proxy_set_header replaces both headers, so whatever the pod sends is thrown away.
• Look up the provider’s address per request. With a literal hostname, nginx resolves it once at startup. When the provider’s addresses change, the broker keeps dialling an old one that’s no longer allowed, and all the sandbox sees is a hang and then a 502.
• Give the pod a placeholder key. With only a base URL set, Claude Code thinks it’s logged out and asks for /login before sending anything. ANTHROPIC_API_KEY=broker gets past that. It isn’t a secret, because the broker overwrites it.
• Expect other tools to need their own lines. Codex opens its stream over a WebSocket, which nginx breaks unless you pass the upgrade headers. The harnesses post covers that per tool.

The broker doesn’t stop an agent sending the model whatever it can read, and it narrows the spending problem rather than closing it. The key can’t leave the sandbox, but while the pod runs an agent can still call the broker directly, and my budget wouldn’t count that: the budget reads what the CLI reports, and the broker counts nothing.

So what limits that today is how long a pod may run, and the broker is the one place a real limit could go.

Getting work in and out

An agent has to get the repository in and its commits out, and that takes a credential for your git host. Any token inside the pod is one the agent can read, so this is the same problem as the model key, with the same answer: a broker.

I didn’t start there. First I copied the repository in and out as git bundles, so the pod needed no access at all. That stopped working when checks moved to the cluster, because they need somewhere to fetch the work from. For a while the pods held a git token, and now they go through the broker.

Each pod gets a token for its own run, which only the broker accepts. The broker holds the real credential, adds it on the way out, and only passes git traffic for that run’s repositories. Git in the pod doesn’t need to know: a url.<broker>.insteadOf setting rewrites every URL to the broker. That setting lives in the environment rather than in .gitconfig, because $HOME is on the shared volume where the agent can edit it.

Agents push to a ref outside refs/heads, something like refs/factory/runs/<run>. Git hosts don’t list those refs or offer them in a pull-request dropdown, so attempts still being judged don’t fill your repository with branches. Checks fetch that ref. When the gates pass, one pipeline step pushes a real branch and opens the pull request. That step runs in the dispatcher, never in a pod, so the credential that can open pull requests never reaches an agent.

So be precise about what “no credentials in the sandbox” means in your own setup. In mine it means no model API key and no git credential, only a few tokens scoped to one run, plus the logins some tools can’t do without.

Two things to get right when the work leaves through a push:

• Wait for a real end state before deleting a pod. An empty answer from kubectl isn’t “finished”. Treating it that way got a pod deleted part-way through its push, and the run was recorded as a success for commits that existed nowhere. Only Succeeded or Failed should end the wait.
• Don’t let a dead pod count as “no work”. If an agent pushes its commits and then the CLI crashes, the pod can’t report what it did. Count the commits on the pushed ref yourself, or retries get told the phase changed nothing.

Test the image the way the pod runs it

The pod needs an image with everything an agent uses already in it: the agent CLI, git, the .NET and Node toolchains, a browser. No off-the-shelf image has that combination, so I build my own.

That image runs under tighter rules than your laptop: a read-only root, a non-root user and an empty workspace. Things that work locally can break there, usually with an error that doesn’t point at the cause. For example, the workspace volume mounts over /workspace, so anything you put there during the build is gone when the pod starts, including $HOME.

So make these problems fail the build, not the pod. The last build step runs every tool by name. Then the image is verified under the pod’s exact constraints (read-only root, uid 65532, no capabilities, empty workspace), including a browser that has to write a screenshot. A version string wouldn’t tell you the browser can start.

What isolation costs

On your own machine a restore is nearly always fast, because your package cache has been filling up since you started. A sandbox starts with nothing, so without a cache every new attempt downloads and unpacks everything. In mine, a cold NuGet restore of a small Aspire AppHost took 438 seconds, on a connection that could have fetched it all in under half a minute. Warm, the same restore took 2 seconds.

I keep packages in a cache that outlives the run, one per node, synced between nodes. The catch is that a phase that poisons a package folder writes to its node’s cache, and the sync copies it to every other node. I’m thinking about adding hashes, or some other way to validate that the cache hasn’t been contaminated, but I haven’t built or tested that yet.

When you design your cluster, think about what it means if everything starts fresh each time. Fresh is the safest option and the slowest one, and every cache you add to win that time back is one more place an agent can leave something behind. Where that balance sits is up to you, but make it a decision.

Who commits

Every commit a factory makes has an author. If you don’t decide who, git decides for you: a run commits as whoever owns the machine, which reads as a person having written code they never saw. In a container there may be no identity at all, and git commit stops partway through a run.

So agents commit under the factory’s own, obviously synthetic name. Those commits reach your repository through a pull request, and a person reviews and merges it. That’s where a human name goes on the work.

Set the identity in the environment, not in git config. A worktree shares its repository’s config file, so a run writing to it would change your own checkout too.

GIT_CONFIG_COUNT=3
GIT_CONFIG_KEY_0=commit.gpgsign  GIT_CONFIG_VALUE_0=false
GIT_CONFIG_KEY_1=user.name       GIT_CONFIG_VALUE_1=<factory identity>
GIT_CONFIG_KEY_2=user.email      GIT_CONFIG_VALUE_2=<factory address>

Signing is off, because an unattended run has nobody there to unlock a key. If your git host requires signed commits, you’d need a key the factory owns, which is one more credential to look after.

What the sandbox doesn’t solve

A sandbox is an additional layer of protection, not the whole answer. These are the gaps it leaves open:

• Data can still leave. Whatever the agent can read, it can send to an allowed host like GitHub. It can also hide data in a DNS lookup, because my policy lets it look up any name.
• An escape reaches the neighbours. The network rules apply to the pod, so code with root on the node is outside them. On mine, that node also runs a registry, a CI runner and the broker holding the real model key.
• Reachable means allowed. My network policy opens a host and a port, not individual routes. If a sandbox can reach an API, it can call all of it, so give sandboxes a small service with only the routes they need.
• Your laptop. With podman on my own machine, the same sandbox is a plain container on my laptop’s kernel, not a VM.

A sandbox limits what an agent can reach. It can’t tell you whether the agent did the right thing.

What’s next

Next up: pipelines. What it took to make code, not the agent, own the sequence: tool boundaries that hold, hand-offs that are typed data, templates per stack, checks that say what they need, and a way for a run to stop and ask a person before the money goes.