↓Skip to main content

Kata Containers: sandboxing AI agents and CI pipelines

·11 mins
Kata-Containers Kubernetes Talos Sandboxing Ai-Agents Ci-Cd Forgejo Security Homelab Platform-Engineering
Table of Contents

My nodes run Talos Linux with Cilium handling the network. Every container on a node still shares one Linux kernel, so a single kernel bug or one over-privileged pod puts everything on that node at risk. That matters more when running untrusted code: CI jobs triggered by pull requests, and whatever an AI agent decides to execute. Kata Containers closes that gap, and in this post I’ll explain how.

What a container shares the Linux kernel #

A container is a Linux process with a restricted view of the system. It can only see its own files and processes, and it can only use the CPU and memory it was given. One kernel enforces all of that, the same kernel that runs every other container on the node and the node itself. A flaw in that kernel, or a container started with privileged: true, lets the process bypass the restrictions which puts the entire host at risk, and gives the container access to the kernel which enables it to access other containers as well.

For trusted workloads that trade is fine. You wrote the code yourself, or you trust the people who did. Two kinds of workload should never be trusted by default. The first is code an AI agent writes and runs on the fly, steered by whatever text the agent read last. The second is a CI job that builds a pull request, installs dependencies and runs scripts from a repository you only half control. Both execute untrusted code on infrastructure that you want to be safe and secure.

How Kata Containers change this #

Kata Containers runs each pod inside its own lightweight virtual machine, with a dedicated kernel. The project describes it as the isolation of a traditional VM with the speed and efficiency of containers, and the Kata blog post on AI workloads makes the same case for sensitive data and dynamic workloads.

Kata containers integrate with Kubernetes and you can configure it per pod. When Kubernetes starts a pod, a setting called a RuntimeClass tells it to start the pod through Kata instead of the standard runtime. Kata boots a small VM, usually in a few seconds, and runs the containers inside it. The VM lives exactly as long as the pod. Everything else about the pod stays the same: logs, exec, probes and services all work, so Kubernetes doesn’t notice the difference.

The major improvement: the isolation between the workload and the node changes. In a standard container that isolation is the within the shared kernel. With Kata it is a hypervisor, the software that runs virtual machines. A process that breaks out of its container lands inside the VM’s own kernel, and reaching the node from there means attacking the hypervisor, which is at least one more technology to break before reaching the node itself.

One VM covers the entire pod, so containers in the same pod share a kernel with each other. The isolation sits between pods.

Why I chose Kata #

I wanted two things: strong isolation, and no new platform to learn. Kata gives me the first through a separate kernel, and the second through the standard RuntimeClass mechanism. Adopting it means one field in a pod spec. I didn’t need a new scheduler or a new orchestrator.

I also looked at gVisor, which takes a different route. It puts a kernel written in user space between the container and the host. The container’s requests go to that kernel, and it makes a narrow set of calls to the host kernel on its behalf. That shrinks what a container can reach, but the user-space kernel still runs as a process on the host and talks to the host kernel, so a flaw there is a flaw on the node. Kata gives the workload a real kernel inside a VM with an additional boundary around it. For code I don’t trust, I want the additional boundary.

Securing CI pipelines #

A CI runner executes code from your repositories, including code from pull requests and dependencies you cannot trust. The standard build-setup makes that even worse. Runners that build container images need a Docker daemon, which means mounting the Docker socket or running a privileged Docker-in-Docker container. Either one is effectively root on the node. A malicious build script that finds it has the node.

My CI runs on Forgejo, and two decisions shape the setup. First, runners are ephemeral: the platform starts one pod per queued job and deletes it when the job ends. Nothing from job A is on disk when job B starts, and a compromised runner has no next job to attack. Second, every one of those pods runs under Kata. Each job gets a fresh VM with a fresh kernel, and a breakout from the job container lands in a VM that is about to disappear.

Forgejo is my example because it’s what I run. This works the same way with GitLab runners, Gitea’s Actions runner or GitHub’s Actions Runners. The requirement is a runner that executes as a pod and lets you set runtimeClassName in its pod template.

Isolating the kernel covers half of CI. The other half is the network. A CI namespace under default-deny egress can reach the git server, the registry and the build cache, and nothing else. It can’t reach other the public internet, other pods, the Kubernetes API or the secret store. A compromised job inside a VM with nowhere to go is already contained.

Running AI agents in a sandbox #

An agent that executes code is a machine for turning text into commands. A prompt injection buried in a web page or a document changes “summarise this” into “run this”, and the agent has no way to tell the difference. I treat everything an agent executes as untrusted, whoever wrote the prompt. That is why I run the same architecture for agents on my homelab as I do for CI.

In a standard container, an exploit in that code reaches the node. In a Kata pod it reaches the VM’s own kernel, inside a VM that exists for this one task and disappears afterwards. Pair that with a network policy that denies all egress except the model API and the tools the agent needs, and the agent can’t contact an unknown host without a change to the policy. Give the pod no credential for the kubernetes API and mount only the secrets the task needs. A successful prompt injection can still happen, but with a separate kernel, a tight egress policy and limited credentials, it ends in a throwaway VM with nowhere to send data to.

Kata contains what the code can reach on the node. What the agent can reach from inside the pod stays your decision, and the egress policy carries most of that weight.

Note: I also use a AI API gateway to further decrease risks by detecting prompt injections, filter out secrets, etc. As that is a whole other subject, it deserves its own blog post once.

What it costs #

Kata isn’t free, and three costs shaped how I use it.

Hardware. The VM needs KVM, the virtualisation support built into the Linux kernel. My homelab runs on bare metal, where the virtualisation support is available. On a virtual machine you need nested virtualisation, which not every platform offers and which adds overhead of its own.

File I/O. Kata shares files between the node and the VM through virtio-fs, the default since Kata 2.0. Every file access crosses the VM boundary, and that has a price. Workloads that move a lot of data through the filesystem pay it on every read and write, so I wouldn’t put a database or a training job over a large dataset in a Kata pod. An agent that calls an API, runs generated code and writes small files barely notices.

Docker-in-Docker. Kata contains privileged Docker-in-Docker, and it still isn’t something I like. Inside the VM the container has full rights. A job that escapes its container controls the VM, along with every secret mounted in the pod and everything the pod can reach on the network. Kata moves that damage off the node but it’s still something I don’t want. Buildah builds images without a daemon and is the alternative I want to try next.

Security and compliance #

Things I do have mostly a link with security or compliance. And if you work under regulation, what you can show and audit matters. The EU AI Act is in force, but it doesn’t require sandboxing or isolation as a technical measure. “Sandbox” in the Act means something else: a regulatory sandbox, a supervised testing scheme under Article 57. What the Act does require is cybersecurity. High-risk AI systems must hold up against unauthorised third parties trying to exploit vulnerabilities, and providers are expected to look at the underlying ICT infrastructure when they choose their security controls.

So the Act asks for appropriate controls on the infrastructure that runs the system, and leaves the choice of control to you. A runtime that gives untrusted code its own kernel, which Kata containers provides, is a control you can explain, and it’s auditable:

  • Every pod in a CI namespace runs with the Kata runtime class, because an admission rule rejects any pod that doesn’t.
  • The rule’s denials and policy reports show it holding, and a cluster that can’t run Kata says so in those reports.
  • The policy lives in Git, so its history shows who changed it and who approved the change. I covered that in the post on GitOps as an audit trail.
  • The kernel inside a pod differs from the node’s kernel.

“Untrusted code runs in a separate kernel” turns from a claim into a rule plus a record of it holding.

How you can run it yourself #

This is the short version and I assume you already have a running Kubernetes cluster.

Step 1: Put the runtime on the nodes #

Every node that runs sandboxed pods needs the Kata runtime and KVM. On Talos you add the kata-containers system extension to your node image. On other distributions the Kata project ships kata-deploy to install it. Check the device and the extension afterwards with these commands:

talosctl -n <node-ip> list /dev/ | grep kvm
talosctl -n <node-ip> get extensions

If kvm is missing, the node is a VM without nested virtualisation, or virtualisation is off in the BIOS.

Step 2: Create the RuntimeClass #

The RuntimeClass gives Kata a name that pods can ask for. The handler must match the runtime name your containerd registers, which is kata on my nodes, so check yours. Create the object with this manifest:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata-qemu
handler: kata

Step 3: Run a pod in it #

Opting in takes one line in the pod spec. Add this field to any pod:

spec:
  runtimeClassName: kata-qemu

To check that it works, compare the kernel inside the pod with the kernel on the node:

kubectl exec <pod-name> -- uname -r
kubectl get nodes -o wide

The first command prints the pod’s own kernel version. The KERNEL-VERSION column of the second shows the node’s kernel. They differ, because the pod runs its own.

Step 4: Make it mandatory #

A RuntimeClass is opt-in, so a pod without the field runs on the shared kernel. For namespaces that run untrusted code, I enforce it at admission with Kyverno. This policy rejects any pod in a namespace labelled ci: "true" that doesn’t use the Kata runtime class:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: ci-pods-must-use-kata
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: require-kata-runtime-class
      match:
        any:
          - resources:
              kinds: ["Pod"]
              operations: ["CREATE"]
              namespaceSelector:
                matchLabels:
                  ci: "true"
      validate:
        message: "Pods in CI namespaces must set runtimeClassName: kata-qemu."
        pattern:
          spec:
            runtimeClassName: kata-qemu

Then lock the network. I use Cilium, as in the Cilium post, but standard Kubernetes network policies do the same job, because a Kata pod looks like any other pod to the network. Deny all egress from the namespace, then allow DNS and the services your jobs need. Mine allows the git server, the registry and the build cache, plus a switch for outbound web traffic so builds can download packages. Add an entry for each service you need and nothing more.

Wrapping up #

Kata Containers gives every pod its own kernel behind a hypervisor boundary. With an enforced runtime class and a default-deny network, untrusted code from a CI job or an AI agent runs in a VM that has little to reach and disappears afterwards. It costs you bare-metal nodes and slower file I/O, and for code you can’t trust it’s a good trade-off.

This setup covers my own CI and my AI agents that run on my homelab, but the weakest spot left is still Docker-in-Docker. Buildah is the next thing I want to try there.