Skip to content

feat(orchestrator): add Kubernetes Kata backend for CLH and QEMU - #3572

Draft
leonmeijer wants to merge 1 commit into
e2b-dev:mainfrom
leonmeijer:codex/kubernetes-kata-runtime
Draft

feat(orchestrator): add Kubernetes Kata backend for CLH and QEMU#3572
leonmeijer wants to merge 1 commit into
e2b-dev:mainfrom
leonmeijer:codex/kubernetes-kata-runtime

Conversation

@leonmeijer

Copy link
Copy Markdown

What

  • add a runtime-neutral Kubernetes SandboxService, InfoService, and port-5007 proxy backend
  • support kata-clh by default and kata-qemu as the operator-controlled fallback through e2b.runtime-class
  • route explicit Kata requests only to OCI/Kata-capable orchestrators in mixed fleets
  • create one Pod, proxy credential Secret, and ingress/egress NetworkPolicy per sandbox
  • add regular-Pod Kubernetes discovery and reconnect a stable StatefulSet identity after Pod-IP changes
  • add guarded OKD manifests, a read-only preflight, renderer, and controller image

Safety and phase-one boundary

  • no Firecracker, Nomad, NBD, UFFD, host networking, host paths, or privileged Pods
  • the controller is non-root; only the tokenless sandbox ServiceAccount receives OpenShift anyuid so envd can run as UID 0 inside the Kata VM
  • sandbox egress always excludes private, loopback, link-local, and metadata CIDRs and uses public DNS instead of cluster DNS
  • Pod and Secret state is consumed through namespace-scoped informers, not Kubernetes API calls in the traffic hot path
  • snapshot resume, pause/checkpoint, auto-pause/resume, volumes, workload IAM, BYOP, and domain egress return typed unsupported errors before Pod creation

Template/build OCI images must already exist and contain /usr/bin/envd; this change intentionally does not translate the Firecracker artifact format.

Validation

  • go test -race ./pkg/kubernetesserver ./cmd/kubernetes-orchestrator
  • targeted API discovery, placement, reconnect, and node-manager suites
  • go vet for the changed orchestrator/API packages
  • shared sandbox-firewall tests
  • static Linux/amd64 controller build
  • shell syntax checks and client-side validation of all 9 rendered OKD resources
  • read-only make PROVIDER=okd plan OKD_CONTEXT=indentia-ap

The live preflight confirms both RuntimeClasses, 7 Ready/schedulable amd64 Kata nodes, required APIs/RBAC, and bind permission for anyuid. It remains READY=false because the worker MachineConfigPool is degraded/updating and there are no current runtime canary Pods.

No cluster resource was created, changed, or deployed. Keep this PR in draft until immutable controller/sandbox images exist, the worker pool is stable, and fresh CLH/QEMU end-to-end canaries pass.

@cla-bot

cla-bot Bot commented Aug 16, 2026

Copy link
Copy Markdown

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Leon Meijer.
This is most likely caused by a git client misconfiguration; please make sure to:

  1. check if your git client is configured with an email to sign commits git config --list | grep email
  2. If not, set it up using git config --global user.email email@example.com
  3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant