Conversation
Give the CLI team the same home the docs team got in #282: a `cli` eval suite (`evals/cli/`, owned by @supabase/cli, exported to cli-eval-results.json with a cli-results.jsonl history) and a `cli` experiment suite scheduled nightly next to regression. Pairing stays folder-driven, so CLI experiment variants that force unusual environments never pair with other teams' evals, and vice versa. Add a `needsDocker` frontmatter field (default true). An eval that can run, and is meaningful, without a Docker daemon sets it false; experiments that stage Docker-less sandboxes select on it via `skipEval` instead of keeping lists of eval ids. Add the experiment-land runtime the CLI arms use: `resolveCliVersion` turns `stable`/`beta` into a concrete CLI version from npm dist-tags (lazily, memoised, overridable, never silently falling back to the pin), and `dockerAwareLocalStackRuntime({ channel, docker })` installs that version and, for `no-daemon`/`absent`, makes Docker unusable in layers the agent cannot undo — DOCKER_HOST pointed at an unbound port on every command, a root-owned `supabase` shim that re-exports it, the real docker binaries removed, and (no-daemon) a `docker` shim that still answers --version so the CLI's managed-stack runtime probe selects Docker as it would on a real host. CI's sandbox makes the socket world-writable, so permissions alone cannot stage this. Built only from `@supabase-evals/sandbox` exports. Experiments: codex-gpt-5.6-luna-cli-{stable,beta,nodaemon,absent} in the `cli` suite, and the pinned Codex Luna experiment tagged `cli` as the baseline column. No evals yet; the first is stacked on this branch. Refs: https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The first CI run of the cli suite lost every beta-channel pair: npm's `beta` dist-tag pointed at 2.118.0-beta.52 while its GitHub release was still a draft, so the .deb download 404'd before the agent started. The resolver now HEAD-checks the release asset and, for beta only, walks back to the newest published `-beta.N`; a missing stable asset throws. The workflow resolves both channels once in `prepare` (overridable via workflow_dispatch inputs) and forwards the pins into the Vercel sandbox, so one nightly never mixes CLI versions across pairs. Make the Docker-less arms structurally Docker-less: `DockerSandboxOptions` gains `mountDockerSocket` (default true) and the arms create their sandbox without the socket, then assert it is absent and that no `docker` resolves on the agent's PATH. DOCKER_HOST and the shims stay as defence in depth. Gate the new code in CI: `apps/framework/tsconfig.json`'s `experiments` include pointed at a nonexistent directory, so experiments were never typechecked; `pnpm check` now typechecks them and runs the `_lib` and `evals/cli` unit tests (`test:cli-lib`). Also: shared `skipUnlessCli`/`skipUnlessDockerless` predicates so mis-tagged evals skip instead of erroring; root-owned read-only runtime marker; single-quote shell escaping in the shims; resolved version logged per session; `needsDocker` frontmatter parse tests; CODEOWNERS for the CLI runtime and arms; CONTRIBUTING spells out which evals each arm picks up.
This was referenced Sep 18, 2026
7 tasks
Brings in #300 (native Grok Build CLI harness). Two conflicts, both additive on each side, resolved by keeping both: - `.github/workflows/eval-refresh.yml`: `XAI_API_KEY` alongside the `SUPABASE_CLI_{STABLE,BETA}_VERSION` run pins, in both the job `env` block and the `.env` heredoc. - `apps/framework/scripts/run-vercel-evals.ts`: `XAI_API_KEY` added to `FORWARDED_ENV_NAMES` ahead of the CLI version pins.
This was referenced Sep 22, 2026
Contributor
Author
|
Closing in favour of the split agreed in the thread with @mattrossman and @Rodriguespn — this PR bundled sandbox plumbing, framework schema, CI wiring and CLI-team experiments into one 1365-line diff that couldn't be reviewed without CLI domain knowledge. It's now four PRs, each reviewable on its own:
The substantive change versus what was here: Not deleting |
Coly010
added a commit
that referenced
this pull request
Sep 22, 2026
Splits the final chunk out of the closed #308: the cli suite/experiment enums, CODEOWNERS scoping, the four forced-environment CLI experiment columns plus their skip predicates, and CI wiring so the eval-refresh and gh-pages-history workflows know about the new suite.
Coly010
added a commit
that referenced
this pull request
Sep 22, 2026
…on (#324) First of two PRs splitting #308's sandbox work out of CLI-team experiment land, per the thread with @mattrossman and @Rodriguespn. This one is independent of #315 and of the `cli` suite entirely — it only widens an existing option. ## What changes `localStackRuntime({ cliVersion })` already accepted an exact version. It now also accepts a channel tag: ```ts localStack: localStackRuntime({ cliVersion: 'beta' }) ``` `'stable'` and `'beta'` resolve against npm's dist-tags for the `supabase` package once, at session start. An eval's own `cliVersion:` frontmatter pin still wins over the runtime option, exactly as before — and because that check short-circuits, a pinned eval never makes the network call at all. The resolver itself moves in from `experiments/_lib/cli-channel.ts` (it has been living in experiment land on the #308 branch). It does two things beyond reading a dist-tag, both of which encode incidents rather than caution: - It verifies the matching `.deb` release asset actually exists, and walks back through earlier `beta.N` candidates when the newest beta's asset has not been published yet. This is a real npm/GitHub draft-release skew we hit on `beta.52` in run 35274398896 — npm had the version, the release asset did not exist, and the install failed inside the sandbox. - It honours `SUPABASE_CLI_STABLE_VERSION` / `SUPABASE_CLI_BETA_VERSION` overrides, which is how a CI run pins one resolved version for all of its jobs. That last part is why `run-vercel-evals.ts` is in the diff: the two pins are added to the env names forwarded into each sandbox's `.env`. Without it, every sandbox job in a run re-resolves "latest" independently and a run can score against two different CLI versions if a release lands mid-run. ## Notes for review - `cliVersion` is typed `CliChannel | (string & {})` rather than `string | CliChannel` — the latter collapses to plain `string` and you lose editor suggestions for the two tags that are the point of the option. - `packages/sandbox`'s `test` script named its one test file literally, so it would have silently skipped the new one. It is now `vitest run test`; `docker.test.ts` already self-skips via `describe.runIf(process.env.SANDBOX_DOCKER_TESTS)`, so it stays inert. ## What is deliberately not here The `docker: 'no-daemon' | 'absent'` staging, `mountDockerSocket`, `needsDocker`, and the runtime marker are all in the follow-up, stacked on this branch. The `cli` eval/experiment suites, CODEOWNERS and workflow wiring come after that, once #315 lands. ## Verification `pnpm format:check`, `pnpm typecheck`, and the sandbox (89), core (162), framework (8), vercel-runner (28) and web (83) suites pass. `pnpm check`'s `test:framework` stage needs `AI_GATEWAY_API_KEY` for its judge and is not run in CI; every other stage was run.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
There is no home for CLI-team evals: the only suites are
benchmark,regression,docs,other; an experiment can only opt out of evals by id; every local-stack experiment installs the pinned CLI into a sandbox where Docker always works. "One task, run across the environments the CLI actually meets" — Docker available / daemon unreachable / absent, stable vs beta CLI — cannot be expressed without touching other teams' experiments.Expected Behavior
Infrastructure only — no evals in this PR (the first one,
build-database-002-stack-lifecycle, is stacked on it in #281). Mirrors how thedocssuite was added in #282.Framework
clieval suite (evals/cli/, owned by@supabase/cli) and acliexperiment suite. Nightly runscli × clinext toregression × regression; results export toapps/web/src/data/cli-eval-results.jsonwith acli-results.jsonlhistory on GitHub Pages. Pairing is by eval folder, so CLI experiment variants never pair with other teams' evals, and other teams' experiments never see CLI evals.needsDockerfrontmatter field (defaulttrue). An eval that can run — and is meaningful — without a Docker daemon setsneedsDocker: false; Docker-less experiments select on it withskipEval: (ev) => ev.metadata.needsDocker !== falseinstead of maintaining id lists.Experiment-land runtime —
experiments/_lib/(composes@supabase-evals/sandboxexports; no framework internals touched)cli-channel.ts: resolvesstable/betato a concrete Supabase CLI version from npm dist-tags (latest/beta), lazily insidestartSession(never on--dry/list), memoised, overridable viaSUPABASE_CLI_STABLE_VERSION/SUPABASE_CLI_BETA_VERSION, and failing loudly rather than falling back to the pin.docker-aware-local-stack.ts:dockerAwareLocalStackRuntime({ channel, docker }).docker: 'available'delegates to the stock runtime with the resolved CLI.'no-daemon'/'absent'build the sandbox from the same exported pieces and make Docker unusable in layers the agent can't undo:DOCKER_HOST=tcp://127.0.0.1:1on every command, a root-ownedsupabasePATH shim that re-exports it, removal of the real docker binaries, and forno-daemona root-owneddockershim that still answers--version(so the CLI's managed-stack runtime probe, fix(cli): improve stack startup defaults and clean managed state cli#6563, selects Docker as on a real host) but fails everything else with the canonical daemon error. Permissions alone can't do this: the CI sandboxchmod 666s the socket. A marker at/tmp/supabase-eval-runtime.jsonrecords channel, resolved version, staged docker state and session start for scorers' metrics.Experiments (Codex GPT-5.6 Luna,
suite: ['cli'])codex-gpt-5.6-luna(existing;cliadded to its suites)-cli-stableinterface: cli, not hosted-cli-betadevelop)-cli-nodaemonneedsDocker: false-cli-absentdockerbinaryneedsDocker: falseDocker-less arms use beta because the CLI's Docker-less path only exists in the managed stack, which ships there.
Review follow-ups (second commit)
Two independent reviews plus a triage of the first CI run of the
clisuite (run 35274398896, via #307) led to:betadist-tag pointed at2.118.0-beta.52while its GitHub release was still a draft, so the.debdownload 404'd and every beta-channel pair errored before the agent started (stable passed for exactly that reason). The resolver now HEAD-checks the release asset; forbetait walks back to the newest published-beta.N, forstableit throws — never falls back to the pin. The workflow resolves both channels once inprepare(overridable via newcli_stable_version/cli_beta_versiondispatch inputs) and forwards the pins into the Vercel sandbox, so one nightly never mixes CLI versions across pairs. Each session logs the version it installed.DockerSandboxOptions.mountDockerSocket(defaulttrue, 3 lines inpackages/sandbox); the Docker-less arms create their sandbox without the socket and assert it's absent and that nodockerresolves on the agent's PATH.DOCKER_HOST+ shims stay as defence in depth.apps/framework/tsconfig.json'sexperimentsinclude pointed at a nonexistent directory (experiments were never typechecked); fixed, andpnpm checknow runs the_libandevals/cliunit tests (test:cli-lib).skipUnlessCli/skipUnlessDockerlesspredicates (mis-tagged evals skip instead of erroring), root-owned read-only runtime marker, single-quote shell escaping in the shims,needsDockerparse tests, CODEOWNERS forexperiments/_lib/and the arms, CONTRIBUTING spells out which evals each arm picks up.Questions for the framework team, deliberately left as-is:
needsDocker: booleanvs arequires: { docker }object (kanad's #307 already uses the boolean); moving the suite→experiment-suite mapping out of the workflowcase; session facts (runtime id, resolved CLI version) on the scoring context to replace the marker file.Related Issue(s)
Refs https://linear.app/supabase/issue/CLI-2398/add-a-docker-less-local-stack-e2e-eval-agent-cli-lifecycle (first eval: #281, stacked on this)
Context: #team-ai feedback thread → Linear project "Evals Driven Development".
Test plan
pnpm checkandpnpm format:checkgreenpnpm --filter @supabase-evals/framework exec vitest run --root ../.. experiments/_lib— these paths are outside every packagetestscript, sopnpm checkdoes not run themtsc --strictoverexperiments/(not covered bypnpm check:apps/framework/tsconfig.json'sexperimentsinclude resolves to a nonexistent dir — worth a one-line fix separately)pnpm eval:dryforregression × regressionandbenchmark × benchmarkunchanged;cli × clihas no evals until feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281unset DOCKER_HOST; realsupabase starterrors surface; teardown leaves nothing behind), and with real agent runs via feat(evals): add build-database-002-stack-lifecycle CLI eval (CLI-2398) #281suite=cli,experiment_suite=cli