feat(cli): add the cli eval suite and experiment suite plumbing

[Coly010]

Final chunk of the closed #308 — the piece @mattrossman asked for: the minimal CLI suite plumbing and CODEOWNERS, standalone, with no CLI experiment internals to review. 15 files, 260 lines, and four of those files are 11–13 lines each.

Everything it depended on is now on main (#315's experiments/<owner>/ layout, and #324 + #325's sandbox options), so this is based on main and the diff is only its own content.

What it adds

cli as an eval suite and an experiment suite, @supabase/cli ownership over /evals/cli/, /experiments/cli/ and cli-eval-results.json, and the workflow wiring so eval-refresh and append-gh-pages-history know the suite exists.

The substance is five environment columns: one scenario run unchanged across forced CLI environments, so a failure isolates to which environment broke. They're thin now that experiments/presets.ts exists — the whole of the Docker-less column is:

export default defineExperiment({
  ...codexGpt6Luna,
  suite: ['cli'],
  // beta: the Docker-less path only exists in the managed stack, which ships in beta.
  localStack: localStackRuntime({ cliVersion: 'beta', docker: 'absent' }),
  skipEval: skipUnlessDockerless,
});

column	environment	picks up
codex-gpt-6-luna-cli-pinned	repo-pinned CLI, Docker available	every interface: cli eval that isn't hosted-linked
…-cli-stable	npm latest	same
…-cli-beta	npm beta	same
…-cli-nodaemon	beta, daemon unreachable	also needs needsDocker: false + projectRunning: false
…-cli-absent	beta, no docker binary	also needs needsDocker: false + projectRunning: false

All five columns share one skipEval: skipUnlessCli, so they run the same eval set and stay comparable — which is the whole point of the suite. An earlier revision made the pinned column the shared benchmark experiment with 'cli' appended to its suite; that experiment has no skipEval, so it picked up hosted-linked evals the other four skip. A CLI-owned pinned experiment fixes that and leaves this PR touching nothing outside CLI-owned paths.

skipUnlessCli / skipUnlessDockerless are the only things left over from the old experiments/_lib/; they live in experiments/cli/lib/, which discovery ignores for free since it only matches *.experiment.ts directly under an owner directory.

Verified, including the parts tests can't reach

format:check, typecheck, and the sandbox, core, framework, test:cli-lib, vercel-runner and web suites all pass.

Two things unit tests can't cover, checked directly instead:

• Discovery, since it's filename-convention-based and fails silently: all 16 experiments resolve and load, experiments/cli/lib/ is correctly not treated as an experiment, and the five columns resolve to the intended runtimes and channels (beta, beta, beta, stable, and none for the pinned baseline).
• The workflow, which only ever runs in CI, no longer resolves channels at all — it passes through the manual cli_stable_version / cli_beta_version dispatch inputs and otherwise lets run-vercel-evals.ts derive the channels from the actual pairs. Pre-filling both env vars would have short-circuited that narrowing.

Each run's results now record the CLI version the sandbox actually installed, read from the session's environment marker, so a stable row names its binary rather than just its channel. That needed cliVersionSchema widened — it rejected every prerelease, so a beta run's real version could not have been recorded at all.

test:cli-lib carries --passWithNoTests deliberately: evals/cli/ has no scorer tests until #281/#314/#316 land, and without the flag the script's exit code depends on which of its two paths happens to be populated.

Note on the column names

These follow the gpt-6-luna rename from #328. They key cli-eval-results.json, and no results exist yet, so there is nothing to churn — but that also means the names should settle before the suite is first run.

Next

#281, #314, #316 and #307 get re-parented onto main once this lands — they currently point at the retained branch of the closed #308. /cc @Rodriguespn @kanadgupta

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
evals	Ignored	Preview	Sep 23, 2026 11:37am UTC

[Rodriguespn]

Thanks for updating this for the GPT-6 changes. Can we make three changes before merging?

• Can we record environmentMarker().cliVersion in raw and exported results, so each stable/beta run names the exact CLI binary?
• Can we remove eager stable/beta resolution from the workflow, pass through only manual overrides, and let run-vercel-evals.ts resolve channels from the actual pairs?
• Can we add a CLI-owned pinned experiment using codexGpt6Luna and skipUnlessCli, so pinned/stable/beta use the same eligibility rules without changing the shared benchmark/regression experiment?

[Coly010]

all three done in fab2e19, CI green.

• results now record the version the sandbox actually installed, read from environmentMarker(), falling back to the eval's frontmatter pin when there's no sandbox. also flipped the precedence in export-results.ts - it had the frontmatter pin winning over what actually ran.
• workflow no longer resolves anything, just passes through the manual cli_stable_version / cli_beta_version inputs. folded the step into "Prepare inputs" since that's all it did once the npm call came out. run-vercel-evals.ts derives the channels from the pairs as intended - good catch that pre-filling both env vars was silently short-circuiting that.
• added codex-gpt-6-luna-cli-pinned with skipUnlessCli, and reverted the shared benchmark experiment back to ['benchmark','regression']. you were right, i'd checked for a config difference and missed that the eligibility predicate is the difference - pinned was picking up hosted-linked evals the other four skip, so the columns weren't comparable. side benefit is this PR now touches nothing outside cli-owned paths.

two things i hit on the way that weren't in your list:

cliVersionSchema was /^\d+\.\d+\.\d+$/, so it rejected every prerelease. recording a beta run's real version would have failed validation outright - widened it to match VERSION_RE in cli-channel.ts. it's the same const behind the frontmatter field, which seems fine since pinning a beta is legitimate.

resolveChannelPins used override ?? resolveCliVersion(...), so an env var set to empty string counted as a pin of '' rather than unset - which is precisely what a blank dispatch input now produces. trims and treats blank as unset, with a test.

[Rodriguespn]

Re-ran the latest version of this PR with #281’s build-database-002-stack-lifecycle scenario across all five CLI experiments. Each row records the exact CLI version, channel, and resolved
runtime.
LGTM 🚢