feat(sandbox): resolve npm channel tags for the local-stack CLI version - #324
Merged
Merged
Conversation
Moves the CLI channel resolver into @supabase-evals/sandbox and lets localStackRuntime's cliVersion accept 'stable'/'beta' in addition to an exact version, resolved against npm's dist-tags at session start. An eval's own cliVersion: frontmatter pin still wins. Forwards the resolved channel pins to Vercel sandbox jobs so a run scores against one version instead of each job re-resolving "latest" independently.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This was referenced Sep 22, 2026
- Export FORWARDED_ENV_NAMES from run-vercel-evals.ts and import it in the test instead of a drifted local copy, so a leaked XAI_API_KEY can no longer break agentEnvironment()'s assertions. - Make cliDebUrl's arch param required and check both amd64 and arm64 release assets, since the resolver's host arch can differ from the sandbox container's; delete the now-unused hostDebArch. - Generalise the beta walk-back to every published X.Y.Z-beta.N version older than the unpublished one (not just the same minor), ordered with a numeric compareBetaVersionsDesc comparator instead of a string compare. - Treat a thrown error for one walk-back candidate as "unavailable" and keep probing the rest, instead of aborting the whole walk-back; the initial dist-tag asset check still fails loud. - Import isRecord from @supabase-evals/core/json instead of a local copy that dropped the array guard. - Guard the version cache's rejection cleanup with an identity check so a stale promise can't evict a newer cached one. - Drop cliDebUrl/hostDebArch from the sandbox barrel (no external consumers) and correct local-stack-runtime's doc comment: channel resolution is memoised per process, not per session.
… resolution Strips agent-added comment cruft (module docblock essay, CI-run-ID call-out, code-shape narration) back to the invariants and external quirks the code can't express on its own; no behaviour change.
Rodriguespn
reviewed
Sep 22, 2026
Rodriguespn
reviewed
Sep 22, 2026
Rodriguespn
requested changes
Sep 22, 2026
Rodriguespn
left a comment
Rodriguespn
left a comment
Contributor
There was a problem hiding this comment.
Thanks for moving CLI channel resolution into @supabase-evals/sandbox; that matches the decision in this thread.
Left two comments over non-404 GitHub errors can select an older beta, and the workflow does not create one channel pin for all sandbox jobs. Both can assign results to the wrong CLI version.
Please fix those two paths and add the focused tests described in the inline comments.
…ls once per run debAssetHeadOk conflated a GitHub 429/5xx with a missing asset (response.ok false for both), which could downgrade a beta release during a GitHub outage and record results under the wrong version. Only a 404 now means "asset not published"; any other non-2xx throws. The beta walk-back loop is updated to match: a 404 still advances to the next candidate, but a thrown error aborts the walk-back and propagates instead of being caught and skipped. Separately, nothing set SUPABASE_CLI_STABLE_VERSION/SUPABASE_CLI_BETA_VERSION, so each sandbox job resolved its own CLI channel version independently, risking two different versions in one fan-out if a release landed mid-run. runPairs now resolves each channel pin once via resolveChannelPins() before runBounded and threads the same pins into every job's .env write.
Rodriguespn
approved these changes
Sep 22, 2026
Rodriguespn
left a comment
Rodriguespn
left a comment
Contributor
There was a problem hiding this comment.
Thx for addressing my feedback. Approving to unblock you but please take a look at this comment before merging it
Union each pair's effective channel (exact eval pins win and need none; otherwise the pair's experiment's localStack.cliChannel) before fan-out, instead of unconditionally resolving both channels. A run with no CLI-channel experiments now does zero npm/GitHub lookups. An experiment that can't be resolved to a config throws instead of silently counting as needing no channel.
…i-version-channels
The controller's channel lookup built experiments/<name>.ts by hand, which stopped matching when experiments moved to experiments/<owner>/*.experiment.ts. Use the shared discovery helper so the path convention lives in one place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of two PRs splitting #308's sandbox work out of CLI-team experiment land, per the thread with @mattrossman and @Rodriguespn. This one is independent of #315 and of the
clisuite entirely — it only widens an existing option.What changes
localStackRuntime({ cliVersion })already accepted an exact version. It now also accepts a channel tag:'stable'and'beta'resolve against npm's dist-tags for thesupabasepackage once, at session start. An eval's owncliVersion:frontmatter pin still wins over the runtime option, exactly as before — and because that check short-circuits, a pinned eval never makes the network call at all.The resolver itself moves in from
experiments/_lib/cli-channel.ts(it has been living in experiment land on the #308 branch). It does two things beyond reading a dist-tag, both of which encode incidents rather than caution:.debrelease asset actually exists, and walks back through earlierbeta.Ncandidates when the newest beta's asset has not been published yet. This is a real npm/GitHub draft-release skew we hit onbeta.52in run 35274398896 — npm had the version, the release asset did not exist, and the install failed inside the sandbox.SUPABASE_CLI_STABLE_VERSION/SUPABASE_CLI_BETA_VERSIONoverrides, which is how a CI run pins one resolved version for all of its jobs.That last part is why
run-vercel-evals.tsis in the diff: the two pins are added to the env names forwarded into each sandbox's.env. Without it, every sandbox job in a run re-resolves "latest" independently and a run can score against two different CLI versions if a release lands mid-run.Notes for review
cliVersionis typedCliChannel | (string & {})rather thanstring | CliChannel— the latter collapses to plainstringand you lose editor suggestions for the two tags that are the point of the option.packages/sandbox'stestscript named its one test file literally, so it would have silently skipped the new one. It is nowvitest run test;docker.test.tsalready self-skips viadescribe.runIf(process.env.SANDBOX_DOCKER_TESTS), so it stays inert.What is deliberately not here
The
docker: 'no-daemon' | 'absent'staging,mountDockerSocket,needsDocker, and the runtime marker are all in the follow-up, stacked on this branch. Theclieval/experiment suites, CODEOWNERS and workflow wiring come after that, once #315 lands.Verification
pnpm format:check,pnpm typecheck, and the sandbox (89), core (162), framework (8), vercel-runner (28) and web (83) suites pass.pnpm check'stest:frameworkstage needsAI_GATEWAY_API_KEYfor its judge and is not run in CI; every other stage was run.