Add submission cli test for create run - #502
Open
anandhu-eng wants to merge 3 commits into
Open
anandhu-eng wants to merge 3 commits into
anandhu-eng wants to merge 3 commits into
Conversation
Run a sample pareto sweep against a dataset replay server on every PR, then dry-run each resulting run folder through `endpoints-submission-cli runs create` so the submission path stays exercised. The replay server answers each prompt with its own reference output, so the accuracy phase scores 100% by construction and no GPU or model is needed. `runs create --dry-run` returns before it reads a token or calls the API, so the job needs no PRISM credentials and creates nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
github-actions
Bot
added
the
size/large
'oracle' carried no meaning here beyond the test fixture's origin — the endpoint is a dataset replay server, not an oracle. Renames the sample point configs (sim-c<N>), the model name (sim-model), and the recorded system_name (sim_ci). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
anandhu-eng
marked this pull request as ready for review
September 11, 2026 15:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Nothing exercised
endpoints-submission-cli runs createagainst real benchmark output, so a change on either side could break the submission path unnoticed. This adds a per-PR job that runs a sample pareto sweep and dry-runs every resulting run folder through it.It adds:
system_desc.json, and callsruns create --dry-runfor each pointpull_request--dry-runprints the parsed payload and returns before it reads a token or calls the API, so the job needs no PRISM credentials and creates nothing.Two things worth knowing:
The points are sized for a fast gate, not for a conformant submission — rules §6.2 asks for 600s / 1200s of steady state, which these do not do.
The workflow installs the submission CLI from
mainrather than from a release, because v1.0.0.0 still expects a flatresult_summary.jsonwhile endpoints writesperformance/result_summary.json. mlcommons/endpoints-submission-cli#78 fixed the parser but landed after the tag. The step carries a comment to swap back to a pinned version once a release includes it.🤖 Generated with Claude Code