feat(metrics): full-run OSL over all agentic turns for accuracy validation

[arekay-nv]

Summary

Resolves #500.

Agentic-inference runs stop performance tracking when the first active user finishes its final trajectory, so turns issued after that boundary are excluded from the windowed output-sequence-length (OSL). Because the excluded tail depends on concurrency, the reported mean OSL differs across Pareto points even when the model produces identical output — which biases the per-turn mean OSL accuracy requirement (see the Tail Management section).

This PR adds a separate full-run OSL computed over every completed turn — tail and dataset repeats included — for use as the accuracy-gate statistic, while leaving the windowed OSL untouched for throughput.

What changed

• commands/benchmark/full_run_osl.py (new): a post-finalize, streaming, prefix-gated scan of events.jsonl that rolls up OSL over all in-population COMPLETE turns. The population is bounded to the performance phase via sample_idx_map.json and fails closed (skips) when that map is unavailable, so accuracy-phase turns can never be counted as performance OSL. Reports n_turns_counted / n_empty / n_errors / n_undecodable so a partial or all-blank run stays visible rather than looking uncomputed.
• metrics_aggregator/tokenization.py: output_tokenization_input — the single shared rule for how a model output (including reasoning and tool calls) is rendered for counting. Both the windowed OslTrigger and the full-run scan use it, so the two numbers are directly comparable.
• metrics_aggregator/token_metrics.py: BatchTokenizer.count_sync for finalize-side (post-event-loop) counting that matches the async path exactly; teardown is guarded so the in-process (n_workers=0) tokenizer never terminates unrelated child processes.
• commands/benchmark/execute.py: computes and attaches the block at finalize, inside the report-writing try/finally (skipped on abort so a partial tail is never reported as complete).
• metrics/report.py: new optional output_sequence_lengths_full_run field, plus a report.txt line labeled as the accuracy OSL (with an explicit note when it is missing or the run is incomplete).
• examples/10_Agentic_Inference/README.md: documents output_sequence_lengths_full_run.output_sequence_lengths.avg as the field to read for the OSL accuracy gate — a run is invalid for the gate when this is null, n_turns_counted == 0, or the run is not complete.

Testing

uv run pytest -m unit \
  tests/unit/commands/test_full_run_osl.py \
  tests/unit/metrics/test_report_builder.py \
  tests/unit/async_utils/services/metrics_aggregator/test_output_tokenization_rule.py

Includes the issue #500 (f) regression guard: the full-run mean stays identical when the performance window closes at different points, even as the windowed mean moves. (Pre-existing macOS-only IPC-socket / CPU-affinity test failures are unrelated to this change.)

Notes

• The single Sync commit can be squashed/renamed on merge.
• Draft: opening for review. A few optional test-hardening follow-ups (an isolation test for a raising output_tokenization_input, a direct test of the tokenizer-teardown guard) are not yet included.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.54962% with 15 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@47cc5c8). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...erence_endpoint/commands/benchmark/full_run_osl.py	87.77%	11 Missing ⚠️
...utils/services/metrics_aggregator/token_metrics.py	71.42%	4 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #504   +/-   ##
=======================================
  Coverage        ?   80.88%           
=======================================
  Files           ?      154           
  Lines           ?    21194           
  Branches        ?        0           
=======================================
  Hits            ?    17143           
  Misses          ?     4051           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leopck]

tested with high concurrency runs previously was heavily impacted after this PR fix, my mean OSL is stable on high concurrency and matching with low concurrency runs