Skip to content

feat(report): add events.jsonl SHA-256 to result_summary.json - #490

Open
arekay-nv wants to merge 1 commit into
mlcommons:mainfrom
arekay-nv:arekay/events_jsonl_sha_logging
Open

arekay-nv wants to merge 1 commit into
mlcommons:mainfrom
arekay-nv:arekay/events_jsonl_sha_logging

Conversation

@arekay-nv

Copy link
Copy Markdown
Collaborator

Summary

Adds a content SHA-256 of the run's events.jsonl to the benchmark report so a
run's raw event log is fingerprinted and verifiable.

  • New field events_sha256 on the Report struct → serialized into
    performance/result_summary.json (alongside qps/tps/run_config).
  • Content-only digest: hashes the file bytes only (streamed in 1 MiB chunks
    via the new utils/hashing.py::sha256_file), so copying or renaming the log
    never changes the value, and large event logs are not read into memory at once.
  • null when no event log was produced (e.g. SIGKILL before the tmpfs salvage),
    so finalization never crashes on a missing log.
  • Attached in _write_report_artifacts after the log is salvaged to
    report_dir, mirroring how per-dataset accuracy is attached post-hoc — keeping
    Report.from_snapshot a pure snapshot→Report function.

Changed files

File Change
utils/hashing.py New sha256_file(path, *, chunk_size) util (mirrors atomic_write.py)
metrics/report.py New events_sha256: str | None field on Report
commands/benchmark/execute.py Compute + attach digest in _write_report_artifacts
tests/unit/utils/test_hashing.py Unit tests for the helper
tests/unit/metrics/test_report_builder.py to_json round-trips the field

Testing

  • sha256_file: content-only (copy/rename → same digest, matches plain
    hashlib), differs on content change, missing file → None, large streamed
    input.
  • Report.to_json carries events_sha256 and defaults to null.
  • Full report_builder + benchmark + benchmark_final_snapshot suites pass
    (no regressions); ruff/ruff-format/license headers clean.

🤖 Generated with Claude Code

…lt_summary.json

Add `events_sha256` to the Report struct so `performance/result_summary.json` carries a SHA-256 over the bytes of the run's events.jsonl. The digest is content-only (streamed via the new `utils.hashing.sha256_file` helper), so copying or renaming the log never changes it; it is null when no event log was produced (e.g. SIGKILL before salvage). Attached in `_write_report_artifacts` after the log is salvaged to report_dir, mirroring how accuracy is attached post-hoc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@arekay-nv
arekay-nv requested a review from a team August 31, 2026 20:40
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@github-actions github-actions Bot added the size/normal PR Review Policy: <=500 non-test lines & <=20 files label Aug 31, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@73981a8). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #490   +/-   ##
=======================================
  Coverage        ?   81.27%           
=======================================
  Files           ?      152           
  Lines           ?    20473           
  Branches        ?        0           
=======================================
  Hits            ?    16639           
  Misses          ?     3834           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/normal PR Review Policy: <=500 non-test lines & <=20 files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants