Question: format for a LIBERO reproduction report with evidence caveats

[yoshihyoda]

Description

Hi vla-eval team,

I read the contributing guide and saw that reproduction reports are especially useful here.

I have been running a small evidence-packaging workflow for public VLA evaluation outputs, and I wanted to ask before opening a PR.

The two runs I can currently summarize are:

• MolmoAct2 on Standard LIBERO

  • recomputed success rate: 98.05%
  • public reference: 98.25%
  • delta: -0.20pt
  • 2000 planned episodes, 1961 successes

• OpenPI pi0.5 on Standard LIBERO

  • recomputed success rate: 96.75%
  • public reference: 96.85%
  • delta: about -0.10pt
  • 2000 planned episodes, 1935 successes

I want to be careful about scope: these are evidence-packaged recomputation results, not proposed official leaderboard scores.

I’m also tracking a separate LIBERO-Pro companion readout for the same models, but I would keep that clearly separated from Standard LIBERO because it is a different benchmark family. I would not include it in an initial report unless maintainers think it is useful as a separate appendix.

I’m not trying to add leaderboard scores, claim official results, or change benchmark protocols.

What I’d like to contribute, if useful, is a small reproduction report in the style of docs/reproductions/db-cogact.md, with:

• model and benchmark reference
• run configuration summary
• denominator and success counts
• suite-level breakdown
• delta against public reference
• evidence caveats
• what should not be overclaimed
• links to public reference sources

Would a report like that be welcome here?

If yes, I can open a small docs-only PR first, and keep any WorldFlux-specific language minimal. The goal would be to contribute a useful reproduction note, not to advertise a separate tool.

If this should stay outside the repo, that is completely fine too. I can keep it as an external reproduction note and simply align the format with what maintainers think is useful.
If reproduction reports in this repo should only cover runs executed directly through vla-eval, I can keep this outside the repo until I have a vla-eval-native run to report.

[MilkClouds]

Thanks for checking before opening a PR!

Quick scope note on the reproduction reports: every one in this repo is for a run executed directly through vla-eval, and each ships with the exact server/benchmark config paths, the raw per-episode result JSON under docs/reproductions/data/, and a pipeline audit (see openpi.md for the shape). That reproducible chain is the whole point of the section, so we can't take in externally-recomputed numbers — there's no config/raw output for us to audit against.

Two specifics on your runs:

• OpenPI π₀.₅ on LIBERO is already covered in openpi.md: our vla-eval run gives 97.7% vs reported 96.9%. Your 96.75% / 96.85% figures don't match either number — exactly the kind of gap that needs a reproducible config + raw outputs to resolve.
• MolmoAct isn't integrated here — we have MolmoBot, evaluated on MolmoSpaces-Bench, not LIBERO. So there's no harness path to verify a "MolmoAct on LIBERO" number yet.

So: if you run these through vla-eval and can share the config + raw data/ JSON, a report in the existing format is very welcome. Until there's a vla-eval-native run, it's best kept as an external note. Thanks again!