Skip to content

Add FMEA scenarios to benchmark suite - #541

Merged
DhavalRepo18 merged 1 commit into
mainfrom
add-fmea-scenarios
Sep 13, 2026
Merged

DhavalRepo18 merged 1 commit into
mainfrom
add-fmea-scenarios

Conversation

@ShuxinLin

Copy link
Copy Markdown
Collaborator

Description

Adds FMEA as a first-class scenario-suite category and includes scenarios 9001 through 9033 (odd IDs) in the all profile.

Type of Change

  • New Benchmark Scenario (Industry/Asset type)
  • Evaluation Metric / Scorer
  • Agentic Orchestration Logic (ReAct, Plan-Execute, etc.)
  • Infrastructure / Tooling Improvement

Industry Relevance

Makes the FMEA benchmark scenarios selectable through fmea_all and includes them in full-suite benchmark runs.

Related Issues

  • None

Testing & Validation

  • Unit Tests: uv run pytest src/benchmark/tests/test_scenario_suite_runner.py -q (44 passed)
  • Scenario Validation: verified fmea_all resolves the 17 requested IDs and all resolves 232 IDs.
  • Data Integrity: no scenario data or sensitive industrial data was added.

Checklist

  • My code follows the project formatting conventions; git diff --check passes.
  • I have performed a self-review of my code.
  • I have updated the documentation accordingly.
  • I have signed off my commit (DCO).

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@DhavalRepo18
DhavalRepo18 self-requested a review September 13, 2026 01:19
@DhavalRepo18
DhavalRepo18 merged commit 5a07a74 into main Sep 13, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants