feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060)

[r0x0r]

Summary

Adds the vLLM startup out-of-memory failure mode to the rocm-core diagnosis
catalog, with a matching print-only remediation recipe.

The error string covers two distinct faults, and the remediation splits on which
one it is:

• a tenancy collision — vLLM's fixed ~90% VRAM reservation runs into memory
  already in use on a shared or busy GPU; lowering the reservation is the fix, or
  steer onto a less-busy device with --gpu <index>.
• a model that genuinely does not fit — lowering the reservation only trades
  an earlier OOM for a later one; the fix is a smaller/quantized model or
  sharding across GPUs with --tensor-parallel-size <n>.

So the wording stays conditional: it never presents --gpu-memory-utilization
as the unconditional answer, and the verify step avoids the tenancy knob so it
does not unconditionally prescribe the tenancy workaround.

What changed

• crates/rocm-core/src/diagnose.rs — KEYWORDS_VLLM_OOM keyword table and
  check_16_vllm_oom. The match is keyword-only: an Examination carries no
  per-GPU VRAM or tenancy fields, so nothing structural can corroborate it — the
  user passes the error via --symptom, or arrives from the serve failure hint.
  The checker is gated to Linux and WSL2 (vLLM is Linux/WSL-only). On
  WSL2 this means a qualifying keyword match now produces a normal
  diagnosis instead of the platform's previous unconditional "out of
  scope" routing -- an exit-code/output contract change worth calling out
  explicitly here, not just in code comments. A required vLLM anchor (the
  word vllm, or one of its distinctive flags/logs) gates the whole
  keyword table, so a bare framework OOM string (e.g. torch.OutOfMemoryError: CUDA out of memory from an unrelated PyTorch job) is not misattributed
  to vLLM.
• crates/rocm-core/src/fix.rs — print-only fix-16-vllm-oom recipe
  (auto_applicable: false, no runner). Catalog count assertion 15 → 16 and the
  AUTO-set assertion strengthened to pin fix-16 as print-only.
• engines/vllm/src/lib.rs — the serve OOM hint points users at
  rocm diagnose --symptom '…'.
• tests/e2e-cucumber — a conditional-remediation scenario
  (@id:diagnose-vllm-oom-is-conditional, @requires-os:linux), its step
  definitions, and the catalog fix-id contract updated.

Dependencies / stacking

• Stacked on feat(vllm): Tackle out of memory errors (EAI-8058) #251 (gpu-out-of-memory, EAI-8058) — this PR targets that
  branch, not main, and stays in draft until feat(vllm): Tackle out of memory errors (EAI-8058) #251 merges upstream.
• Sibling sub-task: feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 (EAI-8059) adds the serve-summary OOM guidance that
  routes users here.

Test plan

• cargo test -p rocm-core --lib — passes (includes the checker tests: a
  high-confidence anchored OOM match, a bare "out of memory" staying below
  threshold, a bare framework OOM without a vLLM anchor not matching at all,
  Linux/WSL gating, and a WSL sub-threshold hit staying in matched instead
  of being routed out of scope).
• cargo test -p rocm-engine-vllm --lib — passes.
• cargo clippy --workspace --all-targets -- -D warnings — clean.
• e2e-cucumber crate compiles; the new @requires-os:linux scenario runs on the
  self-hosted Linux/GPU lane rather than locally.

[volen-silo]

A few observations from a read-through:

• KEYWORDS_VLLM_OOM needs nothing vLLM-specific to hit high confidence: torch.OutOfMemoryError: CUDA out of memory scores 45+45=90 under the top-2 rule, so any ROCm PyTorch job's OOM is reported as "vLLM ran the GPU out of memory at startup" with rocm serve-only remediation. The one vLLM-specific token, gpu_memory_utilization, is weighted lowest and never required — and all three new tests use vLLM-shaped text.
• The description says Linux-gated, but the checker registers for ["linux", "wsl"] and rewrites the diagnose() WSL branch so WSL2 is no longer unconditionally out-of-scope. That's an exit-code contract change worth stating in the body.
• In that branch, sub-threshold hits are dropped from matched, which the DiagnoseReport::matched doc says should never happen.
• feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 deletes the local log_tail_shows_oom this PR still calls, and edits the same oom_utilization_hint — conflict for whichever lands second.
• Body says this stays in draft until feat(vllm): Tackle out of memory errors (EAI-8058) #251 merges; it's currently open.

[r0x0r]

Thanks for the review — addressed both issues in bac9ce2 / rebased and pushed.

• Generic PyTorch OOM misattributed to vLLM: check_16_vllm_oom now requires an explicit vLLM anchor (vllm, gpu_memory_utilization/gpu-memory-utilization, or tensor_parallel/tensor-parallel) before KEYWORDS_VLLM_OOM is even scored. torch.OutOfMemoryError: CUDA out of memory on its own no longer matches at all — added generic_pytorch_oom_without_a_vllm_signal_does_not_match as a regression test for exactly that case. The vLLM engine's serve-failure hint and the matching e2e-cucumber step were updated to include the anchor ('vllm: torch.OutOfMemoryError: ...') so the self-referential rocm serve → rocm diagnose --symptom flow keeps working.
• Sub-threshold hits dropped in the WSL branch: fixed — diagnose() now keeps whatever run_all_checks returns on WSL (empty only when no wsl-applicable checker fired at all), instead of discarding nonzero sub-threshold matches in favor of the out-of-scope message. This restores the DiagnoseReport::matched contract ("every checker that fired at all lands here"). Added wsl_sub_threshold_vllm_signal_is_preserved_not_routed_out_of_scope to cover it.
• Linux/WSL exit-code contract change: confirmed and worth calling out explicitly — yes, check_16_vllm_oom being registered for ["linux", "wsl"] means WSL2 is no longer unconditionally out of scope; a keyword match on WSL now returns a normal diagnosis rather than always routing to the WSL "out of scope" message. This is intentional (vLLM does run under WSL2), but the PR body undersold it by saying "gated to Linux" — I'll tighten that wording.
• Conflict with feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 (log_tail_shows_oom / oom_utilization_hint): noted. feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284 is still open/unmerged, so nothing to resolve yet, but since this PR just touched oom_utilization_hint's suggested --symptom text again, whichever of feat(vllm): enhance OOM diagnostics and guidance in serve summary (eai-8059) #284/feat(diagnose): add the vLLM out-of-memory failure mode to the catalog (EAI-8060) #290 lands second will need to rebase past the other's edit to that function.
• Draft status: this PR is out of draft already (not something I changed); base gpu-out-of-memory (feat(vllm): Tackle out of memory errors (EAI-8058) #251) has since merged its own follow-up fix (fix(vllm): validate --gpu against sysfs GPU count and harden DRM fallback), and this branch is now rebased on top of that tip.

All of cargo test -p rocm-core --lib, cargo test -p rocm-engine-vllm --lib, cargo clippy -p rocm-core -p rocm-engine-vllm --all-targets -- -D warnings, and cargo check -p e2e-cucumber --tests pass locally after the rebase.

[volen-silo]

Round 2. The sub-threshold/matched fix is clean and the regression test targets the contract rather than the implementation — good. The #284 conflict I flagged is refuted: git merge-tree --write-tree --merge-base=a97a39f0 HEAD origin/eai-8059-oom-memory-knobs-note exits 0, the two branches edit the if condition and the format! body respectively. No rebase needed.

Two blocking:

• rocm serve --tensor-parallel-size doesn't exist. fix.rs:379 and diagnose.rs:1343/:1355 all tell the user to run it. The Serve clap variant has no such field and there's no trailing-var-arg or engine-arg passthrough — the only injection point into vLLM's argv is EngineRecipeHint.required_flags, which nothing populates with it. Clap rejects the command. It also contradicts docs/vllm.md:90 ("Serving one model across multiple GPUs is not supported", enforced at main.rs:16687 and rocm-engine-protocol/src/lib.rs:840). So "Case 2 — shard it across more GPUs" isn't actionable through this CLI in any form. diagnose_steps.rs:379 only asserts the literal string --tensor-parallel-size appears in the rendered fix, so it can never catch this. --gpu-memory-utilization and --gpu <index> are both genuine — only Case 2 is fabricated.

• The tensor[-_]parallel anchor alternative reopens the round-1 false-positive class at full confidence. vllm and gpu[-_]memory[-_]utilization are genuinely vLLM-specific; tensor_parallel is standard vocabulary in Megatron-LM, DeepSpeed, torch.distributed.tensor.parallel, HF Accelerate, SGLang and TensorRT-LLM. Hand-simulating the anchor plus the top-2 rule against the actual table:

  Initializing tensor_parallel_size=8 ... torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
    anchor hit, score 90 (torch.outofmemoryerror 45 + cuda out of memory 45) -> HIGH_CONFIDENCE
  
  megatron: tensor-parallel rank 3: HIP out of memory. Tried to allocate 2.00 GiB
    anchor hit, score 75 (hip out of memory 45 + tried to allocate...gib 30) -> HIGH_CONFIDENCE
  

  A Megatron/DeepSpeed training crash gets "vLLM ran the GPU out of memory at startup" with rocm serve-only remediation. Dropping that alternative closes it — I couldn't construct a realistic non-vLLM symptom that clears threshold via the bare vllm substring alone.

Smaller things:

• a_bare_out_of_memory_stays_below_the_match_threshold guards with if let Some(d) — if the checker ever stops firing on that input, the body never runs and the test passes asserting nothing. Its two siblings correctly .expect(...) first.
• The gpu_memory_utilization keyword row is underscores-only while the anchor accepts gpu[-_]memory[-_]utilization. A symptom using the hyphenated CLI spelling — the spelling this PR's own recipe prints — passes the anchor and scores 0 on that row.
• diagnose_steps.rs:350 hardcodes score >= 75; the report publishes high_confidence_threshold, and assert_verdict_follows_scores in the same file reads min_score_for_match off the report with a comment explaining why hardcoding is wrong.
• The two new Then steps repeat the same find-by-id block, and :374 does fix.to_string() + four .contains() over the whole serialized object while the next assertion correctly scopes to fix.get("verify").
• The suggested --symptom in engines/vllm/src/lib.rs:1833 is a canned literal engineered to score 90 against the new table, while the function already holds the real log_tail. As written the round-trip demonstrates the matcher against its own fixture rather than routing the user's actual error.
• Body says "exit-code contract change" — main.rs:2198 states and implements that diagnose exits 0 whether it matched, found nothing, or is out of scope. The output contract change on WSL is real; the exit code isn't.
• Commit 6097efbc's message still says "gated to Linux" / "linux-only"; the body was corrected but the message wasn't.
• AGENTS.md §11 wants stacked PRs in draft until dependencies merge; feat(vllm): Tackle out of memory errors (EAI-8058) #251 is still open and this is out of draft, while the body claims it "stays in draft until feat(vllm): Tackle out of memory errors (EAI-8058) #251 merges".
• docs/vllm.md:135 describes the serve-failure message this PR extends, without the new --symptom pointer or the catalog entry.
• tests/e2e-cucumber/src/expectation.rs:65 justifies @requires-bare-metal with "a scenario needing a catalog match has no premise" on WSL2 — now stale.
• "torch.outofmemoryerror" leaves . unescaped; every other literal-dot pattern in the file is a raw string with \..

One worth confirming as deliberate: on WSL, any nonzero fix-16 score (a bare "vllm ... out of memory" scores 25) now sets out_of_scope to None, so the renderer prints a WEAK entry with apply with: rocm fix fix-16-vllm-oom and the /dev/dxg+WSL-docs routing message disappears. Keeping weak hits in matched is right; coupling out_of_scope to matched.is_empty() rather than to "nothing cleared MIN_SCORE_FOR_MATCH" is the part I'd double-check.

@requires-os:linux is the right tag here, not @requires-bare-metal — WSL2 reports os_family: "linux", so the scenario covers Linux and WSL and skips only native Windows.

[r0x0r]

Thanks for the Round-2 review — pushed d233af7 addressing it.

rocm serve --tensor-parallel-size doesn't exist / contradicts single-GPU serving. Removed the sharding remediation from both surfaces (fix-16-vllm-oom in fix.rs and the check_16_vllm_oom summary/commands in diagnose.rs). The flag doesn't exist and docs/vllm.md states one model is served on a single GPU. The "model genuinely does not fit" branch now points only at a smaller/quantized model.

tensor[-_]parallel anchor reopened false positives. Dropped that alternative from VLLM_ANCHOR_PATTERN; it's a Megatron/DeepSpeed term, so anchoring on it would misattribute those frameworks' OOMs to vLLM. The anchor is now vllm|gpu[-_]memory[-_]utilization (comment updated to explain the omission).

Smaller items:

• a_bare_out_of_memory_stays_below_the_match_threshold now uses .expect() instead of a vacuous if let Some (the weak out of memory hit is always present in matched).
• The gpu_memory_utilization keyword now accepts a hyphen (r"gpu[-_]memory[-_]utilization") to match the anchor.
• diagnose_steps.rs reads high_confidence_threshold from the report instead of a hardcoded 75, and the two new Then steps share a find_vllm_oom helper.
• The --symptom hint in engines/vllm now routes the user's actual failing log line (vllm-anchored) instead of a canned literal.
• Escaped the dot in the torch\.outofmemoryerror keyword regex.
• docs/vllm.md now points at rocm diagnose --symptom '<the error>'; the @requires-bare-metal doc-comment notes the WSL keyword-only exception (the vLLM OOM entry does run on WSL, so it belongs under @requires-os:linux).

Two notes:

• The change is to the output contract, not an exit-code one — rocm diagnose still always exits 0. I've corrected the body wording accordingly.
• One item I deliberately left as-is (want your call): the WSL branch in diagnose() sets out_of_scope from wsl_matches.is_empty() rather than keying off MIN_SCORE_FOR_MATCH. The current code keeps sub-threshold hits in matched (per the DiagnoseReport::matched contract) with out_of_scope = None; keying off the threshold would make a sub-threshold-only WSL result both populate matched and carry the out-of-scope message. Happy to flip it if you'd rather.

[volen-silo]

LGTM.

Round 3. Both round-2 blockers are genuinely fixed, not just claimed: --tensor-parallel-size is gone from the recipe, the diagnose summary and the commands, with a negative e2e assertion locking it out; the anchor is down to vllm|gpu[-_]memory[-_]utilization, and re-simulating the Megatron/DeepSpeed cases confirms they now fail the anchor before the table is scored at all. I re-checked the surviving prescriptions against the clap definitions (--gpu-memory-utilization, --gpu <index> both real), re-derived scores by hand for the anchored / unanchored / sub-threshold cases, confirmed applies_on: LINUX_ONLY still lets rocm fix fix-16-vllm-oom run under WSL2 (current_os() has no wsl concept), and ran fmt, clippy, rocm-core, rocm-engine-vllm and cargo check -p e2e-cucumber --tests — all clean.

Your open question — out_of_scope keyed off wsl_matches.is_empty() rather than MIN_SCORE_FOR_MATCH: keep it as-is. On WSL only fix-16 can fire, and it cannot fire without the vLLM anchor, so the only symptom that suppresses the /dev/dxg routing is one that literally names vLLM or its reservation flag. A topical WEAK entry beats printing both that entry and "out of scope for this platform", which would contradict itself.

Nits (non-blocking)

• PR body still reads "exit-code/output contract change". rocm diagnose always exits 0 (main.rs), so only the output contract changed — and since this repo squash-merges, the body becomes the permanent commit message. Commit 6097efbc's body still says "gated to Linux" / "(linux-only)" against its own &["linux", "wsl"] registration, though that one won't survive the squash.
• oom_utilization_hint interpolates the raw log line into --symptom '<line>' with no quote escaping; an apostrophe breaks the copy-paste. Display-only, nothing is executed, and lines are uncapped in length.
• A line matching only the bare out of memory phrase (e.g. RuntimeError: HIP error: out of memory) scores 25, so the command the serve hint just handed the user comes back WEAK. render_report_text still prints the whole remediation for sub-threshold entries, so it is a mislabeled tier rather than a lost answer — but the round trip the hint advertises is not guaranteed by the keyword table.
• torch\.outofmemoryerror does not match the older fully-qualified torch.cuda.OutOfMemoryError; harmless today because hip out of memory + tried to allocate ... GiB still reaches 75.
• .unwrap_or("out of memory") in oom_utilization_hint is unreachable: a literal containing no newline cannot span a lines() boundary, so the guard above it already guarantees a hit.
• Recipe examples say --gpu-memory-utilization 0.5 while the shared hint constant and docs/vllm.md use 0.1.
• Still out of draft while its base (#251) is open, which AGENTS.md §11 asks against.