fix(install): always keep the SDK's build of the torch an engine pins

[tomastola]

The symptom

A fresh rocm install sdk on a gfx94X host produces a managed runtime that cannot serve. The install exits 0 and every health surface reports ready, but the first serve dies with:

RuntimeError: Failed to infer device type, please set the environment variable
VLLM_LOGGING_LEVEL=DEBUG to turn on verbose logging to help debug the issue.

Earlier in the same log, more plainly: Triton is installed but 0 active driver(s) found (expected 1).

What is actually wrong

Two installers write torch into the same environment. install sdk writes TheRock's build; the engine install that follows writes the build from its own index, pinned to an exact version, replacing it:

SDK phase:  + torch==2.11.0+rocm7.13.0     (hip 7.13.99004)
end state:    torch-2.11.0+gitd0c8b1f      (hip 7.2.53211)
              vllm-0.26.0+rocm723

That torch imports cleanly against the installed SDK and then enumerates no devices, so vLLM resolves UnspecifiedPlatform and refuses to start.

Measured by swapping only torch — same environment, SDK, driver and vLLM:

torch	device_count()	serve
2.11.0+gitd0c8b1f	0	fails
2.11.0+rocm7.13.0	8	serves a real completion

vLLM itself is fine. The served response carried system_fingerprint: vllm-0.26.0-d391b761, so VLLM_PINNED_SPEC does not need to change.

Why neither obvious fix is correct

Both directions have been observed failing:

installed by SDK	engine's pin	letting which side win breaks it
2.9.1+rocm7.14.0a…	2.10.0+git8514f05	the SDK's — a torch release the engine does not accept (the case #264 fixed)
2.11.0+rocm7.13.0	2.11.0+gitd0c8b1f	the engine's — a torch that cannot open a device

So "the engine's pin always wins" and "the SDK's build always wins" each re-create the other bug. Pinning a different vLLM release is not an option either: wheels.vllm.ai/rocm/<version>/ publishes only the rocm723 ABI for 0.23.0 through 0.27.0, so working and broken environments alike run a rocm723 build. The variable is the torch each release pins, not the ABI tag.

The rule

Take the torch release from the engine's pin, and the build from the SDK.

They answer different questions. The release is what the engine was built against; the build is what the installed libraries belong to. Applied to the table above, both rows resolve correctly, and the earlier fix's guarantee is preserved rather than reverted.

Where the SDK publishes no build of that release, the engine's own is kept rather than failing a multi-gigabyte install, and the device check below reports the outcome.

Why the build comes from the manifest, not the environment

This is worth calling out because the obvious implementation is wrong.

Reading "the SDK's torch" from the environment just before the engine install seems natural, and it works — exactly once. On any later install the environment already holds the engine's build, so that reading mistakes it for the SDK's, concludes the runtime is already correct, and leaves it broken permanently. Repeat installs are the common case, not the edge case: a refresh, a re-run, or a user reaching for engines install --reinstall because something already looks wrong.

The SDK's torch is therefore recorded in the runtime manifest at install time and read back from there. Manifests written before this change fall back to deriving the build from the recorded SDK version. Both paths are covered by tests, and the fallback was exercised end to end against a runtime that was already stuck on the engine's build — it recovered.

Why a device check, and why it looks the way it does

A satisfied dependency check does not mean a usable environment. In the broken state the engine's requirements are fully satisfied — its own pinned torch is installed — and the runtime still cannot serve. That is why this went unnoticed until first serve.

Two details matter:

• device_count(), not is_available(). The broken runtime reports is_available() == True with a device count of zero. Only the count distinguishes it.
• The runtime's recorded library paths, not rocm_sdk.initialize_process(). Measured on a runtime that serves correctly: with the SDK library directories on the loader path torch reports 8 devices; with initialize_process() alone it reports 0. A probe built on the latter would condemn healthy runtimes and send people to reinstall — the operation that breaks them.

Why an install can now fail, and only then

If torch could not be settled and the runtime cannot open a device, the install fails instead of exiting 0.

The conjunction is deliberate. An alignment that could not run may still leave a working environment, and a device count of zero is the correct answer wherever no GPU is present — neither alone justifies failing a large install. Together they mean this install produced something that cannot serve, and reporting success for that is the original defect.

The error says the SDK itself installed fine and what to do next, so a transient index failure does not read as a ruined install.

This applies to install sdk too, which needed care. That command deliberately catches engine auto-install failures and downgrades them to a warning, so that a failed engine install does not throw away a multi-gigabyte SDK that installed correctly — and that is still right. But it was catching this failure as well, so the one case where the install really did produce something broken kept exiting 0, which is the reported symptom verbatim. The unusable-runtime failure is now a distinct error type, and only that one is propagated; every other engine failure still warns and keeps the successful exit. The SDK's own audit event is recorded before the command fails, because the SDK install did complete.

Why failures are not all reported the same way

An earlier draft mapped every realignment failure to "the index has no such build". That is a claim about the index, and it would have been made over DNS failures, permission errors and unreachable networks — sending the reader hunting for a wheel that exists. Unavailability is now claimed only when the resolver actually said so; everything else reports the error verbatim. The classifier is deliberately narrow, so an unrecognised message degrades to the honest answer rather than a confident wrong one.

Why every install path, not just the one

The alignment runs wherever an engine is installed into a managed runtime: the auto-install after install sdk, rocm engines install, the lazy install before serving, and engine environment resolution. Covering only the first leaves rocm engines install <engine> --reinstall able to re-break a runtime — and that is precisely the command someone runs when a runtime already looks wrong.

Environments rocm-cli does not own are left untouched, and anyone who wants a different torch can still install it into the environment directly.

What the output looks like

  torch_alignment: realigned
    2.11.0+gitd0c8b1f -> 2.11.0+rocm7.13.0 (the SDK's build of the release the engine pins)
  dependency_check: expected_divergence
  divergence: The package `vllm` requires `torch==2.11.0+gitd0c8b1f`, but `2.11.0+rocm7.13.0` is installed
  action: none; the SDK's build is intended here (see torch_alignment above)
  device_check: usable (8 device(s), torch 2.11.0+rocm7.13.0)

The divergence is real and intended: the engine's exact pin cannot express "same release, the SDK's build". It is reported as a divergence rather than a violation specifically so the remedy for a genuine violation — reinstalling the engine — is not suggested here. Any unmet requirement on any other package still reports as a violation.

How this was verified

On real hardware (MI300X, gfx942, 8 devices)

The root cause was established by a single-variable swap rather than inference: same environment, same SDK, same driver, same vLLM, changing only which torch was installed. That is the table under "What is actually wrong" — 0 devices and a failed serve on one, 8 devices and a real completion on the other.

Each of these was then run end to end against real GPUs:

Scenario	Result
Cold install sdk on a clean runtime	torch realigned, device_check: usable (8 devices), serves a real completion
engines install --reinstall afterwards	re-settles torch instead of re-breaking the runtime
A runtime already stuck on the engine's build	recovers, exercising the pre-change-manifest fallback
SDK index unreachable during rocm engines install	fails with the intended error instead of exiting 0
The updated nightly e2e scenario	passes

The third row matters because it is the state real machines are already in — the fix has to repair them, not just avoid creating new ones.

Unit tests

18 tests, covering the decision surface rather than one happy path:

• Both field failures from the table — the engine's build replaced by the SDK's build of the same release, and the SDK's release corrected to the one the engine pins. Getting either backwards re-creates the other bug, so both directions are pinned.
• The cases where nothing should happen — an already-correct runtime, a loose (non-pinned) requirement, an unidentified SDK build. A fix that acts when it shouldn't is as bad as one that doesn't act.
• A runtime already on the engine's build is still corrected, which is what makes the repair path above hold.
• Failure reporting — that a missing wheel and a DNS or permission failure do not get described the same way.
• The divergence/violation split, in both directions: expected on the realigned package, still a violation on any other.
• The device check's three outcomes, including "not verified" being distinct from "healthy".
• The exit code, through the command's real completion path rather than the predicate behind it.

That last one is worth explaining. install sdk now takes its engine auto-install as a parameter, so a test can drive both outcomes and assert what the command actually returns, without a multi-gigabyte install behind it. One test pins that an unusable runtime fails it (and that the SDK's own audit record is still written first), one that an ordinary engine-install failure does not, and one that the distinction survives being wrapped in context on the way up — flattening it into a plain message is how this would quietly regress.

To confirm that test is load-bearing rather than merely green, the catch was reverted to warning-and-continue and the test was observed to fail. A regression test that cannot fail is decoration.

End-to-end

The nightly scenario covering a second SDK install asserted the wording of the dependency check, which a settled runtime no longer produces. It now asserts the runtime can still open a GPU — the outcome it was always protecting — while still failing on any genuine unmet requirement. That is a stronger assertion than the one it replaces: it is functional rather than a string match, and it is the property that actually broke here.

Static checks

cargo test --workspace --all-targets (500 passing in the CLI crate), cargo clippy --locked --workspace --all-targets -- -D warnings, cargo fmt --check, and the pre-commit hook set all clean.

What is not covered

The install sdk exit code is covered by test rather than on hardware. Reproducing it end to end needs the index reachable for the SDK download and unreachable for the realignment immediately after, and both use the same index — so the honest statement is that the code path is unit-tested and the equivalent path through rocm engines install was confirmed on hardware.

Known limitation

Only torch is realigned. torchvision, torchaudio and triton are left as the engine's install leaves them. That combination served correctly in testing, but the SDK resolves those four as a coordinated set, so the mixed stack has not been validated against the full matrix of supported combinations. Worth a follow-up if the engine's pins and the SDK's ever diverge further than a build identifier.

---

• If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (No xfail row covered this; the affected scenario is a passing GPU scenario that this PR strengthens.)