fix(serve): close serve races and translate GPU selection through visibility masks (EAI-7194)

[r0x0r]

Summary

Closes the concurrency races in rocm serve and completes the visible-ordinal-correctness half of EAI-7194, so GPU selection is honest under an active visibility mask. The branch is two focused commits:

1. 66e6ee5 — close daemon-autostart, GPU-select, and managed-launch races. Adds a FileLock primitive plus two lock files so concurrent serves cannot autostart the daemon twice, race the GPU-selection read, or interleave the managed-launch record write.
2. ccff388 — translate GPU selection through active visibility masks. serve GPU selection and --gpu validation are now interpreted against the devices visible under the active HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES mask instead of the raw amd-smi ordinal space.

What changed (visible-ordinal correctness)

• serve() computes the visible device set once via usable_amd_gpu_indices() and threads it through resolve_gpu_indices, reusing it for both the no-usable-GPU fail-fast and selection.
• Auto-select restricts candidates to the visible set; when every visible device is busy it falls back to the lowest visible ordinal instead of a hardcoded 0.
• An explicit --gpu N that is in range but hidden by the active mask is now rejected up front with an actionable message, rather than being silently reinterpreted by the runtime onto a different device.
• The ROCR advisory is narrowed to describe the mask-relative interpretation, and the launch-lock critical-section comment is corrected.

Deferred (intentional)

Full ROCR physical→HIP export translation is not included. That is a cross-engine contract change (rocm-engine-protocol::apply_gpu_visibility and every engine) on the GPU-required path that cannot be verified without multi-GPU hardware under an active ROCR mask, where a wrong translation would misroute serves to the wrong device. The narrowed advisory remains until that work can be done and verified on hardware.

Testing

• New unit tests: mask-aware auto-select (candidate restriction, lowest-visible fallback, all-masked → none) and --gpu mask rejection.
• Launch-lock regression test now drives the real resolve_gpu_indices entry point serve() uses.
• New e2e Scenario 15 asserts a mask-hidden --gpu is refused. It is exercised on the multi-GPU e2e-gpu lane (on single-GPU lanes the same request is out of range — both are honest refusals); the no-GPU mock lane skips it via @requires-gpu.
• Local gates: cargo test -p rocm (458/458 serial), cargo clippy --workspace --all-targets -D warnings clean, scripts/smoke_local.py ok.

GPU-hardware verification of Scenario 15 will run on the e2e-gpu lane in CI.

[michaelroy-amd]

Current head e6fb3bd967987182fc72b6f14cc016eed83c2d42 is CONFLICTING/DIRTY, so the existing green checks do not validate the merged tree. Please rebase onto current main, resolve the serve/runtime conflicts, then rerun the required suite plus the multi-GPU visibility-mask coverage before re-requesting review. The physical-to-HIP ROCR_VISIBLE_DEVICES translation remains explicitly deferred and should stay called out.

[r0x0r]

Rebased onto current main (now 2e7bf37) — the branch is MERGEABLE again.

Conflicts resolved (3 files, all in the GPU-mask commit):

• apps/rocm/src/main.rs — the no-usable-GPU fail-fast now keeps both sides: main's new scripted_backend_failure bypass and this branch's single reused visible_gpu_indices (so selection and the fail-fast still share one mask-aware probe).
• tests/e2e-cucumber/features/model_serving.feature — kept main's new scenarios 15 (serve-runtime-and-env-selectors-conflict) and 16 (serve-lemonade-preparation-recovery); this branch's masked-index scenario was renumbered to 17 (@id:serve-masked-gpu-index-rejected) to avoid a number collision.
• tests/e2e-cucumber/tests/e2e/serving_steps.rs — kept all step defs from both sides (user_serves_public_bind_no_optin, user_serves_runtime_and_env, the two #[then] selector/public-bind asserts, plus this branch's user_serves_masked_gpu_index).

Re-ran on the merged tree (local, macOS):

• cargo test -p rocm --bin rocm serve — 55 passed
• the multi-GPU visibility-mask coverage — auto_selection* (7), validate_pinned_gpu* (3), launch_lock* (1) all pass
• cargo build -p rocm and cargo test -p e2e-cucumber --no-run both green

The physical-to-HIP ROCR_VISIBLE_DEVICES translation remains explicitly deferred and is still called out as such in the commit body. The @requires-gpu masked-index scenario (17) runs on the GPU lane; the mask unit tests give the merged behaviour deterministic no-GPU coverage. Leak scan clean; all three commits signed + DCO. Re-requesting review.