[KimiK3][AgentX]: H200 KimiK3 Day 0 support

[cquil11]

Summary

• Add Kimi K3 MXFP4 AgentX Day 0 performance coverage on H200, based on the official vLLM recipe.
• Keep benchmark code scheduler-independent: the container entrypoint translates logical topology into vLLM flags, while the H200 runner owns the atomic Slurm allocation, mounts, endpoints, readiness, cleanup, and server.log capture.
• Keep the model default context length; no --max-model-len override is used.

Orientation	Topology	Nodes	AgentX concurrency
Latency	TP8 / PP2	2	1, 2, 4, 8
Balanced	TP4 / PP2 / DP2 / EP8	2	4, 8, 16, 32
Throughput	TP1 / PP2 / DP16 / EP16	4	16, 32, 64, 128

Validation

• Matrix tests: 224 passed.
• Latency targeted run: passed.
• Balanced targeted run: passed.
• Latency c8 + balanced c8: passed.
• Balanced c32: passed cache-pressure and profile phases with zero server errors.
• Throughput c128: sustained the 142-request burst for 600 seconds with zero hard errors, then was short-circuited after server.log upload to release nodes.
• Official full sweep: running.
• Config generation, bash -n, git diff --check, and matrix tests: passed.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

Claude finished @cquil11's task in 8m 0s —— View job

---

Review of PR #2353

• Gather context: read PR diff and changed files
• Validate master config (image accessibility, model prefix, perf-changelog position)
• Review connector, launch script, and benchmark script for bugs
• Post inline comments and summary

Found 1 blocking issue and 1 warning (see inline comments):

1. 🔴 benchmarks/multi_node/agentic/kimik3_fp4_vllm.sh — the throughput (DEP16) orientation uses --data-parallel-hybrid-lb, where each node's API server only balances across its local DP ranks, but the connector and AgentX client send all traffic to the head node only. Node 1's 8 engines would receive zero requests, so the throughput results would silently measure 8 GPUs while allocating 16. Suggested fix: internal LB mode (--headless + --data-parallel-start-rank on rank > 0, drop --data-parallel-hybrid-lb).
2. 🟡 runners/connectors/slurm_pyxis.sh — the 240-minute default allocation wall time conflicts with the config's 120-minute startup budget plus a 4-point agentic sweep; Slurm would kill the job mid-benchmark. Suggest setting MULTINODE_TIME_LIMIT_MINUTES in the config.

Checks that passed: perf-changelog.yaml entry is present and appended at the end of the file; vllm/vllm-openai:kimi-k3 is a public Docker Hub reference; the connector includes the enroot import docker:// reproducibility pattern; --enable-expert-parallel is correctly gated on EP > 1; the kimik3 model prefix is already supported on main; server launch args are multi-line; result files land where the multinode workflow collects them (/workspace ↔ $GITHUB_WORKSPACE, LOGS/agentic).

Note: I could not verify the day-0 image's support for vllm serve --nnodes/--node-rank/--master-addr (latency/balanced orientations) since the vLLM source MCP server and network access were unavailable in this run; those flags match the stated official recipe, so I did not flag them.

[claude]

🔴 BLOCKING: In --data-parallel-hybrid-lb mode, each node runs its own API server that load-balances only across its local DP ranks — vLLM expects an external load balancer in front of the per-node endpoints. But this deployment has no external LB: the connector health-checks and the AgentX client (agentic_srt.sh) send all traffic to the head node (MULTINODE_MASTER_ADDR:$PORT) only.

Why it matters: Node 1's 8 DP engines receive zero requests for the entire throughput sweep (they just execute dummy batches to keep the EP all-to-all in sync). The DEP16 "throughput" results would actually measure an 8-GPU deployment while 16 GPUs are allocated — silently wrong benchmark numbers, roughly half the real throughput.

Fix: Use internal load-balancing mode instead — a single API server on the head node that dispatches to all 16 DP engines, with secondary nodes headless:

Suggested change

	if [[ "$PREFILL_DP_ATTN" == "true" ]]; then
	DEPLOYMENT_ORIENTATION="throughput"
	VLLM_CMD+=(
	--data-parallel-size "$WORLD_SIZE"
	--data-parallel-size-local "$GPUS_PER_NODE"
	--data-parallel-address "$MASTER_ADDR"
	--data-parallel-start-rank "$((NODE_RANK * GPUS_PER_NODE))"
	--data-parallel-hybrid-lb
	)
	if [ "$PREFILL_EP" -gt 1 ]; then
	VLLM_CMD+=(--enable-expert-parallel)
	fi
	if [[ "$PREFILL_DP_ATTN" == "true" ]]; then
	DEPLOYMENT_ORIENTATION="throughput"
	VLLM_CMD+=(
	--data-parallel-size "$WORLD_SIZE"
	--data-parallel-size-local "$GPUS_PER_NODE"
	--data-parallel-address "$MASTER_ADDR"
	)
	if [ "$NODE_RANK" -gt 0 ]; then
	VLLM_CMD+=(
	--data-parallel-start-rank "$((NODE_RANK * GPUS_PER_NODE))"
	--headless
	)
	fi
	if [ "$PREFILL_EP" -gt 1 ]; then
	VLLM_CMD+=(--enable-expert-parallel)
	fi

(Alternatively keep hybrid-lb and add a real LB plus per-node client traffic, but internal LB is the drop-in fix for this single-endpoint connector/client.)

[claude]

🟡 WARNING: The allocation wall time defaults to 240 minutes, but the Kimi K3 config budgets MULTINODE_STARTUP_TIMEOUT_SECONDS=7200 (120 min) for startup alone. If loading the ~1.5 TB checkpoint uses a large fraction of that budget, only ~2 h remain for the full 4-point AgentX sweep (plus warmups and drain waits at concurrency up to 128).

Why it matters: When --time expires, Slurm kills the allocation mid-benchmark — the client srun dies with an opaque failure and the whole (expensive, multi-hour) job has to be rerun.

Fix: Set MULTINODE_TIME_LIMIT_MINUTES explicitly in the kimik3-fp4-h200-vllm-agentic additional-settings (e.g. 480), or make the default here scale with the configured startup timeout instead of a flat 240.

[claude]

Additional findings (outside current diff — PR may have been updated during review):

• 🔴 benchmarks/multi_node/agentic/kimik3_fp4_vllm.sh:96-113 — The DEP16 throughput branch sets --data-parallel-hybrid-lb (kimik3_fp4_vllm.sh:102-113) without --headless, but this repo's own docs (benchmarks/multi_node/llm-d/README.md) explain hybrid-lb makes each node run an independent api-server that only load-balances its own local DP ranks — cross-node fan-out needs an external per-node router, which the Slurm/Pyxis connector doesn't provide. Since the AgentX client (agentic_srt.sh/benchmark_lib.sh:1760) only ever sends traffic to http://localhost:$PORT on the head node, the second node's 8 DP ranks (start-rank 8-15) never receive real requests, so the headline 'throughput-oriented' DEP16 sweep silently measures an 8-GPU deployment instead of 16.

  Extended reasoning...

  The bug: In the DEP/throughput branch of kimik3_fp4_vllm.sh (lines 96-113), the server is launched with --data-parallel-size 16, --data-parallel-size-local 8, --data-parallel-hybrid-lb, and critically no --headless on either node. This is meant to stand up a single cross-node DP16+EP engine spanning the two allocated H200 nodes, one api-server-facing endpoint that fans requests out to all 16 DP ranks.

  Why this doesn't do what it looks like it does: This repo's own documentation at benchmarks/multi_node/llm-d/README.md ("Why one entry per node" section) explicitly documents the semantics of --data-parallel-hybrid-lb: 'With hybrid-lb, every node runs its own api-server ... and internally load-balances its own local DP ranks', and cross-node fan-out is handled by an external router (llm-d's EPP+Envoy), which is why server.sh's add_role() deliberately emits one endpoint per node rather than one shared endpoint. In other words, hybrid-lb does not create one logical multi-node frontend — it creates N independent frontends (one per node), each of which only sees its own local ranks, and something outside vLLM must route requests across those N frontends.

  The code path that triggers it: The Slurm/Pyxis connector added in this PR (runners/connectors/slurm_pyxis.sh) provides no such external router. The client-side srun step runs with --nodelist="$head_node" only (line ~282), i.e. the AgentX client container is placed exclusively on the head node. The client entrypoint, benchmarks/multi_node/agentic_srt.sh, delegates request construction to benchmark_lib.sh's build_replay_cmd, which hardcodes REPLAY_CMD+=" --url http://localhost:$PORT" (benchmark_lib.sh:1760) — i.e. every request in the sweep is sent to the head node's local api-server. The connector's readiness poll (curl ... "http://$head_node:$PORT/health") similarly only ever checks the head node. There is no second router, no per-node endpoint list, and no code that ever contacts the second node's api-server.

  Concrete walkthrough: With WORLD_SIZE=16, GPUS_PER_NODE=8, node 0 (rank 0) starts an api-server on localhost:$PORT that hybrid-lb load-balances across its local ranks 0-7. Node 1 (rank 1) independently starts its own api-server on the same port on its own host, hybrid-lb-balancing local ranks 8-15 — but nothing ever connects a client to it. The AgentX client, running only on the head node, sends its entire concurrency sweep (16, 32, 64, 128) to http://localhost:$PORT, which is node 0's api-server. Node 0 spreads that load across its 8 local ranks; node 1's 8 ranks sit idle for the duration of the run (aside from any lockstep EP all-to-all participation with dummy/empty batches). The result: a benchmark advertised and labeled as "DEP16" / 16-GPU throughput-oriented actually measures an 8-GPU deployment, with no error, warning, or indication in the output that half the allocated GPUs never processed real traffic.

  Why nothing else catches this: The TP/latency and TEP/balanced branches in the same file correctly avoid this trap — they use a single cross-node engine with --headless set on NODE_RANK > 0, so only one real api-server (on the head node) exists and it legitimately spans both nodes via TP. The DEP branch is the odd one out: it's structurally similar to the llm-d pattern (which requires an external router) but is deployed through a connector that was only ever built for the single-api-server case.

  Suggested fix: Either (a) drop --data-parallel-hybrid-lb and instead set --headless on the non-zero node rank (mirroring the TP/TEP branches), so vLLM's default internal-LB rank-0 api-server fans requests out across all 16 DP ranks through one endpoint, or (b) if hybrid-lb is intentionally required for this topology, extend the connector to run/route through both nodes' endpoints (analogous to llm-d's EPP+Envoy) rather than sending all client traffic to a single localhost URL.

[claude]

🟡 This PR's title ("[KimiK3][AgentX]: H200 KimiK3 Day 0 support") and description are English-only, missing the mandatory Simplified Chinese translation required by AGENTS.md (title suffix / <中文标题> and a ## 中文说明 section in the body). This is a documented, explicit repo requirement, not a style preference — please add the Chinese title suffix and a 中文说明 section before merge.

Extended reasoning...

What the rule is: AGENTS.md (the single source of guidance referenced by CLAUDE.md, which states "All guidance for working with this repository lives in AGENTS.md") contains an explicit, mandatory instruction near the top of the file: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...)." The text explicitly says "This applies to every PR and every issue."

Where this PR violates it: PR #2353's title is [KimiK3][AgentX]: H200 KimiK3 Day 0 support — English only, with no / <中文标题> suffix. The description body contains only English sections (## Summary, ## Multi-node connector PoC, ## Validation) and has no ## 中文说明 (or equivalent Chinese) section anywhere. The only bilingual text in the entire PR timeline is the github-actions[bot] recipe-reminder comment, which is explicitly carved out by the rule's own exception clause ("bot-generated comments follow their own workflow templates") — so it does not satisfy the requirement.

Why nothing else in the repo enforces this automatically: the bilingual requirement is a documentation/process convention enforced by human/CODEOWNER review, not by an automated CI check (unlike, say, the PR_REVIEW_CHECKLIST sign-off phrase, which the recipe-reminder bot says is needed to "trigger" a signoff bot). There's no bot or workflow gate in this repository that rejects an English-only PR title/description, so nothing currently stops this PR from merging as-is.

Impact: this is a process/compliance gap, not a functional defect — no benchmark script, config, or runtime behavior is affected, and the six changed files (the Kimi K3 entrypoint, Slurm/Pyxis connector, H200 runner, configs, and changelog) all work correctly regardless of the PR metadata's language. It costs nothing at runtime, but it is an explicit, easily-checked repo rule that this PR does not meet.

Step-by-step proof:

1. Read AGENTS.md line 7 (loaded transitively via CLAUDE.md): the bilingual title/description rule, applying to "every PR."
2. Read PR [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353's title from the PR metadata: [KimiK3][AgentX]: H200 KimiK3 Day 0 support — no / <中文标题> suffix present.
3. Read PR [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353's description: sections are ## Summary, ## Multi-node connector PoC, ## Validation — all English, no ## 中文说明 or any Chinese text.
4. Check the PR timeline for any Chinese content elsewhere: only the bot-generated recipe-reminder comment is bilingual, and the rule explicitly exempts bot-generated comments.
5. Conclusion: the PR title and description do not satisfy the mandatory bilingual requirement in AGENTS.md.

How to fix: rename the PR title to append / <中文标题> (a natural Chinese translation of "H200 KimiK3 Day 0 support"), and add a ## 中文说明 section to the description body mirroring the English summary, following the terminology and style guidance also documented in AGENTS.md (e.g. preserve model names/SKUs/flags in English, use parenthetical English clarification for acronyms on first use).

Note: this same finding was independently raised as a duplicate (bug_006); this report is filed once against the canonical lower-ID instance.

[claude]

🟡 The agentic drain check between concurrency points (wait_for_agentic_servers_idle in agentic_srt.sh) is a silent no-op for the new Kimi K3 server: it only reads the Dynamo-specific dynamo_frontend_active_requests metric plus worker metrics from AIPERF_SERVER_METRICS_URLS, but Kimi K3 is a native vllm serve process (not a Dynamo frontend) and this config never sets AIPERF_SERVER_METRICS_URLS. Since both sums are always 0 for an absent metric, the drain reports 'idle' after ~3 quick polls (~20-30s) regardless of real in-flight/queued requests, so the between-point drain never actually functions for this deployment.

Extended reasoning...

benchmarks/multi_node/agentic_srt.sh's wait_for_agentic_servers_idle() is called between every non-final CONC_LIST point (see the loop at the bottom of the modified agentic_srt.sh) to make sure the server has fully drained before the next concurrency point starts, so that sequential AgentX points are comparable. It computes readiness as:

• frontend_active = metric_sum(fetch_metrics(frontend_url), "dynamo_frontend_active_requests"), where frontend_url = http://localhost:$PORT/metrics
• worker_active = sum of vllm:num_requests_running + vllm:num_requests_waiting over URLs supplied via AIPERF_SERVER_METRICS_URLS

Both quantities are Dynamo-oriented: dynamo_frontend_active_requests is a Dynamo-frontend-only metric, and AIPERF_SERVER_METRICS_URLS is the mechanism previously used to point at Dynamo backend workers. This drain function has, up to now, only ever run in front of Dynamo/srt-slurm deployments, whose /metrics endpoint genuinely exposes dynamo_frontend_active_requests.

The new kimik3-fp4-h200-vllm-agentic config in configs/nvidia-master.yaml is the first to wire a native (non-Dynamo) multi-node vLLM server into this same client script via MULTINODE_CLIENT_SCRIPT=benchmarks/multi_node/agentic_srt.sh. The Kimi K3 server (benchmarks/multi_node/agentic/kimik3_fp4_vllm.sh) is a plain vllm serve ... OpenAI-compatible server. Its /metrics endpoint exposes the vllm:* Prometheus family (including vllm:num_requests_running) but never dynamo_frontend_active_requests, so frontend_active is always 0.0. None of this config's three additional-settings blocks set AIPERF_SERVER_METRICS_URLS (it's only ever set on an unrelated AMD SGLang path in the repo), so worker_urls is empty and worker_active is also always 0.0.

metric_sum() returns 0.0 for a metric name it never finds in the scraped text, so the loop's exit condition frontend_active == 0 and worker_active == 0 is trivially true on every poll for this deployment. Concretely: poll 1 → both 0 → idle_polls=1; poll 2 → both 0 → idle_polls=2; poll 3 (~20-30s later, depending on AIPERF_DRAIN_POLL_SECONDS) → both 0 → idle_polls=3 → the function prints "Agentic servers remained idle for three polls" and returns, regardless of whatever real request load is actually still running or queued on the vLLM server. Note that even though the vLLM server's own metrics endpoint does expose vllm:num_requests_running, the code never reads that metric from frontend_url — only from worker_urls, which stays empty here — so the one signal that would actually work for this deployment is never consulted.

Impact: because this check between points is effectively a fixed short sleep rather than a real drain, the next concurrency point in the sweep could start while requests from the previous point are still in flight (if aiperf's own client-side lifecycle doesn't already fully settle first), which would overlap load across points and skew the AgentX metrics that this config's whole purpose is to produce. The fix is straightforward: either have the drain function also check vllm:num_requests_running/vllm:num_requests_waiting directly on frontend_url (which works for both Dynamo and native vLLM deployments), or set AIPERF_SERVER_METRICS_URLS to the Kimi K3 server's own metrics endpoint in this config's additional-settings.

I'm marking this a nit rather than blocking: aiperf's client-side replay is synchronous per concurrency point (the streaming client only returns after receiving final tokens for outstanding requests), so in practice the server is usually already idle by the time the next point starts even without this check — the drain is largely belt-and-suspenders. But the safeguard the code comments clearly intend ("makes sequential points comparable") is silently inert for this new deployment path, so it's worth fixing.

[claude]

Claude finished @cquil11's task in 6m 59s —— View job

---

Re-review: PR #2353 (restructured — connector removed, lifecycle folded into the H200 runner)

• Gather context: diff vs origin/main, read the 4 changed files
• Verify previous findings against the new structure
• Run standard config validations (image, model prefix, perf-changelog position, enroot import, EP gating)
• Post inline comments and summary

Re-review result: 1 new blocking issue, 1 carried-over blocking issue, 1 carried-over warning (see inline comments).

1. 🔴 NEW — runners/launch_h200-dgxc-slurm.sh:53: PORT is never defined in the multinode workflow env (benchmark-multinode-tmpl.yml sets CONC_LIST/CONC/DURATION but no PORT; the existing srt-slurm client path in this same file hardcodes PORT=8888 at line 486 for exactly this reason). With set -eo and no -u, export PORT="$PORT" exports an empty string, so the server gets vllm serve --port "" and crashes on both ranks — the whole new path fails on first run. Fix: export PORT="${PORT:-8888}". Fix this →
2. 🔴 Still applicable — benchmarks/multi_node/agentic/kimik3_fp4_vllm.sh:50-56: the DEP16 throughput branch still uses --data-parallel-hybrid-lb, where each node's api-server balances only its local DP ranks, but the health check and AgentX client (benchmark_lib.sh:1760) send all traffic to the head node. Node 1's 8 engines get zero real requests, so DEP16 silently benchmarks 8 GPUs while allocating 16. Drop-in fix (internal LB with --headless on rank > 0, mirroring the TP/TEP branch) is in the inline comment.
3. 🟡 Still applicable — runners/launch_h200-dgxc-slurm.sh:68: the flat --time=240 allocation conflicts with the 120-min readiness budget plus enroot import (up to 30-min lock wait) plus the 4-point sweep; Slurm would kill the job mid-benchmark.

Still-applicable minor notes from the previous review (not re-posted inline): the between-point drain check in agentic_srt.sh is a silent no-op for this native vLLM server (it only reads the Dynamo-specific dynamo_frontend_active_requests metric, and this config never sets AIPERF_SERVER_METRICS_URLS); and the PR title/body are still English-only, while AGENTS.md requires a bilingual title (/ <中文标题>) and a ## 中文说明 section.

Checks that passed in the new structure: perf-changelog.yaml entry is appended at the end of the file; vllm/vllm-openai:kimi-k3 is a public Docker Hub reference; the kimik3 model prefix already exists on main; the runner includes the enroot import docker:// reproducibility pattern with a flock guard; --enable-expert-parallel is correctly gated on PREFILL_EP > 1; server launch args are multi-line; NATIVE_MULTINODE is unset on existing dynamo paths (script has no set -u), so existing H200 multinode flows are unaffected; agentic results land where the workflow collects them (AGENTIC_OUTPUT_DIR defaults to /workspace ↔ $GITHUB_WORKSPACE, raw artifacts under LOGS/agentic); the readiness loop's post-loop attempt > 720 check is correct; and the cleanup trap reliably scancels the allocation on exit.

[claude]

🔴 BLOCKING: PORT is never defined anywhere in the native multi-node path. benchmark-multinode-tmpl.yml sets CONC_LIST, CONC, DURATION, etc. in the job env, but no PORT — which is exactly why the existing srt-slurm client path in this same file hardcodes --export=...,PORT=8888,... (line 486). Since this script runs set -eo pipefail without -u, export PORT="$PORT" silently exports an empty string.

Why it matters: The Kimi server gets vllm serve --port "" (argparse fails on both ranks, so the server exits immediately and the readiness loop aborts), and the readiness curl targets the malformed http://$head_node:/health. The entire new path fails on first run.

Fix:

Suggested change

	export PORT="$PORT"
	export PORT="${PORT:-8888}"

[claude]

🔴 BLOCKING (carried over from previous review — still present after the restructure): --data-parallel-hybrid-lb gives every node its own api-server that load-balances only its local DP ranks; cross-node fan-out requires an external router. This repo documents exactly that in benchmarks/multi_node/llm-d/README.md ("With hybrid-lb, every node runs its own api-server … and internally load-balances its own local DP ranks"), where llm-d's EPP/Envoy provides the fan-out. This deployment has no router: the runner health-checks only http://$head_node:$PORT/health, and the AgentX client runs on the head node only and sends every request to http://localhost:$PORT (benchmarks/benchmark_lib.sh:1760).

Why it matters: Node 1's 8 DP ranks never receive real traffic, so the "throughput-oriented" DEP16 sweep silently benchmarks an 8-GPU deployment while allocating 16 GPUs — wrong headline numbers with no error anywhere.

Fix: Use vLLM's internal load-balancing mode (single api-server on rank 0 dispatching to all 16 engines), mirroring the --headless pattern the TP/TEP branch below already uses:

Suggested change

	VLLM_CMD+=(
	--data-parallel-size "$WORLD_SIZE"
	--data-parallel-size-local "$MULTINODE_GPUS_PER_NODE"
	--data-parallel-address "$MULTINODE_MASTER_ADDR"
	--data-parallel-start-rank "$((MULTINODE_NODE_RANK * MULTINODE_GPUS_PER_NODE))"
	--data-parallel-hybrid-lb
	)
	VLLM_CMD+=(
	--data-parallel-size "$WORLD_SIZE"
	--data-parallel-size-local "$MULTINODE_GPUS_PER_NODE"
	--data-parallel-address "$MULTINODE_MASTER_ADDR"
	)
	if [ "$MULTINODE_NODE_RANK" -gt 0 ]; then
	VLLM_CMD+=(
	--data-parallel-start-rank "$((MULTINODE_NODE_RANK * MULTINODE_GPUS_PER_NODE))"
	--headless
	)
	fi

[claude]

🟡 WARNING (carried over from previous review, now in this file): the 240-minute wall time is tight against this path's own budgets: the readiness loop alone allows 7200 s (120 min, matching VLLM_ENGINE_READY_TIMEOUT_S=7200 for the ~1.5 TB checkpoint), the enroot import waits up to flock -w 1800 plus import time, and the remaining time must cover a 4-point AgentX sweep at concurrency up to 128 plus drain waits.

Why it matters: when --time expires, Slurm kills the allocation mid-benchmark — the client srun dies opaquely and the whole multi-hour job must be rerun.

Fix: raise the limit (e.g. --time=480) or derive it from the startup budget instead of a flat 240.

[claude]

🔴 runners/launch_h200-dgxc-slurm.sh:53 does export PORT="$PORT", a self-referential re-export that never gives PORT a literal value in the native multi-node branch. Every other launcher in the repo (including this file's own non-multinode branch) sets PORT=8888 explicitly, but nothing upstream (workflow env, config additional-settings) does so here, so PORT ends up empty and both the readiness curl and the in-container vllm serve --port invocation fail — the entire native multi-node Kimi K3 path would fail on every run.

Extended reasoning...

The bug: At runners/launch_h200-dgxc-slurm.sh:53, the new native-multinode branch does export PORT="$PORT" — a self-referential re-export with no fallback. This line sits among sibling exports that assign real literals (HF_HUB_CACHE, RESULT_DIR, etc.), strongly suggesting it was meant to be export PORT=8888 or export PORT="${PORT:-8888}" but was left as a copy-paste artifact.

Why PORT is actually empty here: Tracing every place PORT could be set before this line executes: the multi-node workflow template (.github/workflows/benchmark-multinode-tmpl.yml) never sets PORT in its env block or in the launch step's exported vars; configs/nvidia-master.yaml's new kimik3-fp4-h200-vllm-agentic additional-settings only set NATIVE_MULTINODE, VLLM_MOE_BACKEND, and the VLLM_DISABLE_* flags; and this script itself has no top-level export PORT=... (unlike launch_h200-cw.sh, launch_b200-cw.sh, launch_h200-nb.sh, and launch_mi300x-amds.sh, which all do export PORT=8888 at the very top of the file). Even this same file's pre-existing non-multinode (else) branch doesn't rely on an ambient PORT — it hardcodes --export=ALL,PORT=8888 on the srun line. So in the new branch, $PORT is genuinely unset when line 53 runs.

Why nothing catches it: The script only sets set -eo pipefail (no set -u), so referencing an unset $PORT does not error — it silently expands to the empty string, and the export makes PORT defined-but-empty going forward, which would also bypass any ${PORT:-8888}-style fallback later in the chain since the variable is no longer unset.

Impact / how it manifests: Two independent failure points from the same empty PORT: (1) the readiness loop at line 133 does curl ... "http://$head_node:$PORT/health", which becomes http://$head_node:/health" — a malformed URL with the port omitted — so curl never succeeds, the loop exhausts all 720 attempts over ~2 hours, and the script exits 1; (2) the server srunstep uses--export=ALL, propagating the empty PORT into the container, where benchmarks/multi_node/agentic/kimik3_fp4_vllm.shbuildsvllm serve ... --port "$PORT"directly (it doesn't sourcebenchmark_lib.sh, so there's no fallback default) — vLLM rejects an empty --port` argument outright. Either path means the entire native multi-node Kimi K3 job — the core feature this PR adds — fails deterministically on every invocation, for all three orientations (TP16, TEP16, DEP16).

Step-by-step proof:

1. Workflow triggers the H200 multi-node runner; the launch step exports RESULT_FILENAME, IS_MULTINODE=true, and the config's additional-settings (NATIVE_MULTINODE=1, VLLM_MOE_BACKEND=marlin, the VLLM_DISABLE_* flags) — none of these is PORT.
2. launch_h200-dgxc-slurm.sh runs with set -eo pipefail (no set -u). It enters the if [[ "$IS_MULTINODE" == "true" && "$NATIVE_MULTINODE" == "1" ]] branch.
3. Line 53: export PORT="$PORT". Since $PORT was never set anywhere upstream, this evaluates to export PORT="".
4. Line 133 (readiness loop): curl --fail "http://$head_node:$PORT/health" expands to curl --fail "http://<head_node>:/health" — a syntactically malformed URL missing the port number. curl fails every attempt.
5. Simultaneously, the server srun (with --export=ALL) passes the empty PORT="" into the container, where kimik3_fp4_vllm.sh runs vllm serve "$MODEL_PATH" ... --port "$PORT", i.e. --port "", which vLLM cannot parse as a valid port and fails to start.
6. Either the container's vLLM process dies immediately (readiness loop's kill -0 check catches this and exits 1 early), or if it somehow lingers, the readiness loop runs out all 720 attempts (~2 hours) and exits 1 anyway. The job never produces a benchmark result.

Fix: Change line 53 to export PORT=8888 (matching every sibling launcher) or export PORT="${PORT:-8888}" if intentional override support is desired.

[claude]

🟡 In the new native multi-node branch, salloc --no-shell (line 61-70) grants a 2-node exclusive 16xH200 allocation, but the cleanup_native_multinode EXIT trap that calls scancel is only registered afterward (line 87), after the squeue-based job_id lookup. If that lookup comes back empty, the script hits exit 1 (line 74) with the allocation still held and un-scancelled, leaking a 16-GPU exclusive block for up to the full 240-minute wall clock.

Extended reasoning...

The bug

salloc --partition=... --nodes="$node_count" --gres="gpu:$gpus_per_node" --exclusive --time=240 --no-shell --job-name="$RUNNER_NAME" (lines 61-69) is a blocking call: because the script runs under set -eo pipefail, control only reaches the next line if salloc returned 0 — i.e. Slurm has already granted the full 2-node, 16xH200 exclusive allocation. Only after that grant does the script resolve the job id via squeue --name="$RUNNER_NAME" -u "$USER" -h -o %A | sed -n '1p' (line 71). If that lookup returns empty — a transient controller RPC lag, or a race where the job has not yet been indexed under its name — the script executes exit 1 at line 74.

Why this leaks the allocation

The cleanup_native_multinode function, which calls scancel "$job_id" in its body, is only wired up via trap cleanup_native_multinode EXIT at line 87 — after the job_id check at lines 72-75. So the exit 1 on line 74 fires with no EXIT trap registered at all, meaning the job that salloc just created is never cancelled. It sits allocated and idle until Slurm reaps it at the --time=240 (4 hour) wall-clock limit, during which no other job can use those two 8xH200 nodes because the allocation was requested with --exclusive.

Why existing code does not prevent it

This is a copy of the pattern already present in the pre-existing single-node path further down in the same file (salloc ...; JOB_ID=$(squeue ...); if [[ -z "$JOB_ID" ]]; then exit 1; fi; trap ... EXIT registered only after that check). That existing path has apparently run in production without observed incident, which is some evidence the empty-after-success case is rare in practice — but the new branch reproduces the same ordering bug for a much larger and more expensive allocation (2 exclusive 8xH200 nodes / 16 GPUs, vs. a single node in the old path).

Impact and mitigating factors

If triggered, this leaks an entire exclusive 2-node/16-GPU H200 block for up to 4 hours, which is a real cost for a shared cluster resource. However, several factors keep this from being a normal-severity, run-breaking bug:

1. salloc --no-shell only returns once the controller has actually granted the allocation, so squeue looking it up by the same job name immediately after should almost always find it — the empty-result window is a narrow, unlikely controller race, not a path any healthy run takes.
2. The stale-same-name scenario would surface a wrong non-empty job_id (picked up by sed -n '1p'), not an empty one, so it does not add likelihood to this specific failure mode.
3. The wider CI workflow already runs a pre-job Slurm cleanup step (scancel --name=${{ runner.name }}) before checkout, which would reap a leaked same-named allocation left behind by a previous run, bounding the practical blast radius even if the race does occur.

Step-by-step proof

1. Runner starts native multinode job; IS_MULTINODE=true, NATIVE_MULTINODE=1.
2. Script reaches line 61: salloc --nodes=2 --gres=gpu:8 --exclusive --time=240 --no-shell --job-name="$RUNNER_NAME". Slurm grants the 2-node, 16-GPU exclusive allocation and salloc returns 0 (required for the script to continue under set -eo pipefail).
3. Line 71: job_id=$(squeue --name="$RUNNER_NAME" -u "$USER" -h -o %A | sed -n '1p'). Assume the Slurm controller has a brief indexing lag and this query returns an empty string.
4. Lines 72-75: if [[ -z "$job_id" ]]; then echo "..." >&2; exit 1; fi fires.
5. At this point in program order, trap cleanup_native_multinode EXIT (line 87) has NOT yet executed — it is textually and temporally after the exit at line 74.
6. The script process exits with no EXIT trap registered, so scancel is never invoked on the allocation created in step 2.
7. The 2-node/16xH200 exclusive allocation remains held (idle, doing nothing) until Slurm terminates it at the 240-minute --time limit.

Suggested fix

Register trap cleanup_native_multinode EXIT (and the INT/TERM traps) immediately after the salloc call succeeds, before the squeue/job_id resolution and its exit 1 branch. cleanup_native_multinode already treats job_id and server_step_pid defensively (2>/dev/null || true), so it is safe to register the trap before those variables are fully populated — it will simply no-op on scancel "" and the kill/wait on an unset server_step_pid in that early-exit case (the job_id value could be captured just before or the scancel call could be made conditional on a non-empty job_id).

[claude]

🟡 kimik3_fp4_vllm.sh doesn't source benchmark_lib.sh or call check_env_vars, unlike every other multi_node entrypoint (e.g. dsv4_fp4_mi355x_sglang-disagg.sh). Since the script only relies on set -euo pipefail, a required var that is set-but-empty (rather than unset) — such as an empty PORT — will silently pass through into the vllm serve command instead of failing fast with a clear error.

Extended reasoning...

AGENTS.md explicitly documents the multi-node/single-node entrypoint convention: source benchmark_lib.sh (check_env_vars, wait_for_server_ready, ...); parameters passed via env vars. Every sibling multi_node server script — including dsv4_fp4_mi355x_sglang-disagg.sh and roughly two dozen others across the repo — sources benchmark_lib.sh and calls check_env_vars on its required inputs before constructing the launch command. kimik3_fp4_vllm.sh is the outlier: it reads MODEL_PATH, MODEL, PORT, VLLM_MOE_BACKEND, PREFILL_DP_ATTN, PREFILL_TP, PREFILL_EP, and others directly into the VLLM_CMD array with no upfront validation step.

The practical gap is narrower than 'no validation at all' — the script does run set -euo pipefail, so set -u already causes an immediate 'unbound variable' abort if any of these vars is genuinely unset when VLLM_CMD is built near the top of the script. What set -u does not catch is a required var that is set but empty (e.g. export PORT=""). check_env_vars (benchmark_lib.sh) explicitly tests -z "${!var:-}", so it rejects both the unset and the empty-string case with a clear, named-variable error message. Without it, an empty PORT here would flow straight into --port "$PORT" and only fail once vllm serve itself rejects the malformed argument, deep into container/Slurm startup rather than at the top of the script.

This exact empty-var scenario is realistic in this PR: the companion runner script (runners/launch_h200-dgxc-slurm.sh) is new native multi-node plumbing that exports PORT, MULTINODE_NODE_COUNT, etc. into the container, and any bug in that plumbing that produces an empty-but-set value would slip straight past this script's only guard (set -u) and manifest as an opaque vLLM startup failure instead of a clear, immediate 'PORT not set' message.

Step-by-step proof:

1. runners/launch_h200-dgxc-slurm.sh exports PORT and other multinode env vars, then execs kimik3_fp4_vllm.sh inside the container via srun ... bash "$server_script".
2. Suppose PORT ends up exported as the empty string (set, but empty) due to any upstream misconfiguration.
3. In kimik3_fp4_vllm.sh, set -u does not fire because PORT is defined (just empty), so execution proceeds past the top of the script.
4. VLLM_CMD is built with --port "$PORT", i.e. --port "", and the script reaches exec "${VLLM_CMD[@]}" before any error is raised.
5. The failure only surfaces once vllm serve itself parses/rejects the empty --port argument, i.e. after container startup and Python/vLLM import overhead, with a vLLM-level error rather than a fast, named 'PORT not set' message from check_env_vars.
6. With source benchmark_lib.sh + check_env_vars PORT MODEL_PATH ... added at the top (matching the sibling scripts), step 3 would instead fail immediately with a clear message naming the empty variable.

Fix: add source "$(dirname "${BASH_SOURCE[0]}")/../benchmark_lib.sh" (matching the sibling scripts' relative path) and a check_env_vars MODEL_PATH MODEL PORT VLLM_MOE_BACKEND PREFILL_DP_ATTN PREFILL_TP PREFILL_EP CONC_LIST MULTINODE_NODE_COUNT MULTINODE_GPUS_PER_NODE MULTINODE_NODE_RANK MULTINODE_MASTER_ADDR" call near the top of the script, before VLLM_CMD` is constructed. This is low-risk and purely additive — it doesn't change any control flow for the well-formed case, it only turns a rare, hard-to-debug failure mode into a fast, clear one, consistent with the pattern every other entrypoint in the repo already follows.

[claude]

🟡 The new native multi-node enroot import block runs flock -w 1800 9 with no failure check, unlike the pre-existing single-node import block in the same file (line ~457) which does flock -w 600 9 || { echo ...; exit 1; }. Because the remote bash -c shell spawned by srun has no set -e of its own, a flock timeout silently falls through to rm -f + enroot import without holding the lock, risking corruption of the shared .sqsh cache if two concurrent jobs race on the same image (e.g. two of the three Kimi K3 orientations, which all share vllm/vllm-openai:kimi-k3).

Extended reasoning...

The new native-multinode code path adds this remote import step:

srun --jobid="$job_id" --nodes=1 --ntasks=1 --nodelist="$head_node" bash -c "
    export ENROOT_CACHE_PATH=\$HOME/.cache/enroot
    mkdir -p \$ENROOT_CACHE_PATH
    exec 9>'$lock_file'
    flock -w 1800 9
    if ! unsquashfs -l '$squash_file' >/dev/null 2>&1; then
        rm -f '$squash_file'
        enroot import -o '$squash_file' 'docker://${IMAGE//#//}'
    fi
"

flock -w 1800 9 is not guarded by || exit 1 (or any other failure check). The string passed to bash -c is a brand new shell invocation with its own execution context — it does not inherit the outer script's set -eo pipefail, which only applies to the top-level launcher process, not to a subshell spawned remotely by srun. So if the flock call times out after 30 minutes (returning non-zero), the script does not abort; it falls straight through to the unsquashfs -l check and, if that also fails (e.g. because the file doesn't exist yet or is a half-written import from the job that's still holding the lock), proceeds to rm -f and enroot import — all without ever holding the lock.

This is a direct regression relative to the established pattern in the very same file. The pre-existing single-node import block a few dozen lines below guards the identical idiom correctly:

flock -w 600 9 || { echo 'Failed to acquire lock for $SQUASH_FILE'; exit 1; }

The PR clearly copied the flock-based serialization idiom for the new native multi-node path but dropped the failure check that makes it safe.

Why this matters: squash_file/lock_file are keyed only by the image name ($(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh) under the shared /data/gharunners/containers directory. All three new Kimi K3 AgentX orientations (latency/balanced/throughput) use the same image, vllm/vllm-openai:kimi-k3, and can be scheduled concurrently as part of a labeled full-sweep. If one job is still mid-import of the ~1.5TB-adjacent image when another job's 30-minute lock wait expires, the second job will rm -f the squash file and start its own enroot import -o over the same path while the first job's import may still be writing to it — corrupting or truncating the cached .sqsh for both jobs and breaking subsequent container launches until someone notices and manually clears the cache.

Step-by-step proof of the failure path:

1. Job A acquires the lock (fd 9 on $lock_file) and starts enroot import -o "$squash_file" docker://vllm/vllm-openai:kimi-k3 — a large image, so this can take a while on a cold cache.
2. Job B (a concurrent orientation using the same image) reaches the same srun ... bash -c "..." block and calls flock -w 1800 9 on the same $lock_file.
3. Job A's import exceeds 30 minutes. Job B's flock call times out and returns exit code 1.
4. Because the bash -c shell has no set -e, Job B's script does not stop — it proceeds to if ! unsquashfs -l '$squash_file' .... Since Job A's import is incomplete, unsquashfs -l fails (file doesn't exist yet, or is invalid/partial), so the condition is true.
5. Job B executes rm -f '$squash_file' — deleting the file Job A is actively writing to — then starts its own enroot import -o '$squash_file' ... concurrently with Job A's still-running import.
6. Both imports now race writes to the same path; whichever finishes last leaves a squash file that may be corrupt/truncated relative to what either import intended, and any job (including a third orientation, or the next day's run) that reads it will fail to launch its container.

Fix: mirror the existing pattern exactly — flock -w 1800 9 || { echo 'Failed to acquire lock for $squash_file'; exit 1; } — so a timed-out lock aborts the remote import step (and the outer srun call fails loudly) instead of silently proceeding without the lock held.

The trigger condition (two same-image jobs overlapping AND one import exceeding the 30-minute wait) is uncommon, and the normal single-writer case works correctly today, so this doesn't need to block merge — but it's a straightforward one-line fix for a defensive-coding gap on newly added code that touches a shared, unversioned cache path.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30342468122
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30342468122