Skip to content

[KimiK3][AgentX]: GB200 KimiK3 Day 0 support / [KimiK3][AgentX]:GB200 KimiK3 Day 0 支持 - #2354

Open
cquil11 wants to merge 27 commits into
mainfrom
agent/kimik3-gb200-agentx
Open

[KimiK3][AgentX]: GB200 KimiK3 Day 0 support / [KimiK3][AgentX]:GB200 KimiK3 Day 0 支持#2354
cquil11 wants to merge 27 commits into
mainfrom
agent/kimik3-gb200-agentx

Conversation

@cquil11

@cquil11 cquil11 commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add GB200 Kimi K3 Day-0 AgentX performance coverage with checked-in srt-slurm recipes.
  • Follow the official GB200 latency, balanced, and throughput strategies with TP16, TEP16, and TP4 × DP4 = EP16 profiles.
  • Use the staged /mnt/numa1/models/Kimi-K3 checkpoint and keep Kimi K3’s native 1M context.
  • Pass every vLLM DP engine metrics endpoint to AIPerf.

Official recipe: Kimi K3 on GB200.

Shared AgentX phase-start behavior is tracked separately in #2375.

Validation

中文说明

  • 通过仓库内置的 srt-slurm 配方,新增 GB200 Kimi K3 Day-0 AgentX 性能覆盖。
  • 参考官方 GB200 延迟、均衡和吞吐量策略,分别测试 TP16、TEP16 及 TP4 × DP4 = EP16。
  • 使用预置于 /mnt/numa1/models/Kimi-K3 的检查点,并保留 Kimi K3 原生 1M 上下文长度。
  • 将所有 vLLM DP 引擎的指标端点传递给 AIPerf。

官方配方:GB200 上的 Kimi K3

共享的 AgentX 阶段启动行为在 #2375 中单独跟踪。

验证

Add srt-slurm TP, TEP, and DEP AgentX profiles plus GB200 runner routing and recipe overlay plumbing.

中文:新增 Kimi K3 GB200 首日 AgentX 支持,包括 srt-slurm TP、TEP、DEP 配置,以及 GB200 运行器路由和配方覆盖逻辑。
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs


感谢你的贡献!请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md,然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人,你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板,包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have

如需进行 PR 验证,请为此 PR 添加 full-sweep-fail-fast 标签(强烈推荐)— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动(flake),重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

Record the draft pull request URL in the append-only benchmark changelog entry.

中文:在仅追加的基准测试变更日志条目中记录草稿 PR 链接。
@cquil11 cquil11 changed the title [KimiK3][AgentX]: GB200 KimiK3 Day 0 support / [KimiK3][AgentX]:GB200 KimiK3 首日支持 [KimiK3][AgentX]: GB200 KimiK3 Day 0 support Jul 27, 2026
中文:将 origin/main 合并到 GB200 Kimi K3 AgentX 分支。
@cquil11
cquil11 marked this pull request as ready for review July 27, 2026 23:30
@cquil11
cquil11 requested a review from a team July 27, 2026 23:30
@claude

claude Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Claude finished @cquil11's task in 8m 24s —— View job


Review: Kimi K3 GB200 AgentX Day-0 support

  • Gather context — read full diff, all three recipes, launcher, and master config
  • Validate master config (prefixes, images, changelog pairing, GPU accounting)
  • Review srt-slurm recipes and launcher changes
  • Post review feedback

Result: 1 warning, no blocking issues. Left inline on agg-gb200-tp8-latency-agentic.yaml:63-64GLOO_SOCKET_IFNAME/NCCL_SOCKET_IFNAME: "$IFACE_NAME" references a variable that nothing in this repo defines and that no other recipe (including the dsv4 GB200 agentic recipes on the same pinned srt-slurm commit) uses; if the pinned srt-slurm templates don't export it at container runtime, multi-node worker bootstrap fails at startup. Applies to all three recipes. Please confirm or drop those two lines.

Everything else verified clean:

  • perf-changelog.yaml entry is correctly appended at the end of the file, paired with the configs/nvidia-master.yaml change.
  • GPU accounting is consistent with the agentic ingestion path (process_agentic_result.py computes num_workers × tp): TP8 → 8 GPUs / 2 nodes, TEP8 → 8 / 2, DEP16 encoded as tp: 16 → 16 GPUs / 4 nodes, matching the dsv4 GB300 agentic-disagg precedent (not the fixed-seq-len tp: 1, ep: 16 convention — this is the right one for agentic).
  • Launcher changes are sound: kimi-k3 model alias matches model.path in all three recipes, the srtslurm.yaml container mapping keys off $IMAGE which matches the recipes' container: value, the recipe-overlay if/else keeps the dsv4 GB200 agentic path intact, and the existing import_squash enroot pattern covers the new image.
  • Recipe structure (identity block, DYNAMO_WHEEL_DIRS, dynamo wheel pin 1.3.0.dev20260618 matching the router version, aiperf_server_metrics, benchmark env) matches the GB300 agentic agg template; enable-expert-parallel is set only in the TEP/DEP recipes, consistent with the ep values in the master config.
  • cluster:gb200-nv runner label and the agentic-coding search-space shape both pass the matrix-logic schema conventions; IS_AGENTIC is derived from the scenario type so the new entries route into the agentic launcher branch.

One non-blocking note: vllm/vllm-openai:kimi-k3 is a floating tag while every other entry pins a version. Since import_squash reuses a valid cached squash keyed by image name, a later upstream re-push of that tag would not be picked up by runners that already imported it — worth keeping in mind if the day-0 image gets patched upstream.

Represent the throughput arm as TP1 x DP16 = EP16 in the master configuration, matching the vLLM recipe.

中文:将吞吐量配置修正为 TP1 × DP16 = EP16,与 vLLM 配方保持一致。
Comment thread runners/launch_gb200-nv.sh
cquil11 added 3 commits July 27, 2026 18:42
Use the pre-staged /mnt/numa1/models/Kimi-K3 checkpoint on every GB200 compute node while retaining shared-filesystem orchestration and caches.

中文:在每个 GB200 计算节点上使用预置的 /mnt/numa1/models/Kimi-K3 检查点,同时保留共享文件系统上的编排与缓存。
Keep the Day-0 changelog focused on the vLLM recipe performance points and use the model default context length.

中文:精简 Kimi K3 首日性能配置,仅保留与 vLLM 配方一致的性能测试点,并使用模型默认上下文长度。
Move latency and balanced profiles to 16-GPU TP/TEP, use TP2 x DP8 for EP16 throughput, and extend the throughput concurrency sweep through 256.

中文:根据 GB200 HBM 容量调整 Kimi K3 AgentX 扫描。将延迟与均衡配置扩展为 16 GPU TP/TEP,吞吐配置改为 TP2 × DP8(EP16),并将吞吐并发扫描扩展至 256。
@cquil11 cquil11 changed the title [KimiK3][AgentX]: GB200 KimiK3 Day 0 support [KimiK3][AgentX]: GB200 KimiK3 Day 0 support / [KimiK3][AgentX]:GB200 KimiK3 Day 0 支持 Jul 28, 2026
The Dynamo vLLM worker entrypoint does not accept vllm serve tool-call or reasoning parser flags; agentic request parsing belongs to the Dynamo frontend.

中文:移除 Dynamo vLLM worker 不支持的 Kimi K3 工具调用与推理解析参数;AgentX 请求解析由 Dynamo frontend 负责。
@github-actions

Copy link
Copy Markdown
Contributor

Remove literal shell placeholders from the vLLM worker environment so Gloo and NCCL use the interfaces selected by the srt-slurm launcher.

中文:移除 vLLM worker 环境中的字面量 shell 占位符,由 srt-slurm launcher 为 Gloo 与 NCCL 选择网络接口。
@github-actions

Copy link
Copy Markdown
Contributor

Patch the pinned agentic srt-slurm checkout to launch one vLLM process per GB200 node, with two local DP ranks and TP2 per rank. Export every per-node metrics endpoint to AIPerf.

中文:修补 AgentX 固定版本的 srt-slurm,使每个 GB200 节点启动一个 vLLM 进程,每个进程承载两个本地 DP rank,并为每个 rank 使用 TP2;同时将所有节点的 metrics endpoint 传给 AIPerf。
@github-actions

Copy link
Copy Markdown
Contributor

Use TP4 x DP4 for the throughput-oriented EP16 profile after TP2 x DP8 exceeded GB200 HBM during MXFP4 weight loading.

中文:TP2 × DP8 在 MXFP4 权重加载阶段超过 GB200 HBM 容量,因此将吞吐优先的 EP16 配置调整为 TP4 × DP4。
@github-actions

Copy link
Copy Markdown
Contributor

Pin a tested srt-slurm revision that natively groups TP GPUs within each DP rank and exports every DP metrics endpoint. Remove the vendored patch and reserve enough HBM for one native 1M-context request.

中文:固定经过测试的 srt-slurm 版本,原生支持每个 DP rank 内的 TP GPU 分组,并导出全部 DP metrics endpoint。删除仓库内补丁,同时为原生 1M 上下文请求预留足够 HBM。
@cquil11
cquil11 marked this pull request as ready for review July 28, 2026 03:04
@cquil11
cquil11 marked this pull request as draft July 28, 2026 03:04
@claude

claude Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Claude finished @cquil11's task in 8m 30s —— View job


Re-review: Kimi K3 GB200 AgentX Day-0 support

  • Gather context — diff vs origin/main, changes since previous review (12 new commits: recipes reworked tp8→TP16/TEP16/DEP16, new srt-slurm pin for kimik3, new agentic result staging + tests)
  • Verify status of previously flagged $IFACE_NAME issue
  • Review new/renamed recipes and launcher/slurm_utils changes
  • Validate master config + perf-changelog rules
  • Post review feedback

Result: 2 blocking issues, 1 warning (all inline). Both previously flagged issues are resolved: the GLOO_SOCKET_IFNAME/NCCL_SOCKET_IFNAME: "$IFACE_NAME" lines are gone from all recipes, and the title/body are now bilingual. The floating vllm/vllm-openai:kimi-k3 tag was also replaced with a pinned NGC image.

New blocking issues:

  1. runners/slurm_utils.sh:56-60copy_agentic_results is now called for every agentic run in launch_gb200-nv.sh, but for dsv4 (not in uses_watchtower_shared_fs), INFMAX_WORKSPACE == GITHUB_WORKSPACE, so each aggregate is cp'd onto itself. GNU cp exits 1 on same-file, the || exit 1 fails the job — the existing dsv4 GB200 agentic sweeps will hard-fail at result staging after a successful benchmark. Suggested a -ef same-file guard inline.

  2. configs/nvidia-master.yaml:8047-8058 — the DEP16 throughput profile encodes tp: 4, but process_agentic_result.py computes GPUs as num_workers × tp with no DP factor, so a 16-GPU (TP4 × DP4) deployment is reported as 4 GPUs and per-GPU throughput is inflated 4×. Repo convention (dsv4 DEP8 → tp: 8 for TP1×DP8) is that master tp holds the per-worker GPU count; the earlier revision's tp: 16 was correct.

Warning: the TP16 latency profile reports ep: 16 while its recipe never sets enable-expert-parallel — vLLM's EP = TP × DP rule only holds with the flag on; the dsv4 pure-TP agentic precedent reports ep: 1.

Non-blocking question: the recipes record identity.frameworks.dynamo: "1.3.0" and the master config router.version: "1.3.0", but the container is ai-dynamo/vllm-runtime:1.4.0-kimi-k3-dev.1 with dynamo.install: false — please confirm the Dynamo runtime inside that image really is 1.3.0, otherwise the recorded router/framework version is wrong.

Everything else checked clean: perf-changelog entry is correctly appended at the end and paired with the master-config change; the pinned NGC image is public; TEP16 recipe (TP16 + enable-expert-parallel) matches its ep: 16 metadata; the kimik3-vs-dsv4 srt-slurm pin split in the launcher keeps the dsv4 path byte-identical; the kimi-k3 model alias matches model.path in all three recipes; and the new test_slurm_utils.py covers the staging happy path and missing-aggregate failure (though not the same-directory case from issue 1).

Comment on lines +8047 to +8058
prefill:
num-worker: 1
tp: 4
ep: 16
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-dep16-throughput-agentic.yaml"
decode:
num-worker: 0
tp: 4
ep: 16
dp-attn: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 BLOCKING: The DEP16 throughput profile encodes tp: 4, so the agentic ingestion reports 4 GPUs for a 16-GPU deployment, inflating per-GPU throughput 4×.

Why it matters: process_agentic_result.py:148 computes num_prefill_gpu = PREFILL_NUM_WORKERS × PREFILL_TP × PP × PCP — there is no DP factor in the multinode path. With num-worker: 1, tp: 4 this yields num_gpus = 4, but the recipe actually allocates 16 GPUs (agg_nodes: 4, gpus_per_agg: 16, TP4 × DP4). All throughput.per_gpu metrics for this profile will be divided by 4 instead of 16 — a 4× inflation of the headline per-GPU numbers, and it makes this profile incomparable with the TP16/TEP16 profiles in the same config.

The repo convention for agentic vLLM DP workers is that master-config tp holds the per-worker GPU count (TP × DP): the dsv4 precedent disagg-gb200-2p1d-dep8-dep8-agentic runs tensor-parallel-size: 1, data-parallel-size: 8 and is encoded as tp: 8 (configs/nvidia-master.yaml:7860). The earlier revision of this PR encoded this profile as tp: 16, which was correct.

Fix:

Suggested change
prefill:
num-worker: 1
tp: 4
ep: 16
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-dep16-throughput-agentic.yaml"
decode:
num-worker: 0
tp: 4
ep: 16
dp-attn: true
prefill:
num-worker: 1
tp: 16
ep: 16
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-dep16-throughput-agentic.yaml"
decode:
num-worker: 0
tp: 16
ep: 16
dp-attn: true

Comment thread runners/slurm_utils.sh
Comment on lines +8014 to +8025
prefill:
num-worker: 1
tp: 16
ep: 16
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml"
decode:
num-worker: 0
tp: 16
ep: 16
dp-attn: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: The TP16 latency profile reports ep: 16, but its recipe (agg-gb200-tp16-latency-agentic.yaml) does not set enable-expert-parallel — experts are TP-sharded, and there is no EP group. vLLM's EP_SIZE = TP × DP rule (cited in commit 86f4e70) only applies when --enable-expert-parallel is on, which is why only the TEP16 and DEP16 recipes set it.

Why it matters: the reported topology metadata is wrong — the frontend will display an EP16 configuration for a run that has no expert parallelism, and it diverges from the repo precedent for pure-TP agentic profiles (dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg reports ep: 1 for its TP8/TP4 recipes without the flag, configs/nvidia-master.yaml:7918).

Fix:

Suggested change
prefill:
num-worker: 1
tp: 16
ep: 16
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml"
decode:
num-worker: 0
tp: 16
ep: 16
dp-attn: false
prefill:
num-worker: 1
tp: 16
ep: 1
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml"
decode:
num-worker: 0
tp: 16
ep: 1
dp-attn: false

(Alternatively, if EP16 is intended for the latency profile, add enable-expert-parallel: true to the recipe — but that would no longer match the official multi_node_tp strategy this profile cites.)

Comment on lines +8045 to +8058
- spec-decoding: none
conc-list: [32, 64, 96, 128, 192, 256]
prefill:
num-worker: 1
tp: 4
ep: 16
dp-attn: true
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-dep16-throughput-agentic.yaml"
decode:
num-worker: 0
tp: 4
ep: 16
dp-attn: true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The DEP16 throughput scenario in configs/nvidia-master.yaml (kimik3-fp4-gb200-dynamo-vllm-agentic) sets prefill num-worker: 1, tp: 4 to represent the TP4×DP4=16-GPU recipe (agg-gb200-dep16-throughput-agentic.yaml), but process_agentic_result.py's GPU accounting only multiplies num-worker * tp (no data-parallel factor), so it computes num_gpus=4 instead of 16. This silently inflates every per-GPU throughput/cost metric published for this scenario by 4x. Fix by setting num-worker: 4, tp: 4 (matching the DEP4 disagg precedent) or otherwise ensuring the accounting resolves to 16.

Extended reasoning...

The bug: configs/nvidia-master.yaml lines 8047-8058 add the throughput-oriented search-space entry for kimik3-fp4-gb200-dynamo-vllm-agentic, encoding the DEP16 profile (TP4 × DP4 = 16 GPUs, per agg-gb200-dep16-throughput-agentic.yaml's tensor-parallel-size: 4, data-parallel-size: 4, resources.gpus_per_agg: 16) as prefill num-worker: 1, tp: 4. The GPU-shape accounting in utils/agentic/aggregation/process_agentic_result.py::_gpu_shape() computes num_prefill_gpu = prefill_num_workers * prefill_tp * prefill_pp * prefill_pcp_size — there is no data-parallel-size factor anywhere in that function or in the env vars wired into it. For this entry that evaluates to 1 * 4 * 1 * 1 = 4, not 16.\n\nCode path: .github/workflows/benchmark-multinode-tmpl.yml (lines 226-227) sets PREFILL_NUM_WORKERS and PREFILL_TP verbatim from the master-config's prefill.num-worker/prefill.tp fields for every agentic scenario. _gpu_shape() reads those env vars and multiplies them (plus PP/PCP, both 1 here) to get num_prefill_gpu, which becomes num_gpus for the aggregated (non-disagg) case. That value is then used directly as the denominator in build_agg(): request_nested["throughput"]["per_gpu"] = {"total_tput_tps": request_flat["total_tput_tps"] / num_gpus, ...}. It's also published in the aggregate JSON as the reported GPU/node count for the run.\n\nWhy nothing else catches it: the actual Slurm deployment is driven entirely by the recipe YAML via CONFIG_FILE (srtctl schedules the real 16-GPU topology regardless of these master-config fields), so the benchmark itself runs correctly on 16 GPUs — this is purely an accounting-side defect in result post-processing, invisible at runtime and not something CI's job-success checks would flag. The sibling TP16/TEP16 entries in the exact same PR set tp: 16 and correctly resolve to num_gpus=16, which shows the intended convention and isolates this DEP16 entry as the one exception. The existing DEP4 disagg precedent (dsv4-fp4-gb300-dynamo-vllm-agentic) reaches its 16-GPU total by encoding DP replicas via num-worker: 4, tp: 4, a pattern this new entry doesn't follow.\n\nImpact: every per-GPU throughput and implied per-GPU-cost figure published for the DEP16 throughput profile (concurrency sweep 32-256) will be reported as 4x higher than reality, since the true 16-GPU denominator is replaced with 4. This is exactly the headline metric a 'throughput' scenario exists to report, so the corruption lands in the most visible part of the published benchmark data, not an edge case.\n\nStep-by-step proof:\n1. Recipe agg-gb200-dep16-throughput-agentic.yaml sets resources: {gpus_per_node: 4, agg_nodes: 4, agg_workers: 1, gpus_per_agg: 16} and vllm_config.aggregated: {tensor-parallel-size: 4, data-parallel-size: 4} — 4 nodes × 4 GPUs/node = 16 GPUs actually allocated and used (TP4 × DP4).\n2. configs/nvidia-master.yaml's new throughput search-space entry sets prefill: {num-worker: 1, tp: 4, ...}.\n3. The workflow template exports PREFILL_NUM_WORKERS=1, PREFILL_TP=4 verbatim from those fields.\n4. _gpu_shape() computes num_prefill_gpu = 1 * 4 * 1 * 1 = 4; since decode_num_workers=0, num_gpus = num_prefill_gpu + 0 = 4.\n5. build_agg() divides total_tput_tps (measured across the real 16-GPU deployment) by num_gpus=4 instead of 16 — the published per-GPU throughput is exactly 4x too high, and the aggregate JSON reports 4 GPUs for a run that used 16.\n\nFix: encode the DP replication via num-worker instead of collapsing it, i.e. set prefill num-worker: 4, tp: 4 (matching the DEP4 disagg precedent already in this same file), which yields num_prefill_gpu = 4 * 4 = 16. This changes only the accounting env vars used by result post-processing; the actual deployment remains fully recipe-driven via CONFIG_FILE and is unaffected.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The config at configs/nvidia-master.yaml:8079-8090 is unchanged from the original report: prefill/decode still set num-worker: 1, tp: 4 for the DEP16 throughput profile. process_agentic_result.py's _gpu_shape() (utils/agentic/aggregation/process_agentic_result.py:148) still computes num_prefill_gpu = prefill_num_workers * prefill_tp with no DP factor, so this resolves to num_gpus=4 for a 16-GPU (TP4xDP4) deployment, inflating per-GPU throughput 4x. No fix has been applied.

Comment thread runners/test_slurm_utils.py
Comment on lines +8012 to +8025
- spec-decoding: none
conc-list: [1, 2, 4, 8]
prefill:
num-worker: 1
tp: 16
ep: 16
dp-attn: false
additional-settings:
- "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml"
decode:
num-worker: 0
tp: 16
ep: 16
dp-attn: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new Kimi K3 latency search-space entry (kimik3-fp4-gb200-dynamo-vllm-agentic, conc-list [1,2,4,8]) sets prefill.ep: 16 / decode.ep: 16, but its recipe (agg-gb200-tp16-latency-agentic.yaml) never sets enable-expert-parallel — it's pure dense TP16 with no MoE expert sharding, unlike its TEP16/DEP16 siblings which do set it. Per repo convention for dense-TP-only agentic configs (see dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg), this should be ep: 1; as written, process_agentic_result.py will publish this TP16 profile as an EP16 topology in the aggregated results metadata.

Extended reasoning...

The bug: In configs/nvidia-master.yaml (added by this PR, ~lines 8012-8025), the latency-oriented Kimi K3 GB200 search-space entry sets:

prefill:
  num-worker: 1
  tp: 16
  ep: 16
  dp-attn: false
  additional-settings:
  - "CONFIG_FILE=recipes/vllm/kimi-k3/agentic/agg-gb200-tp16-latency-agentic.yaml"
decode:
  num-worker: 0
  tp: 16
  ep: 16
  dp-attn: false

But the recipe it points at, agg-gb200-tp16-latency-agentic.yaml, only sets tensor-parallel-size: 16 in its vllm_config.aggregated block — it never sets enable-expert-parallel. That makes it a pure dense TP16 deployment with no MoE expert-parallel sharding at all (expert-parallel degree is effectively 1).

Contrast with siblings: The other two new recipes in this PR, agg-gb200-tep16-balanced-agentic.yaml and agg-gb200-dep16-throughput-agentic.yaml, both explicitly set enable-expert-parallel: true (the latter combined with data-parallel-size: 4 for TP4×DP4=EP16). Their master-config entries correctly carry ep: 16. The latency recipe is the odd one out — it deliberately omits expert-parallel (per the official vLLM recipe's multi_node_tp strategy, as opposed to multi_node_tep/multi_node_dep), yet its master-config entry still copies ep: 16 from its siblings.

Repo convention: The established pattern for dense-TP-only agentic aggregate configs is ep: 1. E.g. dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg sets tp: 8, ep: 1 and tp: 4, ep: 1 for its two search-space entries, whose recipes (agg-gb300-tp8-mtp-agentic.yaml, agg-gb300-tp4-mtp-agentic.yaml) likewise only set tensor-parallel-size with no enable-expert-parallel — the exact same shape as this Kimi K3 TP16 latency recipe.

Downstream impact: process_agentic_result.py reads PREFILL_EP/DECODE_EP from the master config into the published aggregate result rows as prefill_ep/decode_ep, and derives the top-level ep field as max(prefill_ep, decode_ep). So every published/visualized result point for this TP16 latency profile will be labeled ep: 16, contradicting both the actual serving topology and the recipe's own comment ("TP16 across four GB200 nodes"). This is metadata-only — num_gpus is computed independently from tp × pp × pcp (or num_workers × tp), so GPU accounting, per-GPU throughput, and cost calculations are all unaffected. The only consequence is that anyone reading the published result JSON or a dashboard built on it will see an incorrect ep value for this topology.

Step-by-step proof:

  1. Master config sets prefill.ep: 16 and decode.ep: 16 for the conc-list: [1, 2, 4, 8] search-space entry pointing at agg-gb200-tp16-latency-agentic.yaml.
  2. That recipe's vllm_config.aggregated only has tensor-parallel-size: 16 — no enable-expert-parallel, no data-parallel-size — so at serve time vLLM runs a plain TP16 dense deployment with EP degree 1.
  3. The launcher (launch_gb200-nv.sh) passes PREFILL_EP/DECODE_EP from the master config (not from the recipe) into the benchmark environment.
  4. process_agentic_result.py reads those env vars into prefill_ep/decode_ep on each result row and computes ep = max(prefill_ep, decode_ep) = 16.
  5. The final published/aggregated JSON for this TP16 latency profile therefore reports ep: 16, even though no expert parallelism is configured — a topology mislabel visible in dashboards and downstream analysis.

Fix: Change prefill.ep and decode.ep to 1 for this latency search-space entry, matching the dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg convention for dense-TP-only recipes.

Enable AIPerf burst phase starts only for the Kimi K3 GB200 saturation recipes so warmup and profiling reach the requested concurrency within the benchmark window. Inter-turn replay timing remains unchanged.

中文:仅在 Kimi K3 GB200 饱和度测试配置中启用 AIPerf 阶段起始突发,使预热和正式测试能在基准测试窗口内达到目标并发度,同时保持轮次间回放时序不变。
@github-actions

Copy link
Copy Markdown
Contributor

cquil11 added 3 commits July 27, 2026 22:58
Remove the duplicated AIPerf command-builder hook and Kimi-specific opt-ins. The shared default is tracked in InferenceX PR #2375.

中文:移除重复的 AIPerf 命令构建逻辑和 Kimi 专用开关,统一依赖 InferenceX PR #2375 提供的共享默认行为。
Merge the latest main branch while preserving the Kimi K3 changelog entry at the append-only tail.

中文:合并最新 main 分支,并将 Kimi K3 变更日志条目保留在只追加文件的末尾。
Bring in the merged AgentX v1.0 AIPerf default and current benchmark updates while preserving the Kimi K3 changelog entry at the tail.\n\n中文:合并 AgentX v1.0 AIPerf 默认行为及最新基准测试更新,并保留位于末尾的 Kimi K3 变更日志条目。
@cquil11
cquil11 marked this pull request as ready for review July 28, 2026 04:16
@claude

claude Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Claude finished @cquil11's task in 2m 44s —— View job


Re-review: Kimi K3 GB200 AgentX Day-0 support

  • Gather context — diff vs origin/main, commits since last review (03:05Z)
  • Verify status of previously flagged blocking issues and warning
  • Review new changes (burst phase starts, merges from main)
  • Validate master config + perf-changelog rules
  • Post review feedback

Re-review: no new issues, but the two previously flagged blocking issues are still unresolved. Since the last review the only non-merge commits are 64de559 (added per-recipe AIPerf burst phase-start opt-ins) and ff5fb06 (removed them again in favor of the shared default from #2375, which is now merged into this branch) — they net to zero, and the deferral to #2375 is correct. All 8 PR-owned files (runners/, the three kimi-k3 recipes, configs/nvidia-master.yaml) are byte-identical to the state reviewed at 03:05Z.

Still open from the previous review (see the existing inline comments — not repeating them):

  1. 🔴 runners/slurm_utils.sh:56-60copy_agentic_results is called for every agentic run in launch_gb200-nv.sh:652, but for dsv4 (not in uses_watchtower_shared_fs) INFMAX_WORKSPACE == GITHUB_WORKSPACE, so each aggregate is cp'd onto itself, GNU cp exits 1, and the || exit 1 fails the existing dsv4 GB200 agentic sweeps after a successful benchmark. The suggested -ef same-file guard from the previous inline comment still applies. Fix this →
  2. 🔴 configs/nvidia-master.yaml:8079-8090 — the DEP16 throughput profile still encodes tp: 4 for a 16-GPU (TP4 × DP4) deployment; process_agentic_result.py computes GPUs as num_workers × tp with no DP factor, so per-GPU throughput is inflated 4×. Repo convention (dsv4 DEP8 → tp: 8) is that tp holds the per-worker GPU count; tp: 16 (as in an earlier revision) is correct. Fix this →
  3. 🟡 configs/nvidia-master.yaml:8044-8057 — the TP16 latency profile still reports ep: 16 while its recipe never sets enable-expert-parallel; per the dsv4 pure-TP precedent this should be ep: 1.

The non-blocking question about the Dynamo runtime version inside vllm-runtime:1.4.0-kimi-k3-dev.1 vs the recorded router.version: "1.3.0" also remains open. The perf-changelog.yaml entry is still correctly appended at the end of the file after the merges from main.

Comment thread runners/test_slurm_utils.py
Allow the full-context DEP saturation sweep up to 3600 seconds to drain accelerated warmup requests. The grace period exits early when the servers become idle.

中文:将 Kimi K3 DEP 全上下文饱和度测试的加速预热排空上限延长至 3600 秒;服务空闲后会提前结束等待。
@github-actions

Copy link
Copy Markdown
Contributor

@cquil11 cquil11 changed the title [KimiK3][AgentX]: GB200 KimiK3 Day 0 support / [KimiK3][AgentX]:GB200 KimiK3 Day 0 支持 [KimiK3][AgentX]: GB200 KimiK3 Day 0 support Jul 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@cquil11

cquil11 commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

/stage-results 30337121602

@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@cquil11 staged run 30337121602: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-07-28~r30337121602

This shared staging slot remains available until the next /stage-results request. Staging workflow

cquil11 added 2 commits July 28, 2026 16:16
中文:将最新 main 分支合并到 Kimi K3 GB200 支持分支。

# Conflicts:
#	perf-changelog.yaml
Restore the legacy AgentX AIPerf integration with the barrier-aware idle cap, and retain the GB200 Kimi K3 performance changelog entry at the append-only tail.

中文:同步最新 main,恢复带屏障感知空闲上限的旧版 AgentX AIPerf 集成,并将 GB200 Kimi K3 性能变更记录保留在追加式文件末尾。
@github-actions

Copy link
Copy Markdown
Contributor

Add gb200-nv_3 to the GB200 cluster runner pool so the Kimi K3 sweep can use the capacity provisioned for this bring-up.

中文:将 gb200-nv_3 注册到 GB200 集群运行器池,使 Kimi K3 扫描能够使用本次适配新增的容量。
@github-actions

Copy link
Copy Markdown
Contributor

@cquil11 cquil11 changed the title [KimiK3][AgentX]: GB200 KimiK3 Day 0 support [KimiK3][AgentX]: GB200 KimiK3 Day 0 support / [KimiK3][AgentX]:GB200 KimiK3 Day 0 支持 Jul 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant