Skip to content

[None][doc] Document the three generation-side time breakdown segments #18213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nv-yna merged 2 commits into from
Sep 9, 2026
Merged

[None][doc] Document the three generation-side time breakdown segments #18213

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
74763c9
[None][doc] Document the three generation-side time breakdown segments
nv-yna Aug 25, 2026
c54eddc
Merge branch 'main' into user/yna/time-breakdown-readme-gen-segments
nv-yna Aug 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 32 additions & 17 deletions tensorrt_llm/serve/scripts/time_breakdown/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ Client Request
┌─────────────────────────────────────────────────────────────────┐
│ CONTEXT SERVER (Prefill) │
│ │
│ [ctx_server_arrival_time] ─── Context server receives request │
│ [ctx_server_arrival_time] ─── Context server receives request │
Comment thread
coderabbitai[bot] marked this conversation as resolved.
│ | │
│ v │
│ [ctx_arrival_time] ────────── After tokenization │
Expand Down Expand Up @@ -140,7 +140,13 @@ Client Request
│ [gen_server_arrival_time] ─── Generation server receives req │
│ | │
│ v │
│ [gen_arrival_time] ────────── After KV cache received │
│ [gen_arrival_time] ────────── After request parsing + IPC │
│ | │
│ v │
Comment thread
coderabbitai[bot] marked this conversation as resolved.
│ [gen_kv_cache_transfer_start] ─ KV cache transfer begins │
│ | │
│ v │
│ [gen_kv_cache_transfer_end] ─── KV cache transfer complete │
│ | │
│ v │
│ [gen_first_scheduled_time] ── Scheduled for decode │
Expand Down Expand Up @@ -264,29 +270,38 @@ For each generation step, the following CPU and GPU times are recorded:

5. **Generation Preprocessing** (`gen_preprocessing`)
- **Time Period**: `gen_server_arrival_time` → `gen_arrival_time`
- **Description**: Python overhead & initialization when generation server receives the request
- **Description**: Generation server overhead: request parsing and IPC from the OpenAI server to the LLM executor

6. **Generation Queue Wait** (`gen_queue_wait`)
- **Time Period**: `gen_arrival_time` → `gen_kv_cache_transfer_start`
- **Description**: Time the request waits in the generation executor queue before the KV cache transfer begins

7. **Generation KV Transfer** (`gen_kv_transfer`)
- **Time Period**: `gen_kv_cache_transfer_start` → `gen_kv_cache_transfer_end`
- **Description**: GPU-to-GPU transfer of the cached key-value tensors from the context server to the generation server
- **Note**: Sourced from `kv_cache_transfer_start` / `kv_cache_transfer_end` in the generation server's `timing_metrics`. When those fields are absent, this segment and the two surrounding it all evaluate to zero, and the `gen_arrival_time` → `gen_first_scheduled_time` interval is not attributed to any segment.

6. **Generation Queue** (`gen_queue`)
- **Time Period**: `gen_arrival_time` → `gen_first_scheduled_time`
- **Description**: Time spent in queue including KV cache transfer
8. **Generation Post Transfer** (`gen_post_transfer`)
- **Time Period**: `gen_kv_cache_transfer_end` → `gen_first_scheduled_time`
- **Description**: Time from KV cache transfer completion to the request being scheduled for generation

7. **Generation First Token Postprocessing** (`gen_postprocessing`)
9. **Generation First Token Postprocessing** (`gen_postprocessing`)
- **Time Period**: `gen_first_scheduled_time` → `gen_server_first_token_time`
- **Description**: Time to generate and send first token from generation server
- **Description**: Time to generate and send the first token from the generation server

### Disaggregation Server Metrics

8. **Disaggregation Preprocessing** (`disagg_preprocessing`)
- **Time Period**: `disagg_server_arrival_time` → `ctx_server_arrival_time`
- **Description**: Routing overhead from disagg server to context server
- **Includes**: Request routing, load balancing, network transfer
10. **Disaggregation Preprocessing** (`disagg_preprocessing`)
- **Time Period**: `disagg_server_arrival_time` → `ctx_server_arrival_time`
- **Description**: Routing overhead from disagg server to context server
- **Includes**: Request routing, load balancing, network transfer

9. **Disaggregation Relay** (`disagg_relay`)
- **Time Period**: `ctx_server_first_token_time` → `gen_server_arrival_time`
- **Description**: Handoff from context server to generation server
- **Includes**: Network transfer, KV cache metadata relay via disagg router
11. **Disaggregation Relay** (`disagg_relay`)
- **Time Period**: `ctx_server_first_token_time` → `gen_server_arrival_time`
- **Description**: Handoff from context server to generation server
- **Includes**: Network transfer, KV cache metadata relay via disagg router

10. **Disaggregation Postprocessing** (`disagg_postprocessing`)
12. **Disaggregation Postprocessing** (`disagg_postprocessing`)
- **Time Period**: `gen_server_first_token_time` → `disagg_server_first_token_time`
- **Description**: Routing overhead from generation server back through disagg server
- **Includes**: Response aggregation, network transfer to client
Expand Down
Loading