Skip to content

Use TensorRT automatic workspace sizing - #1229

Open
zsqdx wants to merge 2 commits into
lightvector:masterfrom
zsqdx:agent/raise-trt-workspace-limit
Open

Use TensorRT automatic workspace sizing#1229
zsqdx wants to merge 2 commits into
lightvector:masterfrom
zsqdx:agent/raise-trt-workspace-limit

Conversation

@zsqdx

@zsqdx zsqdx commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • remove KataGo's fixed 1 GiB TensorRT workspace cap
  • use TensorRT's device-dependent default, which scales with the target GPU's total memory
  • keep the ONNX graph and all FP32 / kOBEY precision constraints unchanged

Fixes #1228.

Root cause

setMemoryPoolLimit() excludes tactics that need more workspace than the configured limit. For the reported b11 transformer at batch 576, the 12-head 19x19 attention score tensor alone is about 1.68 GiB in FP16, so the fixed 1 GiB limit can leave a fused TensorRT node with no implementation.

TensorRT already provides automatic device-dependent sizing: when the workspace limit is not explicitly set, it defaults to the target GPU's total memory. This is only a tactic-selection ceiling, not a preallocation; each execution context allocates the workspace required by its selected plan. NVIDIA documents both behaviors here: https://docs.nvidia.com/deeplearning/tensorrt/latest/reference/troubleshooting-faq.html#how-do-i-choose-the-optimal-workspace-size

This is preferable to moving the fixed limit from 1 GiB to another constant. On TensorRT 10.16.1 / RTX 5090, a 2 GiB cap fixes b576 but fails again at b1024, whereas the automatic default builds b576, b1024, and b2048.

Validation

Tested current master with TensorRT 10.16.1.11, CUDA 13.2, and 8x RTX 5090 using b11c768h12nbt3tflrs-fson-silu:

max batch / configuration Result
b576, fixed 1 GiB Error Code 10
b576, fixed 1.5 / 1.75 GiB Error Code 10
b576, fixed 2 GiB Builds
b1024, fixed 2 GiB Error Code 10
b576 / b1024 / b2048, TensorRT automatic default All build
b576, automatic default, two NN servers on one GPU Builds; about 5.47 GiB total, unchanged from the 2 GiB-cap build
b576, automatic default, 16 NN servers on 8 GPUs Completes 60k visits at 56,284.55 visits/s

testgpuerror -quick also passes 669 positions; batched and unbatched errors are identical and all margins remain within limits.

The existing build mutex continues to serialize TensorRT engine builds, so allowing the device default does not multiply builder workspace across NN server threads.

@zsqdx zsqdx changed the title Raise TensorRT workspace limit for large transformer batches Use TensorRT automatic workspace sizing Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TensorRT: NHWC transformer trunk fails to build at large maxBatchSize (Error Code 10)

1 participant