Use TensorRT automatic workspace sizing - #1229
Open
zsqdx wants to merge 2 commits into
Open
Conversation
zsqdx
marked this pull request as ready for review
August 4, 2026 16:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kOBEYprecision constraints unchangedFixes #1228.
Root cause
setMemoryPoolLimit()excludes tactics that need more workspace than the configured limit. For the reported b11 transformer at batch 576, the 12-head 19x19 attention score tensor alone is about 1.68 GiB in FP16, so the fixed 1 GiB limit can leave a fused TensorRT node with no implementation.TensorRT already provides automatic device-dependent sizing: when the workspace limit is not explicitly set, it defaults to the target GPU's total memory. This is only a tactic-selection ceiling, not a preallocation; each execution context allocates the workspace required by its selected plan. NVIDIA documents both behaviors here: https://docs.nvidia.com/deeplearning/tensorrt/latest/reference/troubleshooting-faq.html#how-do-i-choose-the-optimal-workspace-size
This is preferable to moving the fixed limit from 1 GiB to another constant. On TensorRT 10.16.1 / RTX 5090, a 2 GiB cap fixes b576 but fails again at b1024, whereas the automatic default builds b576, b1024, and b2048.
Validation
Tested current master with TensorRT 10.16.1.11, CUDA 13.2, and 8x RTX 5090 using
b11c768h12nbt3tflrs-fson-silu:testgpuerror -quickalso passes 669 positions; batched and unbatched errors are identical and all margins remain within limits.The existing build mutex continues to serialize TensorRT engine builds, so allowing the device default does not multiply builder workspace across NN server threads.