Skip to content

feat: add FP8-K NVFP4-V KV cache support - #18206

Draft
meenchen wants to merge 9 commits into
NVIDIA:mainfrom
meenchen:fp8-k-nvfp4-v-mixed-kv-integration
Draft

feat: add FP8-K NVFP4-V KV cache support#18206
meenchen wants to merge 9 commits into
NVIDIA:mainfrom
meenchen:fp8-k-nvfp4-v-mixed-kv-integration

Conversation

@meenchen

@meenchen meenchen commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Description

Draft integration for an FP8-K / NVFP4-V paged KV cache in the PyTorch runtime.

  • adds the mixed cache quantization mode and argument validation
  • stores compact FP8 K plus packed NVFP4 V and V block scales
  • routes separate K/V buffers and scale metadata to FlashInfer TRTLLM-Gen FMHA
  • sizes the mixed cache correctly in the hybrid cache manager
  • preserves CUDA-graph-stable cache views and counter buffers

Kernel dependencies:

Validation

  • rebased onto current TensorRT-LLM main
  • Python syntax checks pass
  • focused unit tests are included for quantization mode, page indexing, cache sizing, and hybrid cache allocation
  • end-to-end consumer validation remains gated on publishing an ABI-coherent TRTLLM-Gen/FlashInfer cubin package

This is intentionally a draft until the kernel and package dependencies land.

Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Signed-off-by: weimingc <17592131+meenchen@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant