Optimize run-end decode kernel with chunked splat stores#8915
Draft
joseph-isaacs wants to merge 13 commits into
Draft
Optimize run-end decode kernel with chunked splat stores#8915joseph-isaacs wants to merge 13 commits into
joseph-isaacs wants to merge 13 commits into
Conversation
The decode benchmark previously only covered the bool kernel. Add coverage for runend_decode_primitive across element widths and run lengths so the general-case decode loop can be measured. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
The shared primitive/varbinview decode loop paid a large per-run cost: the trimmed-ends iterator next() compiled to an outlined call per run, BufferMut::push_n_unchecked round-tripped the buffer length through memory on every run, and runs shorter than a vector register fell into a scalar one-element-at-a-time fill loop. The nullable arm additionally made an outlined BitBufferMut::append_n call per run and a memset call per null run. Rewrite runend_decode_slice to iterate the ends slice directly and expand each run with unconditional 32-byte splat stores through a raw cursor into a buffer over-allocated by one chunk, so short runs (the common case) complete in a single store with no tail loop. The nullable arm now prefills element validity with the majority run validity and only rewrites minority ranges, mirroring the bool decode strategy. runend_decode_typed_primitive now takes the raw ends slice plus offset instead of a pre-trimmed iterator. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
LLVM loop-idiom rewrote the byte-sized splat store loop into a memset call per run, so u8 decode saw almost no gain from the chunked-store rewrite. Replicate the element bytes into a u128 word with a runtime multiply and store two unaligned 16-byte words per chunk instead; the computed word is opaque to loop-idiom, and 8-byte elements also get full-width stores. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
The u128 word stores fixed byte-sized runs but compiled to GPR pair stores, halving fill bandwidth for long runs of every element width. Keep the unrolled element stores (which compile to full-width vector stores) for elements of two bytes and up, and use the opaque word splat only for single-byte elements, escaping to an explicit memset for byte runs of 256 bytes or more where memset wins. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Align run_end_decode with the conventions used by the other run-end benches: route allocation through vendored mimalloc (decode allocates its output inside the timed region, same rationale as run_end_compress) and generate run lengths, values, and validity from a seeded StdRng so branch-heavy decode strategies are measured against unpredictable inputs rather than a fixed pattern. Add a randomized differential test that checks runend_decode_primitive against a naive element-by-element reference expansion across u8/u16/u32/ u64/f32, covering offsets, zero-length runs, both sides of the byte-run memset threshold, run clamping, and non-nullable/mixed/all-invalid validity. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
The release-profile ablation benchmark showed the inline chunked splat loop losing its wide vector stores when monomorphized inside the register-heavy nullable decode arm: nullable u64 decode at 1024-element runs cost ~24us vs ~17us for the old push_n path, while the identical loop in the non-null arm showed no regression. Runs of at least 256 bytes now dispatch to a #[inline(never)] fill whose codegen cannot be polluted by the surrounding loop: write_bytes (memset) for single-byte elements, a dedicated vector fill loop otherwise. This also subsumes the previous byte-only memset threshold. Also add the temporary ablation benchmark used to attribute each decode change (iterator removal, chunked stores, byte word path, prefill validity) so the per-change numbers are reproducible. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
The release-profile ablation showed the per-run long-run threshold check taxing short runs (the general case) by ~1ns/run on wide element types. Restructure splat_run so the first chunk store is unconditional: runs no longer than one chunk finish on a single branch exactly like the plain chunked loop, and only multi-chunk runs pay the long-run check before dispatching to the out-of-line fill. With this, the same-binary ablation shows every type/run-length cell at parity or better against the original kernel, including nullable u64 long runs. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Merging this PR will regress 3 benchmarks
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decompress_utf8[(100000, 4096)] |
947.1 µs | 1,772.1 µs | -46.56% |
| ❌ | Simulation | decompress[u64, (100000, 4096)] |
401 µs | 728.4 µs | -44.95% |
| ❌ | Simulation | decompress[u8, (1000, 16)] |
17.8 µs | 24.3 µs | -26.71% |
| ⚡ | Simulation | decompress[u16, (100000, 4)] |
950.7 µs | 443.9 µs | ×2.1 |
| ⚡ | Simulation | runend_decompress_u32 |
3 ms | 1.4 ms | ×2.1 |
| ⚡ | Simulation | decompress[u32, (100000, 4)] |
1,053.9 µs | 567.8 µs | +85.63% |
| ⚡ | Simulation | decompress[u16, (10000, 4)] |
109.9 µs | 59.3 µs | +85.33% |
| ⚡ | Simulation | decompress[u8, (100000, 4)] |
836.5 µs | 461.3 µs | +81.35% |
| ⚡ | Simulation | decompress[u16, (100000, 16)] |
320.9 µs | 190.9 µs | +68.15% |
| ⚡ | Simulation | decompress[u32, (10000, 4)] |
128.7 µs | 80 µs | +60.94% |
| ⚡ | Simulation | take_indices[(100000, 4)] |
1,327.9 µs | 841.3 µs | +57.84% |
| ⚡ | Simulation | decompress[u8, (1000, 4)] |
31.2 µs | 20 µs | +55.76% |
| ⚡ | Simulation | decompress[u8, (100000, 16)] |
243.2 µs | 156.4 µs | +55.49% |
| ⚡ | Simulation | decompress[u64, (100000, 4)] |
1,278.3 µs | 832 µs | +53.65% |
| ⚡ | Simulation | decompress[u8, (10000, 4)] |
111.6 µs | 73.8 µs | +51.08% |
| ⚡ | Simulation | decompress[u64, (1000, 16)] |
31.2 µs | 20.8 µs | +49.97% |
| ⚡ | Simulation | decompress[u32, (100000, 16)] |
435.1 µs | 306.8 µs | +41.83% |
| ⚡ | Simulation | take_indices[(10000, 4)] |
171.8 µs | 121.9 µs | +40.96% |
| ⚡ | Simulation | decompress[u64, (10000, 4)] |
148.6 µs | 106.1 µs | +40.02% |
| ⚡ | Simulation | decompress[u16, (10000, 16)] |
46 µs | 33.2 µs | +38.77% |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/runend-decompression-perf-5yrfqs (57175be) with develop (d9619b8)
Footnotes
-
46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
robert3005
reviewed
Jul 23, 2026
| use vortex_buffer::BufferMut; | ||
| use vortex_mask::Mask; | ||
| use vortex_runend::compress::runend_decode_typed_primitive; | ||
| use vortex_runend::trimmed_ends_iter; |
Contributor
There was a problem hiding this comment.
I think we can remove this method now trimmed_ends_iter
joseph-isaacs
added
the
action/benchmark-sql-full
github-actions
Bot
removed
the
action/benchmark-sql-full
Contributor
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Contributor
Benchmarks: Vortex queries 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.124x ✅, 2↑ 0↓)
datafusion / parquet (0.142x ✅, 2↑ 0↓)
duckdb / vortex-file-compressed (0.164x ✅, 2↑ 0↓)
duckdb / parquet (0.139x ✅, 2↑ 0↓)
File Size Changes (1 files changed, -90.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.140x ❌ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.140x ❌, 1↑ 8↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.009x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.007x ➖, 0↑ 0↓)
datafusion / parquet (0.983x ➖, 2↑ 0↓)
datafusion / arrow (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.008x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.004x ➖, 0↑ 0↓)
duckdb / parquet (1.011x ➖, 0↑ 1↓)
duckdb / duckdb (1.010x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.042x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.007x ➖, 0↑ 0↓)
datafusion / parquet (1.020x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.036x ➖, 0↑ 1↓)
duckdb / vortex-compact (1.025x ➖, 0↑ 1↓)
duckdb / parquet (1.026x ➖, 0↑ 1↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.056x ➖, 0↑ 9↓)
datafusion / vortex-compact (1.021x ➖, 0↑ 4↓)
datafusion / parquet (1.051x ➖, 1↑ 11↓)
duckdb / vortex-file-compressed (1.055x ➖, 1↑ 19↓)
duckdb / vortex-compact (1.026x ➖, 0↑ 2↓)
duckdb / parquet (1.032x ➖, 0↑ 3↓)
duckdb / duckdb (1.048x ➖, 0↑ 5↓)
No file size changes detected. |
Contributor
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.068x ➖, 0↑ 1↓)
datafusion / vortex-compact (0.939x ➖, 0↑ 0↓)
datafusion / parquet (0.983x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.984x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.902x ➖, 1↑ 0↓)
duckdb / parquet (0.984x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Statistical and Population Genetics 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.979x ➖, 1↑ 0↓)
duckdb / vortex-compact (0.891x ✅, 4↑ 0↓)
duckdb / parquet (0.987x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.993x ➖, 1↑ 1↓)
datafusion / parquet (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.978x ➖, 2↑ 1↓)
duckdb / parquet (1.001x ➖, 0↑ 0↓)
duckdb / duckdb (0.967x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -0.0% overall, 99↑ 102↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.042x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.028x ➖, 0↑ 0↓)
datafusion / parquet (0.963x ➖, 0↑ 0↓)
datafusion / arrow (0.986x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (1.027x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.017x ➖, 0↑ 0↓)
duckdb / parquet (1.006x ➖, 0↑ 0↓)
duckdb / duckdb (1.015x ➖, 0↑ 0↓)
No file size changes detected. |
Contributor
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.922x ➖, 1↑ 0↓)
datafusion / vortex-compact (0.927x ➖, 1↑ 0↓)
datafusion / parquet (1.011x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.986x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.941x ➖, 0↑ 0↓)
duckdb / parquet (0.991x ➖, 0↑ 0↓)
|
Contributor
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.895x ✅, 21↑ 0↓)
datafusion / parquet (0.923x ➖, 13↑ 0↓)
duckdb / vortex-file-compressed (0.914x ➖, 17↑ 1↓)
duckdb / parquet (0.954x ➖, 1↑ 0↓)
duckdb / duckdb (0.957x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: Appian on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.990x ➖, 0↑ 0↓)
datafusion / parquet (0.983x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.997x ➖, 0↑ 0↓)
duckdb / parquet (1.000x ➖, 0↑ 0↓)
duckdb / duckdb (0.998x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Contributor
Benchmarks: TPC-H SF=10 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.049x ➖, 0↑ 1↓)
datafusion / vortex-compact (1.006x ➖, 0↑ 0↓)
datafusion / parquet (0.969x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed (1.087x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.071x ➖, 0↑ 0↓)
duckdb / parquet (1.031x ➖, 0↑ 0↓)
|
0ax1
reviewed
Jul 24, 2026
| // vendored mimalloc to keep glibc malloc (which varies across runner images) out of the | ||
| // measured trace. | ||
| #[global_allocator] | ||
| static GLOBAL: MiMalloc = MiMalloc; |
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
0ax1
reviewed
Jul 24, 2026
| /// # Safety | ||
| /// | ||
| /// `dst` must be valid for writes of `len` elements. | ||
| #[inline(never)] |
0ax1
reviewed
Jul 24, 2026
| /// | ||
| /// The allocation behind `base` must have room for at least | ||
| /// `max(pos, end) + decode_chunk_len::<T>()` elements. | ||
| #[inline(always)] |
Share the intermediate decode kernels (v0-v2, n2) in benches/shared/ decode_variants.rs so the existing ablation and a new distribution-sweep ablation measure byte-identical code. The new benchmark sweeps run length and element width (non-nullable and nullable) plus a validity-density sweep, reporting decoded-elements/sec throughput so each change can be attributed to the data distribution it is sensitive to: - v1/v0 (drop iterator chain + per-run buffer bookkeeping): a per-run fixed cost, so it only helps at short runs. - v2/v1 (chunked splat stores vs push_n_unchecked scalar fill): the large win for wide elements at short runs (up to ~5-7x on u32/u64 at run 2), fading to parity by run 64+. - v3/v2 (byte word-splat + memset threshold): the u8 short-run win. - n2/n0 and n3/n2 (majority-prefill validity): the density sweep shows the prefill advantage is smallest at 50/50 validity and grows as validity skews either way, symmetric about 50%. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
An earlier revision gave byte elements a dedicated splat path: store a replicated 0x0101.. word (whose runtime multiply is opaque to LLVM's loop-idiom pass) so short u8 runs would not collapse into a memset call per run. The later unconditional-first-chunk restructuring already prevents that collapse: a run no longer than one 32-byte chunk completes in the inline, fixed-length vector stores and never reaches the idiom-collapsible loop. That makes the word splat redundant, and a same-binary A/B shows it is also measurably *slower* than just letting bytes use the generic element path -- the wrapping_mul plus unaligned u64 stores lose to the 16-byte vector stores LLVM emits for the fixed-length element loop. For u8 the generic path is 3-26% faster across every run length (e.g. run 16: 6.9 -> 5.9us, run 64: 3.4 -> 2.7us); widths > 1 are unchanged. Remove the ~35 lines of byte-specific unsafe so all widths share one splat path. The rejected kernel is preserved as `nonnull_v3_byte_splat` in run_end_decode_distribution so the comparison stays reproducible. Verified: decode_matches_reference differential test still passes; u8 decode asm shows inline movdqu stores for short runs with memset only behind the long-run threshold. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
The long-run fill was an element loop, which the compiler emits at the baseline target width (16-byte SSE2 stores, movdqu x2 per iteration -- confirmed in the release asm). memcpy, by contrast, is resolved by the libc at runtime to the widest implementation the host supports (AVX2, or rep movsb on ERMS parts), so it reaches a store width the compiled loop cannot. Seed one cache line by hand, then repeatedly copy the filled prefix onto the tail. Each memcpy costs a call, so this only pays once the run is long enough to amortize it: an interleaved fill-level probe puts the crossover at ~2 KiB for u16/u32/u64 alike (below it doubling is up to 2x slower, at and above it is 1.1-1.4x faster), so the doubling path is gated on DOUBLING_FILL_BYTES. Byte elements keep plain memset, which the libc already dispatches well at every length. At decode level this is the first win at long runs, where every earlier change in this branch was at parity with the original kernel: u32 run 1024 1.38x, u32 run 4096 1.56x, u64 run 256 1.15x, u64 run 1024 1.22x, u64 run 4096 1.49x. Sub-threshold cells are unchanged (same code path). The previous element-loop fill is preserved as nonnull_v3_elem_fill in run_end_decode_distribution so the comparison stays reproducible, and the sweep now runs to 4096-element runs so every width crosses the threshold. The differential test gains a 3000-element run-length case so the doubling path is covered for every element width. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
A value whose bytes are all equal -- 0, or all-ones (-1 for signed integers), both common in real columns -- can be filled with a single memset. memset writes without reading anything back, so it beats the doubling fill, which re-reads the prefix it just wrote. Recognise such values with arithmetic on a same-sized integer rather than a walk over the value bytes, so nothing ever reads padding. This also subsumes the previous size_of::<T>() == 1 special case: a byte element is trivially byte-uniform, so it reaches the same memset it did before. Measured with a within-function data contrast (the same benchmark over zero-valued versus arbitrary-valued runs), which cancels the codegen-unit difference described below: +2% to +15% on byte-uniform runs past the fill threshold, median about +7%, and unchanged for arbitrary values. An isolated fill-level probe puts the effect higher (1.17-1.32x); the decode level number is the honest one to quote. Also document a measurement hazard found while validating this: nonnull_v3 and zeros_v3 call the real kernel across a crate boundary while every other variant is a bench-local copy that inlines further. That is worth up to ~40% at short runs on identical algorithms and always favours the bench-local copy, so shipped-versus-local comparisons are lower bounds. The header of run_end_decode_distribution now says so. The differential test now gives a quarter of runs the value zero so the memset path is covered for every element width. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The run-end decode kernel is a hot path for decompressing run-end encoded arrays. This change replaces the element-by-element
push_n_uncheckedapproach with chunked splat stores that write full cache lines at a time, improving throughput especially for short runs and byte-sized elements.Key improvements:
fill_runfunction to preserve vector codegen in the main decode loopfill_range_uncheckedWhat changes are included in this PR?
Core decode kernel (
encodings/runend/src/compress.rs):trimmed_ends_iteriterator chaintrim_end()helper to adjust run ends by offset and clamp to lengthsplat_run()function with specialized paths for byte vs. wider elementsfill_run()out-of-line helper for long runsrunend_decode_slice()to use slice iteration and chunked stores instead ofpush_n_uncheckedrunend_decode_typed_primitive()signature to acceptrun_ends: &[E]andoffsetparametersBenchmarks:
run_end_decode_ablation.rsbenchmark isolating each optimization step (v0 original → v1 slice loop → v2 chunk stores → v3 shipped)run_end_decode.rswith primitive array benchmarks across varying run lengths and nullable/non-nullable casesWhat APIs are changed? Are there any user-facing changes?
The public API
runend_decode_typed_primitive()signature changed:runend_decode_typed_primitive(run_ends: impl Iterator<Item = usize>, ...)runend_decode_typed_primitive(run_ends: &[E], offset: usize, ...)This is an internal API used only by the run-end encoding module, so there are no user-facing changes. The public
runend_decode_primitive()function signature remains unchanged.https://claude.ai/code/session_013SG5c8pTHVHCSqvDZy6KFv