fix: read string-dict offsets at their true ptype width (#215)#218
Merged
Conversation
A `vortex.dict` string column whose values are FSST-compressed crashed the scan with a raw `IndexOutOfBoundsException` in `VarBinArray$DictMode.dictReadOff`. Root cause: `DictEncodingDecoder.decodeUtf8DictProto` ran the decoded values through `VarBinArray.toOffsetMode` and then hardcoded `PType.I64` as the offsets ptype of the resulting `DictMode`. But `toOffsetMode` only builds fresh I64 offsets on its slow path — on the fast path it returns the already-decoded `OffsetMode` unchanged, keeping its own ptype. FSST decompresses its values to a child with I32 offsets (Rust: FSST's `codes_offsets_ptype` is a narrow wire ptype read during decompression — see encodings/fsst/src/array.rs — while the canonicalized value offsets are freshly built and independent of it). The DictMode carrier thus claimed an 8-byte stride over a 4-byte-stride buffer (12 bytes for 3 offsets), so reading offset 1 as an 8-byte long walked off the segment. Fix: carry `dictValues.offsetsPtype()` so the carrier matches the buffer it actually materialized (parse-don't-validate). Additionally harden `dictReadOff` to read every signed/unsigned offset width at its true stride and to throw a helpful `VortexException` on out-of-range indices rather than a bare `IndexOutOfBoundsException` (ADR 0003 bounds discipline). Corpus evidence: uci-magic-gamma-telescope (Raincloud, vortex-data 0.69.0) — its utf8 `class` column is a `vortex.dict` over `vortex.fsst` values. Export now completes and matches the Parquet sibling on all 19020 rows x 11 cols. Closes #215 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #215, found by Raincloud triage round 2 (#205) on
uci-magic-gamma-telescope.Root cause — the decoder carrier, not the buffer:
DictEncodingDecoder.decodeUtf8DictProtopassed a hardcodedPType.I64offsets ptype toVarBinArray.ofDict, buttoOffsetMode's fast path returns the decoded array unchanged with its own ptype — FSST decompresses to I32 offsets (12-byte buffer, 4-byte stride).DictMode.dictReadOffthen did 8-byte reads over a 4-byte-stride buffer → rawIndexOutOfBoundsException. Per the Rust reference (encodings/fsst/src/array.rs+canonical.rs), the narrowcodes_offsetswire ptype is consumed only during decompression; the canonicalized value offsets are authoritative.Fix (both remedies): the carrier now uses
dictValues.offsetsPtype()so ptype and buffer stride agree, anddictReadOffreads every signed/unsigned offset width at its true stride with aVortexExceptionbounds guard replacing the bare IOOBE (ADR 0003 discipline) — this also retires the narrowly-scoped IOOBE arm in the conformance harness's gap catch.Validation:
VarBinArrayTestDict group 3 → 12 tests (all 8 offset ptypes parameterized + the exact #215 mislabeled-width guard); reader suite 1113 green; magic-gamma exports all 19,020 rows and matches the Parquet oracle on all 209,220 cells — matrix entry flipped took(19 ok / 1 gap / 227 untriaged).Closes #215
🤖 Generated with Claude Code