fix: scan nested struct columns as full-range chunk sources (#207)#219
Merged
Conversation
`countries-of-the-world` (vortex-data 0.69.0) has a top-level struct column `data` that is itself a `vortex.struct` layout. Scan threw `VortexException: scan: column 'data' has 0 flats but the widest column has 1` from `ScanIterator.buildChunks`; `schema`/`count`/`inspect` were unaffected. Root cause: `ScanIterator.collectFlats` recognized flat, dict, zoned, and chunked layouts but not struct. A nested struct column's top layout is `vortex.struct`, so it collected zero leaves and chunk planning saw a 0-vs-N width it refused. Fix: treat a nested `vortex.struct` layout as a single full-range chunk source (like a dict leaf) rather than descending into its per-field children — those children are separate physical columns, not row chunks, and span the same row range as the parent. This mirrors the Rust reference (`vortex-layout/src/layouts/struct_/reader.rs`: `StructReader` decodes all field children over the parent's full range and inserts a leading `Bool` validity child only when the struct dtype is nullable). Decode routes through the layout registry's new `StructLayoutDecoder`, which reassembles a `StructArray` from the field children (masking each field with the struct-level validity when present). `sliceArray` gains a `StructArray` arm so the shared full-range struct column slices per chunk. Evidence on the real file: `count` = 262, `select region country_code` renders correct values, and the #207 chunk-planning exception is gone — the scan decodes the struct column into a `StructArray`. CSV export of the struct column itself is still unsupported (a separate rendering limitation, not chunk planning): `CsvExporter.cellValue` throws its clear `unsupported array type: StructArray`. Closes #207 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dfa1
force-pushed
the
fix/207-nested-struct
branch
from
841495a to
53e7c0e
Compare
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #207, found by the Raincloud conformance suite (#205) on
countries-of-the-world.Root cause:
ScanIterator.collectFlatsrecognizedflat/dict/zoned/chunkedlayouts but notstruct, so a top-level column that is itself a struct collected zero chunk leaves and chunk planning rejected the 0-vs-N mismatch.Fix (mirrors the Rust
StructReader: field children decode over the parent's full row range; a nested struct is never chunked independently):collectFlatstreats a nested struct layout as a single full-range chunk source;StructLayoutDecoder(registered inLayoutRegistry.registerDefaults()) reassembles aStructArrayviactx.decodeChild, applying struct-level validity to fields when nullable;ScanIterator.sliceArrayslicesStructArrayper chunk by recursively slicing each field.Zone-map pruning and filtered scans keep recognizing built-in layouts only (documented SPI scope) — nested struct columns plan conservatively.
Validation: 5 new tests (
StructLayoutDecoderTest+ registry default); reader suite green; full integrationverifygreen (no scan regressions).countries-of-the-worldnow scans andselects correctly; its CSV export stops at a separate, newly-split gap —CsvExporterhas noStructArrayrendering — filed as #217 and the matrix entry reclassifiedgap:207 → gap:217. Conformance suite 21/21.Closes #207
🤖 Generated with Claude Code