fix: null rows silently decoded as values (three validity representations dropped) + NullArray CSV export

.github/workflows/raincloud-conformance.yml

@@ -0,0 +1,64 @@
+name: Raincloud conformance
+
+# Real-world cross-implementation conformance (issue #205): hydrates the Raincloud
+# corpus (Vortex files written by the Python bindings + Parquet oracles) and runs
+# RaincloudConformanceIntegrationTest against everything hydrated. Scheduled, not on
+# the PR path: hydration depends on upstream dataset URLs that rot independently of
+# our code, so a failure here is a finding to triage, not a broken build.
+
+on:
+  schedule:
+    - cron: '0 5 * * 1'   # Mondays 05:00 UTC
+  workflow_dispatch:
+    inputs:
+      max-mb:
+        description: 'Per-slug artifact size cap in MB (0 = uncapped)'
+        default: '200'
+
+jobs:
+  conformance:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+
+      - name: Set up Azul Zulu JDK 25
+        uses: actions/setup-java@v5
+        with:
+          distribution: zulu
+          java-version: '25'
+
+      - name: Cache Maven repository
+        uses: actions/cache@v6
+        with:
+          path: ~/.m2/repository
+          key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
+          restore-keys: |
+            ${{ runner.os }}-maven-
+
+      - uses: actions/setup-python@v6
+        with:
+          python-version: '3.12'
+
+      # keyed on the matrix + pinned tag: a corpus change rebuilds, otherwise
+      # weekly runs reuse artifacts instead of re-fetching upstream sources
+      - name: Cache Raincloud corpus
+        uses: actions/cache@v6
+        with:
+          path: ~/.cache/raincloud
+          key: raincloud-corpus-${{ hashFiles('scripts/hydrate-raincloud-corpus.sh', 'integration/src/test/resources/raincloud/expected-status.csv') }}
+
+      - name: Hydrate corpus
+        run: scripts/hydrate-raincloud-corpus.sh --max-mb "${{ github.event.inputs.max-mb || '200' }}"
+
+      - name: Run conformance suite
+        run: ./mvnw verify -pl integration -am -Dit.test=RaincloudConformanceIntegrationTest
+
+      - name: Publish per-slug results to job summary
+        if: always()
+        run: |
+          {
+            echo '## Raincloud conformance'
+            echo '```'
+            grep -hE "Tests run|slug" integration/target/failsafe-reports/*.txt || true
+            echo '```'
+          } >> "$GITHUB_STEP_SUMMARY"

CHANGELOG.md

@@ -11,6 +11,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - CSV export renders unsigned integer columns (U8–U64) with their unsigned values — high-half values previously printed as two's-complement negatives (uci-wine `magnesium` U8 132 exported as -124), silent corruption found by the Raincloud conformance suite. ([#208](https://github.com/dfa1/vortex-java/issues/208))
 - `fastlanes.rle` decodes F64/F32 value pools (`LazyRleDoubleArray`, `LazyRleFloatArray`) — files from the Python bindings RLE-encode double columns with long constant runs, which previously failed to scan with `unsupported ptype F64`. ([#209](https://github.com/dfa1/vortex-java/issues/209))
+- Lazy dict decode now covers I8/U8/I16/U16 value columns (`DictByteArray`, `DictShortArray`) — files from the Python bindings dict-encode narrow-integer columns, which previously failed to scan with `unsupported ptype for lazy dict`. ([#206](https://github.com/dfa1/vortex-java/issues/206))
+- CSV export handles all-null (`DType.Null`) columns as empty fields instead of throwing `unsupported array type: NullArray`. ([#211](https://github.com/dfa1/vortex-java/issues/211))
+- Null rows no longer silently decode as values: wrapper decoders now propagate the row validity that files from the Python bindings carry deep in the encoding tree. Three representations were dropped — a trailing validity child on `fastlanes.bitpacked` (reached through `vortex.alp`/`vortex.zigzag`/`fastlanes.for`, which delegate validity to their encoded child per the Rust `ValidityChild` contract), dict pools with invalid slots, and dict codes with their own validity — across both the eager `vortex.dict` decoder and the lazy dict layout path. Found on real data: penguins and kepler exported invented values (`32.1`, `0.0`) for thousands of null cells. ([#210](https://github.com/dfa1/vortex-java/issues/210))
+
+### Added
+
+- Real-world conformance suite against the [Raincloud](https://github.com/spiraldb/raincloud) corpus: 247 public datasets whose Vortex files are written by the Python bindings, each validated value-for-value against its Parquet sibling. `scripts/hydrate-raincloud-corpus.sh` hydrates any subset (cache → mirror → local build), `RaincloudConformanceIntegrationTest` tests whatever is hydrated against the checked-in per-dataset status matrix, and a weekly workflow runs a size-capped sweep. First triage found four reader gaps on real data, including one silent-corruption bug (unsigned integers rendered as signed). ([#205](https://github.com/dfa1/vortex-java/issues/205))
 
 ## [0.12.0] — 2026-07-04
 

CLAUDE.md

@@ -63,6 +63,8 @@ Trunk-based. PRs fine but always squash or rebase — no merge commits. Keep com
 ./mvnw verify -pl integration -am          # integration (failsafe, NOT surefire)
 ./mvnw verify -pl integration -am -Dit.test="RustWritesJavaReadsIntegrationTest#method"
 ./bench JavaVsJniReadBenchmark.javaReadVolume   # benchmark — always ClassName.methodName filter
+scripts/hydrate-raincloud-corpus.sh --max-mb 200   # hydrate real-world conformance corpus (#205),
+                                                   # then verify -Dit.test=RaincloudConformanceIntegrationTest
 ```
 
 Regenerate after editing `.fbs`/`.proto` (both generators are in-house, no external tools):

csv/src/main/java/io/github/dfa1/vortex/csv/CsvExporter.java

@@ -13,6 +13,7 @@
 import io.github.dfa1.vortex.reader.array.LongArray;
 import io.github.dfa1.vortex.reader.array.ShortArray;
 import io.github.dfa1.vortex.reader.array.MaskedArray;
+import io.github.dfa1.vortex.reader.array.NullArray;
 import io.github.dfa1.vortex.reader.array.VarBinArray;
 import io.github.dfa1.vortex.reader.VortexReader;
 import io.github.dfa1.vortex.reader.ScanIterator;
@@ -159,6 +160,9 @@ private static String cellValue(Array arr, long rowIdx) {
             // Nullable columns decode as MaskedArray: null rows export as an empty field, valid
             // rows defer to the inner values array.
             case MaskedArray ma -> ma.isValid(rowIdx) ? cellValue(ma.inner(), rowIdx) : "";
+            // All-null columns (DType.Null) hold only a row count: every cell is an empty
+            // field, same rule as a MaskedArray null row.
+            case NullArray ignored -> "";
             default -> throw new VortexException(
                     "unsupported array type for CSV export: " + arr.getClass().getSimpleName());
         };

csv/src/test/java/io/github/dfa1/vortex/csv/CsvExporterTest.java

@@ -5,7 +5,9 @@
 import io.github.dfa1.vortex.core.model.PType;
 import io.github.dfa1.vortex.writer.VortexWriter;
 import io.github.dfa1.vortex.writer.WriteOptions;
+import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder;
 import io.github.dfa1.vortex.writer.encode.NullableData;
+import io.github.dfa1.vortex.writer.encode.PrimitiveEncodingEncoder;
 import org.junit.jupiter.api.Test;
 import org.junit.jupiter.api.io.TempDir;
 
@@ -71,6 +73,32 @@ void exportsToWriter(@TempDir Path tmp) throws Exception {
         assertThat(lines[2]).isEqualTo("2.7");
     }
 
+    @Test
+    void exportsAllNullColumnAsEmptyFields(@TempDir Path tmp) throws Exception {
+        // Given a file mixing a value column with an all-null (DType.Null) column —
+        // the shape that made real-world exports throw (#211); the long[] carrier for
+        // the null column only supplies the row count
+        Path vortex = tmp.resolve("data.vortex");
+        DType.Struct schema = new DType.Struct(
+                List.of(ColumnName.of("id"), ColumnName.of("empty")),
+                List.of(DType.I64, DType.NULL),
+                false);
+        try (FileChannel ch = FileChannel.open(vortex, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             // explicit encoder list: the default cascading set has no DType.Null codec
+             VortexWriter writer = VortexWriter.create(ch, schema, WriteOptions.defaults(),
+                     List.of(new NullEncodingEncoder(), new PrimitiveEncodingEncoder()))) {
+            writer.writeChunk(Map.of(ColumnName.of("id"), new long[]{1L, 2L}, ColumnName.of("empty"), new long[2]));
+        }
+        Path csv = tmp.resolve("out.csv");
+
+        // When
+        CsvExporter.exportCsv(vortex, csv);
+
+        // Then — null cells are empty fields, same as MaskedArray null rows
+        List<String> result = Files.readAllLines(csv);
+        assertThat(result).containsExactly("id,empty", "1,", "2,");
+    }
+
     @Test
     void suppressesHeaderWhenConfigured(@TempDir Path tmp) throws Exception {
         // Given

docs/compatibility.md

@@ -43,6 +43,36 @@ resolves only the standalone decoders in `reader`; no encoder class is loaded.
 | Duplicate struct field names | Rust writer rejects ("StructLayout must have unique field names"); Rust reader tolerates foreign files (first-match access) | ⚠️ Deliberate divergence on read: Java rejects such files with `VortexException("duplicate field name in file schema")` instead of tolerating them — the name-keyed `Chunk` API cannot represent both columns, and silent column loss is worse than a loud failure on a file the reference writer refuses to produce. Java's writer mirrors the Rust writer's rejection. |
 | Blank / control-character field names | Wire-legal; the Rust writer produces `""` and whitespace-only names. NUL (`U+0000`) additionally aborts the Rust toolchain: Arrow FFI schema export hits a panic-cannot-unwind in `arrow-rs` (`ffi_stream::get_schema`) and SIGABRTs the process (measured against vortex-jni 0.75.0) | ⚠️ Deliberate strictness BOTH ways: vortex-java's writer refuses blank and control-character field names (`IllegalArgumentException`), and its reader rejects files carrying them (`VortexException` naming the producing pipeline as the likely bug) — the JSON-`""`-key principle: wire-legal is a floor, not a policy. Printable names of any shape (`$`-runs, spaces inside, emoji) are legal and round-trip intact both directions (measured; pinned by `ColumnNameEdgeCasesIntegrationTest`). |
 
+## Real-world conformance: the Raincloud corpus
+
+Cross-implementation conformance against [Raincloud](https://github.com/spiraldb/raincloud)
+(tracked in [#205](https://github.com/dfa1/vortex-java/issues/205)): 247 curated public
+datasets whose `.vortex` files are written by the Vortex **Python** bindings
+(`vortex-data 0.69.0`, pinned via Raincloud `v0.2.1`) with a Parquet sibling built from the
+same Arrow table. `RaincloudConformanceIntegrationTest` reads each hydrated `.vortex` with
+vortex-java, exports CSV, and diffs it against the Parquet sibling read by hardwood (the
+oracle) — a zero-diff proves every value survived the Python-write / Java-read boundary.
+
+```bash
+scripts/hydrate-raincloud-corpus.sh --max-mb 200      # cache → mirror → local build
+./mvnw verify -pl integration -am -Dit.test=RaincloudConformanceIntegrationTest
+```
+
+Per-slug status lives in `integration/src/test/resources/raincloud/expected-status.csv`
+(`ok` must pass; `gap:<issue>` must still fail, so a fix flips the entry in the same change;
+`untriaged` runs and reports without failing the build). A scheduled workflow
+(`raincloud-conformance.yml`) hydrates a size-capped subset weekly. Current triage —
+18 `ok`, 2 known gaps: nested struct columns in scan
+([#207](https://github.com/dfa1/vortex-java/issues/207)), narrow dict-offset ptypes in
+string dicts ([#215](https://github.com/dfa1/vortex-java/issues/215));
+227 slugs untriaged. Fixed so far by this suite: lazy dict U8/U16 values
+([#206](https://github.com/dfa1/vortex-java/issues/206)), unsigned integers rendered signed
+([#208](https://github.com/dfa1/vortex-java/issues/208) — silent corruption), RLE over F64
+([#209](https://github.com/dfa1/vortex-java/issues/209)), row validity dropped by wrapper
+decoders ([#210](https://github.com/dfa1/vortex-java/issues/210) — silent corruption),
+all-null columns in CSV export
+([#211](https://github.com/dfa1/vortex-java/issues/211)).
+
 ## Encodings
 
 | Encoding ID                 | Decoder                          | Encoder                          | Decode | Encode | Notes                                                                 |