[GH-3115] Add netcdf.metadata data source for NetCDF file metadata#2829
Merged
Conversation
Add a read-only Spark DataSourceV2 (`netcdf.metadata`) that extracts NetCDF metadata -- dimensions, variables, attributes, and spatial extent -- without reading data arrays, mirroring the geotiff.metadata source. Reading: - Ranged remote reads through a Hadoop-backed ucar RandomAccessFile with a random read-policy hint, so only the header and 1-D coordinate arrays are transferred, never the whole file. - Column pruning: path/driver/fileSize come from the file listing without opening the file; coordinate arrays are read only when the extent is requested. NetCDF-native schema: - dimensions[], variables[] (dimensions, shape, CF units/long_name/ standard_name, _FillValue), and a globalAttributes map. - geoTransform/cornerCoordinates are nullable; geoTransform is emitted only for evenly spaced grids (best-fit-line test, robust to float32 coordinate quantization), and irregular grids report a centers-only extent. - crs/srid resolved from the CF grid_mapping variable (crs_wkt/spatial_ref), falling back to the equivalent global attributes. Correctness: - Per-file partition values for bin-packed partitions. - Scan equality accounts for the pushed limit. - Unsigned attributes are widened to their true value. - Non-numeric or non-finite coordinates degrade to a null extent. Closes apache#3115. The NetCDF counterpart of the geotiff.metadata source (apache#2846). Tested against classic NetCDF-3 and NetCDF-4/HDF5 fixtures; docs and nav added.
jiayuasu
force-pushed
the
netcdf-sedonainfo-support
branch
from
July 17, 2026 08:48
2e39b87 to
d2ce3e5
Compare
jiayuasu
changed the title
…scoping, CRS via proj4sedona
Grid and extent correctness:
- Skip CF bounds/climatology variables when selecting the grid-defining variable
and prefer variables whose trailing dimensions have resolvable 1-D numeric
coordinate variables, so lat_bnds(lat, nv) can no longer hijack the grid.
- Resolve coordinate variables and grid_mapping references in the data
variable's group and its ancestors (NUG scoping) instead of a global
short-name search, fixing wrong bindings in nested-group files.
- Decode packed coordinates: apply CF scale_factor/add_offset and widen
_Unsigned integer storage, so transforms are in real-world units.
- Tighten the regularity test: strict monotonicity plus a per-step tolerance
(0.1% relative, floored at a few ulps of the stored magnitude) plus the
half-pixel best-fit bound, so locally uneven grids like [0, 1.4, 2] no
longer receive an affine transform while fine float32 grids still do.
CRS (minimal CF support):
- SRID lookup via proj4sedona instead of GeoTools, removing this data
source's only dependency on the optional LGPL GeoTools runtime.
- Parse the extended grid_mapping form ("crs: lat lon").
- Infer EPSG:4326 for a latitude_longitude grid mapping with no explicit
ellipsoid/datum parameters when the file carries no CRS WKT.
- Parameter-only projected grid mappings remain untranslated (documented).
Data source behavior:
- fallbackFileFormat now returns a stub FileFormat that raises a clear
UnsupportedOperationException for catalog paths (CREATE TABLE ... USING),
which previously threw a NullPointerException.
- Directory handling: defaults never override explicit user options
(recursiveFileLookup=false keeps Hive-style partition discovery), and the
NetCDF extension filter now also applies to multi-directory loads.
- readToByteChannel drains channels that accept partial writes.
Adds 9 tests (26 total) and 5 fixtures covering each fix; documentation
updated, including CRS resolution rules and downstream RS_SetSRID/RS_SetCRS
usage.
…g, option handling CRS: - EPSG:4326 is no longer inferred from the latitude_longitude mapping name alone; it now requires positive WGS 84 identification (datum/CRS name or WGS 84 ellipsoid parameters, with a Greenwich prime meridian). - grid_mapping references resolve absolute (/crs) and relative (sub/crs, ../crs) group paths in addition to proximity-searched plain names. Grid and coordinates: - Grid selection also excludes variables referenced by ancillary_variables, coordinates, cell_measures, and formula_terms (token-split, keyed-form aware), so declared ancillary variables cannot shadow the data variable. - Coordinate lookup adds a lateral fallback matched strictly by dimension identity, finding NUG coordinate variables that live outside the data variable's lineage (e.g. a sibling group) while still rejecting identically named dimensions from unrelated groups. - _FillValue/missing_value honor the owning variable's _Unsigned semantics (byte -1 with _Unsigned reports 255, matching the enhanced value). RS_FromNetCDF interop: - NetCdfReader resolves full-path variable names (e.g. "sub/temp", as reported by netcdf.metadata for nested groups) and binds coordinate variables from the record variable's group before falling back to the whole-file search, making the documented discovery-to-load workflow work for nested files. Data source options: - Explicit option checks are case-insensitive, so mixed-case spellings such as RecursiveFileLookup=false reliably suppress the defaults. - A user-supplied pathGlobFilter disables the glob-path rewrite so Spark applies both the path glob and the filter natively. - The unsupported-fallback FileFormat now fails on instantiation, rejecting catalog DDL that supplies an explicit schema (which skips inferSchema). - readToByteChannel returns the transferred count on zero-progress channel writes instead of busy-spinning. Adds 7 tests (33 total) and 3 fixtures; extends 3 fixtures; documentation updated for the revised CRS and reference-resolution semantics.
…selection, loader parity
CRS:
- EPSG:4326 now requires positive WGS 84 identification by datum/CRS *name*;
ellipsoid parameters can only veto, never qualify, so datums such as CR05
that use the WGS 84 ellipsoid are no longer misreported as WGS 84.
- The expanded grid_mapping form with multiple mappings ("geographic: lat lon
projected: x y") selects the mapping whose coordinate list matches the
grid's (y, x) coordinates, so the reported CRS always corresponds to the
coordinates used for the extent; an unmatched multi-mapping is ambiguous
and reports no CRS.
Grid selection and schema:
- CF metadata references are resolved from their owning variable and
excluded by variable identity, so a path reference like /aux/temperature
no longer shadows an unrelated root variable sharing its basename.
- Variable dimension names are qualified with their declaring group's path
(sub/temp reports [sub/lat, sub/lon]), joinable against the dimensions
inventory.
- Native unsigned 64-bit coordinates decode as unsigned; values at or above
2^63 no longer surface as negative.
RS_FromNetCDF parity (NetCdfReader):
- Coordinate variables are matched by dimension identity in both the scoped
and lateral searches, so a same-named variable over an unrelated dimension
(declared earlier in the file) can no longer silently shrink the raster.
- Non-spatial (band) dimension variables resolve in the record variable's
scope with the same identity validation, fixing nested files whose band
dimension shadows a root dimension of a different length.
- Coordinate reads decode _Unsigned, scale_factor, and add_offset, so packed
files georeference identically in the loader and the metadata reader; the
element reader also accepts unsigned NetCDF-4 types.
Adds 8 tests (40 total) and 6 fixtures; documentation updated for the
stricter datum inference, mapping selection, and qualified dimension names.
… identity, ensemble datum
RS_FromNetCDF (NetCdfReader):
- Coordinate candidates are validated against the specific requested
dimension, not just any dimension of the record variable, so a same-named
variable over a different axis (e.g. a local lat(time) next to the true
lat(lat)) can no longer silently reshape the raster.
- Path-resolved coordinate names are validated the same way; explicitly
naming coordinates over unrelated dimensions now fails loudly instead of
returning a mis-sized raster. The long form of getRaster keeps accepting
any record dimension, since the user names the coordinate variables and
the axis is derived from the variable itself.
netcdf.metadata:
- Expanded grid_mapping clauses resolve their coordinate references relative
to the data variable and match the grid's actual coordinate variables by
identity, so a clause referencing same-named coordinates in another group
(/geo/x vs the root x) is never selected by basename overlap. When the
grid coordinates cannot be resolved, basename matching against the grid
dimension names remains as a best-effort fallback.
- The EPSG:6326 ensemble name ("World Geodetic System 1984 ensemble") is
recognized as a WGS 84 identifier.
Adds 4 tests (44 total) and 3 fixtures.
… CRS name vetoes, lateral clause matching RS_FromNetCDF (NetCdfReader): - The long form of getRaster validates each explicitly named coordinate against its intended spatial axis: the lat coordinate must be attached to the record variable's Y (second-to-last) dimension and the lon coordinate to its X (last) dimension, so non-spatial, duplicate, or swapped coordinates are rejected instead of silently reshaping the raster. - Axis slice indices are derived from the coordinate variables' actual dimension identities rather than name matching, and the band/plane split compares axis indices instead of dimension names, so a band dimension sharing a coordinate's name can no longer be miscounted. netcdf.metadata: - Explicit reference_ellipsoid_name and prime_meridian_name values that contradict WGS 84 (e.g. GRS 1980, Paris) veto the EPSG:4326 inference, consistent with the documented any-disagreeing-identifier rule. - Expanded grid_mapping clause references that are plain names not visible in the data variable's scope fall back to the same CF-aware lateral resolution used for the grid coordinates themselves, so a mapping tied to sibling-group coordinates (e.g. /coords/x referenced as "x") is selected correctly. Adds 5 tests (49 total) and 3 fixtures.
… axes, repeated dimensions netcdf.metadata: - An expanded grid_mapping clause — including a single one — is selected only when its resolved coordinate set covers both of the grid's (y, x) coordinate variables. CF's extended form explicitly scopes a mapping to its listed coordinates, so a "geographic: lat lon" clause tied to 2-D auxiliary coordinates no longer labels the 1-D x/y metre extent with a geographic CRS; the extent is still reported, unlabeled. RS_FromNetCDF (NetCdfReader): - The four-argument overload now implements its documented contract for non-trailing axes: each named coordinate prefers a candidate serving the default trailing axis and otherwise accepts one attached to any record dimension, the axes are derived from the coordinates' actual dimension identities, and the raster plane is extracted with stride-aware permuted indexing. Every remaining dimension becomes a band dimension in row-major order; for trailing axes the arithmetic reduces exactly to the previous contiguous-plane form. - Repeated dimensions in a variable's shape resolve to the default trailing occurrence when applicable and are rejected as ambiguous otherwise, and the band/plane split works by axis index throughout, fixing band metadata for shapes like temp(n, y, n). Adds 3 tests and reworks 1 (52 total, with value-level assertions verifying the permuted plane walk), plus 3 fixtures.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new read-only Spark DataSourceV2, netcdf.metadata, to expose NetCDF file metadata (dimensions, variables, attributes, CRS/extent) efficiently—designed to avoid loading data arrays and to support ranged reads for cloud/object storage, analogous to the existing geotiff.metadata source.
Changes:
- Introduces
netcdf.metadataDataSourceV2 implementation (table/scan/reader + Hadoop-backed random-access reader for ranged reads). - Adds comprehensive Spark-side and common-module NetCDF reader tests and extends NetCDF raster reading logic (coordinate resolution, axis permutation, packed/unsigned handling).
- Registers the data source and documents usage/schema in the Sedona docs site.
Reviewed changes
Copilot reviewed 12 out of 42 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/NetCdfMetadataDataSource.scala | Implements the netcdf.metadata data source and directory/glob default behaviors; provides V1 fallback rejection with a descriptive error. |
| spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/NetCdfMetadataTable.scala | Defines the table and the NetCDF-native output schema for the data source. |
| spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/NetCdfMetadataScanBuilder.scala | Implements scan building, LIMIT pushdown handling, and scan equality semantics. |
| spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/NetCdfMetadataPartitionReaderFactory.scala | Creates per-file readers while preserving per-file Hive partition values even in bin-packed Spark partitions. |
| spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/NetCdfMetadataPartitionReader.scala | Extracts NetCDF header metadata and (optionally) coordinate arrays to compute extent/transform/CRS with CF-aware scoping rules. |
| spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/netcdfmetadata/HadoopRandomAccessFile.scala | Adds a Hadoop FSDataInputStream-backed random-access implementation intended to encourage ranged reads on object stores. |
| spark/common/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister | Registers NetCdfMetadataDataSource for Spark format lookup. |
| spark/common/src/test/scala/org/apache/sedona/sql/netcdfMetadataTest.scala | Adds extensive end-to-end tests for schema fields, CRS/extent logic, column pruning, LIMIT pushdown, and ranged-read behaviors. |
| common/src/main/java/org/apache/sedona/common/raster/netcdf/NetCdfReader.java | Enhances NetCDF raster extraction logic (explicit coordinate validation, permuted axes, unsigned/packed decoding, missing/validity rules). |
| common/src/test/java/org/apache/sedona/common/raster/RasterConstructorsTest.java | Adds regression tests for packed band metadata, missing-value behavior, and int64 validity comparisons. |
| docs/tutorial/files/netcdfmetadata-sedona-spark.md | Documents the new netcdf.metadata source, schema semantics, CRS/extent rules, and usage patterns. |
| mkdocs.yml | Adds the new NetCDF metadata doc page to the site navigation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes Add netcdf.metadata data source for NetCDF file metadata #3115 (the NetCDF counterpart of thegeotiff.metadatasource added in [GH-2824] Add geotiff.metadata data source for GeoTIFF file metadata #2846).What changes were proposed in this PR?
Add a read-only Spark DataSourceV2 (
netcdf.metadata) that extracts NetCDF file metadata without reading data arrays, the NetCDF counterpart togeotiff.metadata(#2846).Usage
Output schema
One row per file, with a NetCDF-native schema rather than a copy of the GeoTIFF one:
pathdriver"NetCDF"fileSizeformatNetCDF(classic) /NetCDF-4width,heightsrid,crsgrid_mappingvariablegeoTransformcornerCoordinatesdimensionsvariablesglobalAttributesKey implementation details
ucar.unidata.io.RandomAccessFilewith a random read-policy hint (fs.option.openfile.read.policy=random), so object stores (S3A) transfer only the header and 1-D coordinate arrays via range requests — never the full file. Classic NetCDF headers are contiguous (a few requests); NetCDF-4/HDF5 metadata is scattered (more, smaller requests).path/driver/fileSizecome from the file listing without opening the file; coordinate arrays are read only whengeoTransform/cornerCoordinatesis requested.geoTransformis emitted only when coordinates lie within half a pixel of the best-fit line (robust to float32 quantization); irregular grids report a centers-only extent, matchingRS_FromNetCDFconventions.grid_mappingvariable (crs_wkt/spatial_ref), then global-attribute fallback; SRID via a GeoTools EPSG lookup.newWriteBuilderthrowsUnsupportedOperationException.Files
spark/common/.../io/netcdfmetadata/—NetCdfMetadataDataSource,NetCdfMetadataTable,NetCdfMetadataScanBuilder,NetCdfMetadataPartitionReaderFactory,NetCdfMetadataPartitionReader,HadoopRandomAccessFileMETA-INF/services/org.apache.spark.sql.sources.DataSourceRegisterdocs/tutorial/files/netcdfmetadata-sedona-spark.md+mkdocs.ymlnav entryHow was this patch tested?
17 tests in
netcdfMetadataTest.scalawith exact-match assertions:test.nc: metadata, geoTransform, cornerCoordinates, dimensions, per-variable metadata (CF attributes,_FillValue/missing_value,isCoordinate), empty globalAttributesRS_FromNetCDFformat="NetCDF-4", extent, and unsigned (ubyte) attribute wideninggrid_mappingvariable (EPSG:4326)geoTransformwith centers-onlycornerCoordinatespath/driver/fileSizeare selectedVerified locally on Spark 3.4/Scala 2.12 and cross-compiled on Spark 4.0/Scala 2.13.
Did this PR include necessary documentation updates?