Skip to content

add: RSM Volume Builder — merge scan(s) into a gridded reciprocal-space volume - #129

Open
pecomyint wants to merge 2 commits into
mainfrom
dev-peco
Open

add: RSM Volume Builder — merge scan(s) into a gridded reciprocal-space volume#129
pecomyint wants to merge 2 commits into
mainfrom
dev-peco

Conversation

@pecomyint

@pecomyint pecomyint commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds the one thing rsMap3D (https://github.com/AdvancedPhotonSource/rsMap3D) can do that DashPVA could not: gridding every frame of one or more completed scans onto a single interpolated 3D reciprocal-space volume (xu.Gridder3D). We already converted angles to Q per frame, but could only show a bounded live point cloud — never a merged volume.

Launch with DashPVA rsmgrid or the Post Analysis launcher tile. Output uses the existing HDF5 volume format, so results open in Workbench's 3D viewer with no new viewer code.

Design notes worth reviewing

  • Two passes are required, not an optimization. Gridder3D latches fixed_range on its first call when KeepData is set, after which points outside the first batch's range are silently dropped. Global bounds must be known and set via dataRange(fixed=True) before any gridding.
  • Masked pixels are excluded, never zeroed. Gridder3D.data is a per-bin mean, so a zeroed-but-present pixel would count as a real intensity = 0 measurement and bias its voxel low.
  • Energy/UB differences warn, they don't block. Each file's own UB is applied, so the merge already lands in a common crystal-fixed HKL frame — that's why rsMap3D applies UB per scan and doesn't block either. HKL is also geometrically energy-independent.
  • Axes are labelled H/K/L, not Q — passing UB to Ang2Q.area returns hkl, not Q in Å⁻¹.
  • Optional I0 normalization from entry/data/metadata/ca/<name>, matching rsMap3D's monitor division. Without it, scans at different exposure/attenuation won't match in intensity where they overlap.
  • Batch size comes from a byte budget, not a frame count — a fixed 32 frames needs ~6 GB of coordinate arrays on a 2048² detector.
  • grid_origin is shifted half a voxel: Gridder3D's axes are bin centers, but the PyVista viewer treats the origin as a cell corner.
  • The mask is stored in the viewer's display orientation and that state is never persisted, so the dialog asks rather than guessing.

Refactor

utils/rsm_converter.py gains build_file_geometry() / q_for_frames() so QConversion/HXRD/init_area are built once per file instead of once per frame, using xrayutilities' batched angle arguments. Purely additive — no existing def removed, create_rsm() and get_sample_and_detector_circles() keep their signatures and behavior (a test asserts bit-identical Q against the old path). area_det_viewer.py and hpc_rsm_consumer.py have their own inline setups and are untouched.

Separate commit

687e943 fixes a pre-existing bug found while testing: the HDF5 metadata readers used hasattr(ds, 'asstr') to detect string datasets, but every Dataset has that method — it only raises on numeric ones, and the error was swallowed. voxel_spacing/grid_origin/grid_dimensions_cells/intensity_range were coming back as reprs like '[6 6 6]'. Reviewable/cherry-pickable on its own.

Testing

227 tests pass (18 new). Grid dims in tests are deliberately asymmetric — with nx==ny==nz a transpose bug still produces a correctly-shaped array. Coverage: batched-vs-per-frame Q equivalence, hot-pixel→voxel round trip through save/load, mask exclusion vs zeroing, mask transpose, I0 normalization, cell-corner origin, and a guard on the fixed_range latch we depend on.

Disjoint from #127 — zero file overlap.

Not yet validated on real data

  • Detector axis-0 convention (cch1/pw1 vs row/col) against a known Bragg peak — pre-existing in rsm_converter, but now load-bearing. A swap would produce a transposed volume that still looks plausible.
  • Half-voxel origin, visually in Workbench.
  • Which metadata/ca/ key is I0 at the beamline.

Known gaps vs rsMap3D

No flat-field correction, no user-specified grid crop, no detector tilt (pre-existing rsm_converter limitation).

The metadata readers used `hasattr(ds, 'asstr')` to decide whether a dataset
holds strings, but every h5py Dataset has that method regardless of dtype --
it only raises when called on a numeric one. The TypeError was swallowed by
the surrounding except, so numeric arrays fell through to the stringifying
fallback: voxel_spacing, grid_origin, grid_dimensions_cells and
intensity_range all came back as reprs like '[6 6 6]' instead of [6, 6, 6].

Downstream volume code has defensive fallbacks, so this surfaced as volumes
silently rendering with default spacing/origin rather than as an error.

Use h5py.check_string_dtype(ds.dtype) at all three sites instead.
…ce volume

Ports rsMap3D's core capability: gridding every frame of one or more
completed scans onto a single interpolated 3D volume via xrayutilities'
Gridder3D. DashPVA already converted angles to Q per frame but could only
show a bounded live point cloud, never a merged, interpolated volume.

Launch with `DashPVA rsmgrid` or the Post Analysis launcher tile. Output uses
the existing HDF5 volume format, so results open in the Workbench 3D viewer.

Engine (utils/rsm_gridder.py):
- Two passes are required, not an optimization. Gridder3D latches
  fixed_range on its first call when KeepData is set, after which points
  outside the first batch's range are silently dropped -- so global bounds
  must be known and set via dataRange(fixed=True) before any gridding.
- Masked pixels are excluded from the arrays, never zeroed. Gridder3D.data
  is a per-bin mean, so a zeroed-but-present pixel would count as a real
  'intensity = 0' measurement and bias its voxel low.
- Optional I0 normalization from entry/data/metadata/ca/<name>, matching
  rsMap3D's monitor division. Without it, scans taken at different exposure
  or attenuation do not match in intensity where they overlap.
- Energy/UB differences warn rather than block: each file's own UB is
  applied, so the merge already lands in a common crystal-fixed HKL frame,
  and HKL is geometrically energy-independent. rsMap3D does not block either.
- Batch size is derived from a byte budget rather than a fixed frame count,
  which would need ~6 GB for the coordinate arrays on a 2048^2 detector.
- grid_origin is shifted half a voxel: Gridder3D's axes are bin centers but
  the PyVista viewer treats the origin as a cell corner.
- Axes are labelled H/K/L, not Q: passing UB to Ang2Q.area returns hkl.

utils/rsm_converter.py gains build_file_geometry()/q_for_frames() so
QConversion/HXRD/init_area are built once per file instead of once per
frame, using xrayutilities' batched angle arguments. create_rsm() and
get_sample_and_detector_circles() are unchanged for existing callers.

The mask is stored in the viewer's display orientation and that state is
never persisted, so the dialog asks rather than guessing.

Known gaps vs rsMap3D: no flat-field correction, no user-specified grid
crop, no detector tilt (a pre-existing rsm_converter limitation).
@pecomyint
pecomyint requested a review from Osayi-ANL August 3, 2026 21:54
@pecomyint

Copy link
Copy Markdown
Collaborator Author

@Osayi-ANL Please let me know if you have started working on this. Open to hearing your thoughts on this since you have extensively worked on 3D HKL work. This is a new feature that beamlines would find useful.

@pecomyint

Copy link
Copy Markdown
Collaborator Author

@Osayi-ANL when you have a chance, please work on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant