Skip to content

[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused readers and uppercase names#862

Merged
ColinLeeo merged 6 commits into
apache:developfrom
Young-Leo:fix/tree-model-uppercase-read
Jul 16, 2026
Merged

[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused readers and uppercase names#862
ColinLeeo merged 6 commits into
apache:developfrom
Young-Leo:fix/tree-model-uppercase-read

Conversation

@Young-Leo

Copy link
Copy Markdown
Contributor

Summary

When reading tree-model TsFiles through TsFileDataFrame, series reads could
return empty arrays in two situations. This PR fixes both, which together were
responsible for "read one series OK, then read another series returns empty" on
a reused reader.

Both fixes go on top of the existing tree-model support (#816).


Fix 1 — C++: make StringArrayDeviceID::split_table_name() idempotent

Problem
Device IDs are cached and reused across queries. split_table_name() /
init_prefix_segments() appended to prefix_segments_ every time it was
called, so on a reused reader the prefix segments accumulated across successive
queries. The stale/duplicated prefix segments corrupted the per-device path
columns, so a second series read on the same reader could match nothing and
return empty (and it also leaked the previously allocated segment pointers).

Change
init_prefix_segments() now clears and frees the existing prefix_segments_
before rebuilding, making split_table_name() idempotent. Repeated calls
produce a stable, correct segment vector.

Files: cpp/src/common/device_id.cc, test cpp/test/common/device_id_test.cc
(DeviceIdTest.SplitTableNameIsIdempotent).


Fix 2 — Python: read series with uppercase measurement / path names

Problem
Tree TsFiles have no table schema, so the Python dataset layer reads them by
synthesizing a virtual table via query_table_on_tree. Because the table model
is case-insensitive, the native layer normalizes the synthesized column names
and path segments to lower case (e.g. Temperature -> temperature), while
the dataset catalog keeps the original casing from timeseries metadata. The two
tree read paths then looked the field column and device path up
case-sensitively, so an uppercase measurement or path segment failed the lookup
and returned an empty array:

with TsFileDataFrame(path) as tsdf:
    tsdf["root.ln.wf01.wt01.Temperature"][:]
    # -> array([], dtype=float64)   # expected [0.5, 1.5, 2.5, 3.5, 4.5]

Change Match field column names and device path segments case-insensitively in _read_series_by_row_tree and _read_arrow_tree, consistent with the case-insensitive table model. The original casing is still surfaced to users; only the internal lookup against the lower-cased result set is normalized.

Files: python/tsfile/dataset/reader.py, regression test in python/tests/test_tsfile_dataset.py.


Tests

  • New C++ regression test: DeviceIdTest.SplitTableNameIsIdempotent.
  • New Python regression tests covering reused-reader reads and uppercase measurement/path names.
  • No public API change; logical series names keep their original casing.

…ppercase names

Two related fixes for reading tree-model TsFiles through TsFileDataFrame, which together caused 'read one series OK, then reading another series returns empty' on a reused reader.

C++: make StringArrayDeviceID::split_table_name()/init_prefix_segments() idempotent. Device IDs are cached and reused across queries; the previous code appended to prefix_segments_ on every call, so prefixes accumulated (and leaked) across successive queries and corrupted per-device path columns, making a second series read match nothing. Now it clears/frees existing prefixes before rebuilding. Adds DeviceIdTest.SplitTableNameIsIdempotent.

Python: read series with uppercase measurement/path names. query_table_on_tree synthesizes a case-insensitive table schema, so the native layer lower-cases column names and path segments, while the dataset catalog keeps the original casing. The tree read paths matched field columns and device paths case-sensitively and returned empty for uppercase names. _read_series_by_row_tree and _read_arrow_tree now match case-insensitively. Adds a regression test.
@Young-Leo
Young-Leo force-pushed the fix/tree-model-uppercase-read branch from 6586660 to d13cd23 Compare July 9, 2026 13:03
Comment on lines +106 to +112
// Idempotent: device IDs are cached and reused across queries, so clear
// previous prefixes before rebuilding to avoid accumulation and leaks.
for (const auto& prefix_segment : prefix_segments_) {
delete prefix_segment;
}
prefix_segments_.clear();

@ColinLeeo ColinLeeo Jul 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a good idea but useful...

@ColinLeeo

Copy link
Copy Markdown
Contributor

In the tree model, both device paths and measurements are case-sensitive, so we should not simply normalize them to lowercase. Otherwise, case-distinct paths/measurements may be overwritten or mixed. The Python layer should preserve the original casing and resolve values by exact column position. On the C++ side, the tree query path does not seem to lowercase these names directly, but ResultSet name-based lookup is case-insensitive, so there may still be a similar collision risk that we should verify/fix as well.

…ollision regression test

Revert the .lower() case-insensitive column/path matching in the tree read paths (from d13cd23). It let single-variant uppercase measurements read, but conflated case-distinct measurements (e.g. temperature vs Temperature) into one series. Restore case-sensitive matching (native still lower-cases names, so uppercase reads are empty for now) and add a regression test asserting case-distinct measurements do not collide. Two tree-model tests are intentionally red pending the C++ root fix (TableSchema for tree queries must not lower-case names).
Tree-model device paths and measurement names are case-sensitive, but the

virtual TableSchema synthesized in query_on_tree lower-cased them via the

TableSchema constructor. This collapsed case-distinct series (e.g.

"temperature" vs "Temperature") into one column, so reads returned empty

or mixed data.

- TableSchema: add virtual_table flag; skip lower-casing table/measurement

  names and keep column_pos_index_ in original case for virtual tables.

- find_column_index/find_id_column_order: match case-sensitively for

  virtual tables, unchanged (lower-cased) for real tables.

- query_on_tree: build the synthesized schema with virtual_table=true.
@ColinLeeo
ColinLeeo merged commit e5e4ccc into apache:develop Jul 16, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants