[Python][C++] Fix tree-model TsFileDataFrame returning empty data on reused readers and uppercase names#862
Merged
ColinLeeo merged 6 commits intoJul 16, 2026
Conversation
…ppercase names Two related fixes for reading tree-model TsFiles through TsFileDataFrame, which together caused 'read one series OK, then reading another series returns empty' on a reused reader. C++: make StringArrayDeviceID::split_table_name()/init_prefix_segments() idempotent. Device IDs are cached and reused across queries; the previous code appended to prefix_segments_ on every call, so prefixes accumulated (and leaked) across successive queries and corrupted per-device path columns, making a second series read match nothing. Now it clears/frees existing prefixes before rebuilding. Adds DeviceIdTest.SplitTableNameIsIdempotent. Python: read series with uppercase measurement/path names. query_table_on_tree synthesizes a case-insensitive table schema, so the native layer lower-cases column names and path segments, while the dataset catalog keeps the original casing. The tree read paths matched field columns and device paths case-sensitively and returned empty for uppercase names. _read_series_by_row_tree and _read_arrow_tree now match case-insensitively. Adds a regression test.
Young-Leo
force-pushed
the
fix/tree-model-uppercase-read
branch
from
July 9, 2026 13:03
6586660 to
d13cd23
Compare
ColinLeeo
reviewed
Jul 15, 2026
Comment on lines
+106
to
+112
| // Idempotent: device IDs are cached and reused across queries, so clear | ||
| // previous prefixes before rebuilding to avoid accumulation and leaks. | ||
| for (const auto& prefix_segment : prefix_segments_) { | ||
| delete prefix_segment; | ||
| } | ||
| prefix_segments_.clear(); | ||
|
|
Contributor
There was a problem hiding this comment.
It's not a good idea but useful...
Contributor
|
In the tree model, both device paths and measurements are case-sensitive, so we should not simply normalize them to lowercase. Otherwise, case-distinct paths/measurements may be overwritten or mixed. The Python layer should preserve the original casing and resolve values by exact column position. On the C++ side, the tree query path does not seem to lowercase these names directly, but ResultSet name-based lookup is case-insensitive, so there may still be a similar collision risk that we should verify/fix as well. |
…ollision regression test Revert the .lower() case-insensitive column/path matching in the tree read paths (from d13cd23). It let single-variant uppercase measurements read, but conflated case-distinct measurements (e.g. temperature vs Temperature) into one series. Restore case-sensitive matching (native still lower-cases names, so uppercase reads are empty for now) and add a regression test asserting case-distinct measurements do not collide. Two tree-model tests are intentionally red pending the C++ root fix (TableSchema for tree queries must not lower-case names).
Tree-model device paths and measurement names are case-sensitive, but the virtual TableSchema synthesized in query_on_tree lower-cased them via the TableSchema constructor. This collapsed case-distinct series (e.g. "temperature" vs "Temperature") into one column, so reads returned empty or mixed data. - TableSchema: add virtual_table flag; skip lower-casing table/measurement names and keep column_pos_index_ in original case for virtual tables. - find_column_index/find_id_column_order: match case-sensitively for virtual tables, unchanged (lower-cased) for real tables. - query_on_tree: build the synthesized schema with virtual_table=true.
ColinLeeo
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When reading tree-model TsFiles through
TsFileDataFrame, series reads couldreturn empty arrays in two situations. This PR fixes both, which together were
responsible for "read one series OK, then read another series returns empty" on
a reused reader.
Both fixes go on top of the existing tree-model support (#816).
Fix 1 — C++: make
StringArrayDeviceID::split_table_name()idempotentProblem
Device IDs are cached and reused across queries.
split_table_name()/init_prefix_segments()appended toprefix_segments_every time it wascalled, so on a reused reader the prefix segments accumulated across successive
queries. The stale/duplicated prefix segments corrupted the per-device path
columns, so a second series read on the same reader could match nothing and
return empty (and it also leaked the previously allocated segment pointers).
Change
init_prefix_segments()now clears and frees the existingprefix_segments_before rebuilding, making
split_table_name()idempotent. Repeated callsproduce a stable, correct segment vector.
Files:
cpp/src/common/device_id.cc, testcpp/test/common/device_id_test.cc(
DeviceIdTest.SplitTableNameIsIdempotent).Fix 2 — Python: read series with uppercase measurement / path names
Problem
Tree TsFiles have no table schema, so the Python dataset layer reads them by
synthesizing a virtual table via
query_table_on_tree. Because the table modelis case-insensitive, the native layer normalizes the synthesized column names
and path segments to lower case (e.g.
Temperature->temperature), whilethe dataset catalog keeps the original casing from timeseries metadata. The two
tree read paths then looked the field column and device path up
case-sensitively, so an uppercase measurement or path segment failed the lookup
and returned an empty array:
Change Match field column names and device path segments case-insensitively in
_read_series_by_row_treeand_read_arrow_tree, consistent with the case-insensitive table model. The original casing is still surfaced to users; only the internal lookup against the lower-cased result set is normalized.Files:
python/tsfile/dataset/reader.py, regression test inpython/tests/test_tsfile_dataset.py.Tests
DeviceIdTest.SplitTableNameIsIdempotent.