feat: add ReplacePartitions core class (PR1 of 2 for #637) by shangxinli · Pull Request #776 · apache/iceberg-cpp

shangxinli · 2026-06-24T00:05:57Z

Summary

Adds the iceberg::ReplacePartitions class — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data files in that partition. Produces an overwrite snapshot with "replace-partitions"="true" in the summary. Unpartitioned tables replace all existing data files.

This is PR1 of 2 for #637. See tracking issue #775 for the split rationale.

What's in PR1

iceberg::ReplacePartitions class extending MergingSnapshotUpdate:
- Builder API: AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes
- Apply / Summary / CleanUncommitted overrides
- Uses ManifestFilterManager for partition-based manifest filtering
SnapshotSummaryFields::kReplacePartitions = "replace-partitions" constant
ReplacePartitions forward declaration in type_fwd.h
CMake + Meson registration

What's deferred to PR2

Transaction::NewReplacePartitions() API wiring on Table / Transaction
End-to-end test file (replace_partitions_test.cc)

Design notes

Follows the Java ReplacePartitions semantics: operation type is overwrite, summary carries "replace-partitions"="true".
Mirrors the existing C++ FastAppend / MergeAppend patterns (factory Make(), MergingSnapshotUpdate base, builder error collector).
Unpartitioned table case treated as table-wide replace via DeleteByRowFilter(AlwaysTrue()), matching Java BaseReplacePartitions.

Test plan

Clean build (cmake --build build)
clang-format clean
End-to-end behavioral tests land in PR2 alongside the Table/Transaction API wiring

Closes part of #637 (PR2 will close it).
Tracking: #775

Adds iceberg::ReplacePartitions — the dynamic partition overwrite operation. Each AddFile() registers the file's partition for replacement of all existing data and delete files in that partition. Produces an overwrite snapshot with "replace-partitions=true" in the summary. Unpartitioned tables replace all existing data files. Extends MergingSnapshotUpdate (matching Java's BaseReplacePartitions) so the full data+delete manifest pipeline, custom-summary-property handling, and conflict-validation helpers are inherited. AddFile() unconditionally calls DropPartition(spec_id, file->partition) — for unpartitioned specs the partition value is empty and the filter manager matches every file in that spec, so no separate AlwaysTrue path is needed. Touched partitions are tracked in a PartitionSet; Validate() uses the partition-scoped overloads of ValidateAddedDataFiles / ValidateNoNewDeleteFiles, or skips entirely when no partitions were staged. Changes: * iceberg::ReplacePartitions extending MergingSnapshotUpdate with builder API (AddFile, ValidateAppendOnly, ValidateFromSnapshot, ValidateNoConflictingData, ValidateNoConflictingDeletes) and Validate() override. * SnapshotSummaryFields::kReplacePartitions = "replace-partitions". * MergingSnapshotUpdate::SetSummaryProperty promoted from private to protected so subclasses can stash custom summary entries that survive commit retry via the cached-rebuild path. * Forward declaration in type_fwd.h. * CMake + Meson source registration. Public API wiring (Table::NewReplacePartitions(), Transaction::NewReplacePartitions()) and end-to-end tests are deferred to PR2. Tracking: apache#775 Related: apache#637

manuzhang · 2026-06-24T09:16:51Z

+        current_metadata, starting_snapshot_id_, replaced_partitions_, snapshot, io));
+  }
+  if (validate_conflicting_deletes_) {
+    ICEBERG_RETURN_UNEXPECTED(ValidateNoNewDeleteFiles(


Java BaseReplacePartitions.validate also calls validateDeletedDataFiles, so concurrent overwrite/delete commits in the replaced partition are rejected

…aFiles - AddFile() now uses DeleteByRowFilter(AlwaysTrue()) for unpartitioned specs instead of DropPartition with empty partition values, matching Java BaseReplacePartitions which treats unpartitioned tables as a table-wide replace. - Validate() now also calls ValidateDeletedDataFiles when ValidateNoConflictingDeletes is enabled, mirroring Java where validateNewDeletes gates both checks. This rejects concurrent overwrite/delete commits in the replaced partitions. - New replace_by_row_filter_ flag drives the AlwaysTrue path in Validate() for the unpartitioned case.

manuzhang

It looks replace_partitions.h is missing from src/iceberg/update/meson.build.

zhjwpku · 2026-06-25T07:30:56Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

zhjwpku · 2026-06-25T09:55:35Z

+
+namespace iceberg {
+
+/// \brief Replaces partitions in a table with new data files.


Do we need to mention This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use the {@link OverwriteFiles overwrite API} to explicitly overwrite data. like java?

Changed to '/// \note This is provided to implement SQL compatible with Hive table operations but is not recommended. Instead, use OverwriteFiles to explicitly overwrite data.'

…date - Add replace_partitions.h to src/iceberg/update/meson.build (manuzhang) - Add Java-style note recommending OverwriteFiles for non-Hive use (zhjwpku) - Validate(): call DataSpec() to reject no-data-files replace, matching Java BaseReplacePartitions which throws when no file was added (manuzhang)

DataSpec() also errors on multi-spec stages, which would silently break a ReplacePartitions that touches files from more than one spec. Use the flags AddFile() sets (replace_by_row_filter_ / replaced_partitions_) instead — the same condition that already gated the no-op early-return below, now promoted to an error to match Java BaseReplacePartitions.

shangxinli · 2026-06-25T18:46:42Z

In the PR description, should "the iceberg::ReplacePartitions class extending SnapshotUpdate" be changed to "the iceberg::ReplacePartitions class extending MergingSnapshotUpdate"?

Updated the PR description — "extending SnapshotUpdate" → "extending MergingSnapshotUpdate". Thanks for catching that.

…e-spec enforcement

wgtmac

Codex was used to review Java parity against Iceberg Java's ReplacePartitions, BaseReplacePartitions, MergingSnapshotProducer, and TestReplacePartitions. The inline comments cover the remaining behavior, API documentation, and test gaps.

- Call DataSpec() unconditionally in Validate so a replace mixing an unpartitioned/all-void file with files from another spec is rejected instead of committing a table-wide replace. - Override Apply() to translate the fail-any-delete error into Java's "Cannot commit file that conflicts with existing partition" wording. - Run ValidateDeletedDataFiles before ValidateNoNewDeleteFiles in both branches to match Java's deterministic failure ordering. - Trim Java-comparison/implementation-history comments to focus on the rule; align the ValidateNoConflictingDeletes contract with Java. - Add direct ReplacePartitions tests: partitioned and unpartitioned replacement, empty and mixed specs, append-only validation, and same/different-partition data and delete concurrency. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ions-core # Conflicts: # src/iceberg/update/merging_snapshot_update.h

wgtmac

Codex was used for this Java parity review against the current Iceberg Java ReplacePartitions, BaseReplacePartitions, and tests. The remaining inline comments cover public contracts, comment cleanup, and focused test gaps.

- replace_partitions.h/.cc: drop Java-comparison and fix-history detail from Apply()/Validate() and the header; state only the contract. - Document the default idempotent-validation behavior on the class and correct the ValidateAppendOnly() contract wording (fails on any partition replacement and pre-existing delete files, not just data-file deletion). - ReplaceLeavesOtherPartitions: assert live file paths instead of summary counts. - AppendOnlyConflictTranslated: assert the full translated message with the partition path (x=1). - Add tests: all-void spec table-wide replace, unpartitioned conflict validation, concurrent data-file removal rejected by ValidateNoConflictingDeletes, no-validation idempotent same-partition replace, and format-v3 DV cleanup. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- NoValidationSamePartitionConcurrentReplaceCommits: assert the replacement is the only live file after the idempotent commit, not just that it succeeded. - AllVoidSpecReplacesWholeTable: also assert added-data-files == 1 so the summary fully pins the all-void file as the sole survivor. - ReplaceCleansUpDeletionVectors: assert the exact DV path before and its absence after, instead of a raw live-delete count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

zhjwpku

Generally LGTM, except for two minor nits from me.

wgtmac

Thanks @shangxinli for working on this! I took the liberty to address open comments and align with the Java behavior as much as possible.

shangxinli force-pushed the feat/replace-partitions-core branch 4 times, most recently from ae973f6 to e33f433 Compare June 24, 2026 01:01

shangxinli mentioned this pull request Jun 24, 2026

feat: implement ReplacePartitions (two-PR plan) #775

Open

shangxinli force-pushed the feat/replace-partitions-core branch from e33f433 to 12dc230 Compare June 24, 2026 01:13

manuzhang reviewed Jun 24, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc

manuzhang reviewed Jun 25, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc Outdated

manuzhang suggested changes Jun 25, 2026

View reviewed changes

zhjwpku reviewed Jun 25, 2026

View reviewed changes

shangxinli added 2 commits June 25, 2026 09:36

wgtmac reviewed Jul 2, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc Outdated

wgtmac reviewed Jul 2, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc Outdated

shangxinli added 2 commits July 11, 2026 13:22

Address review: void-transform unpartitioned check and DataSpec singl…

b03d790

…e-spec enforcement

Merge branch 'main' into feat/replace-partitions-core

52ef133

manuzhang added this to the 0.4.0 milestone Jul 21, 2026

wgtmac reviewed Jul 23, 2026

View reviewed changes

shangxinli and others added 2 commits July 23, 2026 13:40

Merge remote-tracking branch 'upstream/main' into feat/replace-partit…

b971977

…ions-core # Conflicts: # src/iceberg/update/merging_snapshot_update.h

wgtmac reviewed Jul 24, 2026

View reviewed changes

shangxinli and others added 2 commits July 25, 2026 15:25

zhjwpku reviewed Jul 27, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc Outdated

zhjwpku reviewed Jul 27, 2026

View reviewed changes

Comment thread src/iceberg/update/replace_partitions.cc

zhjwpku approved these changes Jul 27, 2026

View reviewed changes

wgtmac added 2 commits July 29, 2026 15:37

Merge branch 'main' into feat/replace-partitions-core

4ea9e5d

align more with java

da91bb4

wgtmac approved these changes Jul 29, 2026

View reviewed changes

wgtmac merged commit 41435b8 into apache:main Jul 29, 2026
18 checks passed


		namespace iceberg {

		/// \brief Replaces partitions in a table with new data files.

Uh oh!

Conversation

shangxinli commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in PR1

What's deferred to PR2

Design notes

Test plan

Uh oh!

manuzhang Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

shangxinli Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

manuzhang left a comment

Choose a reason for hiding this comment

Uh oh!

zhjwpku commented Jun 25, 2026

Uh oh!

zhjwpku Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

shangxinli Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

shangxinli commented Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhjwpku left a comment

Choose a reason for hiding this comment

Uh oh!

wgtmac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shangxinli commented Jun 24, 2026 •

edited

Loading