[spark] adapt data evolution merge into for tables with existing deletion vectors#8431
Merged
Merged
Conversation
JingsongLi
reviewed
Jul 2, 2026
| fillerRow != null || firstRowId + numWritten == rowId, | ||
| s"Cannot fill row ID gaps before any real row for first row ID $firstRowId.") | ||
| while (firstRowId + numWritten < rowId) { | ||
| recordWriter.write(fillerRow) |
Contributor
There was a problem hiding this comment.
This physically writes one filler row for every deleted row-id gap. Since numRecords comes from the original DataFileMeta.rowCount, a table with deletion vectors where only a few rows survive in a large file can still rewrite almost the whole original row range as filler data during MERGE. Can we add a gap-size guard/metric and either fail/compact for very sparse ranges, or avoid materializing gaps with a sparse row-id mapping or synthetic placeholder reader?
Contributor
Author
There was a problem hiding this comment.
Thanks for pointing this out! I think:
- For data-evolution tables, the filler rows are used to preserve row-id continuity. They only materialize
structured fields, while blob fields are written as NULL, so blob payloads are not copied. - I agree that materializing deletions is useful, but it needs a separate compaction design. It may change row ids, invalidate indexes, and require broader file/index rewriting, so I prefer to handle it in a follow-up
compaction PR instead of MERGE INTO. - Sparse row-id mapping or placeholder reader would also affect the whole read path and row-id semantics, so I think it deserves a separate discussion if we want to go in that direction.
Contributor
Author
There was a problem hiding this comment.
I've added a warn log for this!
Contributor
|
+1 |
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
If call merge into on some data evolution tables with existing deletion vectors, there may exist some row id gaps of the merged output. Like:
original table values:
After removing the second row, then merge into update with some source
The merged result would be
At that case, we should fill the gap of deleted value.
Tests
a new
DataEvolutionDeletionTestBaseto unify all spark tests about data-evolution deletion