Skip to content

[spark] adapt data evolution merge into for tables with existing deletion vectors#8431

Merged
JingsongLi merged 4 commits into
apache:masterfrom
steFaiz:spark_merge_into_dv_tables
Jul 2, 2026
Merged

[spark] adapt data evolution merge into for tables with existing deletion vectors#8431
JingsongLi merged 4 commits into
apache:masterfrom
steFaiz:spark_merge_into_dv_tables

Conversation

@steFaiz

@steFaiz steFaiz commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Purpose

If call merge into on some data evolution tables with existing deletion vectors, there may exist some row id gaps of the merged output. Like:
original table values:

(1, "a"), (2, "b"), (3, "c")

After removing the second row, then merge into update with some source

(1, "new_a")

The merged result would be

_ROW_ID, old_val, new_val

(0, "a", "new_a")
(2, "c", null)

At that case, we should fill the gap of deleted value.

Tests

a new DataEvolutionDeletionTestBase to unify all spark tests about data-evolution deletion

@steFaiz steFaiz changed the title [spark] adpat data evolution merge into for tables with existing deletion vectors [spark] adapt data evolution merge into for tables with existing deletion vectors Jul 2, 2026
fillerRow != null || firstRowId + numWritten == rowId,
s"Cannot fill row ID gaps before any real row for first row ID $firstRowId.")
while (firstRowId + numWritten < rowId) {
recordWriter.write(fillerRow)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This physically writes one filler row for every deleted row-id gap. Since numRecords comes from the original DataFileMeta.rowCount, a table with deletion vectors where only a few rows survive in a large file can still rewrite almost the whole original row range as filler data during MERGE. Can we add a gap-size guard/metric and either fail/compact for very sparse ranges, or avoid materializing gaps with a sparse row-id mapping or synthetic placeholder reader?

@steFaiz steFaiz Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out! I think:

  1. For data-evolution tables, the filler rows are used to preserve row-id continuity. They only materialize
    structured fields, while blob fields are written as NULL, so blob payloads are not copied.
  2. I agree that materializing deletions is useful, but it needs a separate compaction design. It may change row ids, invalidate indexes, and require broader file/index rewriting, so I prefer to handle it in a follow-up
    compaction PR instead of MERGE INTO.
  3. Sparse row-id mapping or placeholder reader would also affect the whole read path and row-id semantics, so I think it deserves a separate discussion if we want to go in that direction.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a warn log for this!

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit d435ec9 into apache:master Jul 2, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants