[spark] Fix V2 write with Hive dynamic partition column order#8414
Merged
YannByron merged 1 commit intoJul 3, 2026
Merged
Conversation
kerwin-zk
force-pushed
the
fix-spark-v2-hive-dynamic-partition-order
branch
3 times, most recently
from
9459c61 to
4505435
Compare
There was a problem hiding this comment.
Pull request overview
This PR addresses incorrect handling of Hive-style dynamic partition column ordering during Spark V2 writes, ensuring Paimon can accept queries whose partition columns appear in Hive’s expected positions without breaking non-Paimon inserts.
Changes:
- Adds a parser/analyzer rule that marks INSERT statements containing Hive-style dynamic partitions.
- Extends Paimon write analysis to strip the marker and resolve output columns correctly for dynamic partition overwrites (including Hive ordering).
- Adds regression tests covering both non-Paimon inserts and Paimon V2 dynamic overwrite scenarios.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/InsertOverwriteTableTestBase.scala | Adds regression tests for non-Paimon dynamic partition inserts and Paimon V2 dynamic overwrite with Hive partition column order. |
| paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/AbstractPaimonSparkSqlExtensionsParser.scala | Introduces a rule to tag Hive dynamic partition INSERTs by wrapping the query plan. |
| paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonHiveDynamicPartitionQuery.scala | Adds a lightweight logical wrapper node to carry dynamic partition column metadata through analysis. |
| paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/analysis/PaimonAnalysis.scala | Updates V2 write resolution to detect/strip the wrapper and apply Hive-style output ordering when needed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+390
to
+395
| private def withNewQuery(insert: InsertIntoStatement, query: LogicalPlan): InsertIntoStatement = { | ||
| insert.getClass | ||
| .getMethod("withNewChildInternal", classOf[LogicalPlan]) | ||
| .invoke(insert, query) | ||
| .asInstanceOf[InsertIntoStatement] | ||
| } |
YannByron
reviewed
Jul 2, 2026
| } | ||
| } | ||
|
|
||
| test("Paimon parser: dynamic partition marker is removed for non-Paimon inserts") { |
Contributor
There was a problem hiding this comment.
maybe “Paimon Insert: overwrite format(parquet) table in static mode”?
YannByron
reviewed
Jul 2, 2026
| query.transformDown { case PaimonHiveDynamicPartitionQuery(_, child) => child } | ||
| } | ||
|
|
||
| private def resolveHiveDynamicPartitionWrite( |
Contributor
There was a problem hiding this comment.
resolveHiveDynamicPartitionWrite and resolveDynamicPartitionOverwrite are near duplicates. Both methods share the same shape: check hiveStyleDynamicPartitionOutput, and on match do "by-position → by-name", otherwise a single by-name resolve. Only the way dynamicPartitionColumns is obtained differs (from the marker vs. from table.partitionKeys()).
YannByron
reviewed
Jul 2, 2026
| table: DataSourceV2Relation, | ||
| dynamicPartitionColumns: Seq[String]): Option[Seq[Attribute]] = { | ||
| val partitionKeys = table.table.asInstanceOf[SparkTable].getTable.partitionKeys().asScala.toSeq | ||
| val dynamicPartitionSet = dynamicPartitionColumns |
Contributor
There was a problem hiding this comment.
It is a misleading no-op alias.
Contributor
|
@kerwin-zk left some comments. |
kerwin-zk
force-pushed
the
fix-spark-v2-hive-dynamic-partition-order
branch
from
4505435 to
c2ee502
Compare
Contributor
Author
|
@YannByron done |
YannByron
approved these changes
Jul 3, 2026
Contributor
|
+1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix V2 write with Hive dynamic partition column order
Tests
CI