[SPARK-58165][SQL][CONNECT] Fix SubqueryExpression dropping nested subquery plans in IN values by LuciferYang · Pull Request #57308 · apache/spark

LuciferYang · 2026-07-16T14:41:33Z

What changes were proposed in this pull request?

SubqueryExpression (in sql/api's internal/columnNodes.scala) was the only ColumnNode that returned children = Seq.empty even when its SubqueryType was IN(values), whose values: Seq[ColumnNode] can hold nested column nodes -- including another SubqueryExpression. It also did not override normalize(), unlike every sibling node.

This PR makes SubqueryExpression consistent with the other nodes:

children now exposes IN.values (and stays empty for SCALAR/EXISTS). The subquery plan ds is intentionally not a child -- it is a DataFrame plan, not a column node, and the converters handle it separately.
normalize() is overridden to strip origin and normalize the IN.values, mirroring sibling nodes.

Why are the changes needed?

A nested subquery is reachable through the public 4.1 API, e.g.

t1.select(max($"v")).scalar().isin(t2.select($"c"))
// WHERE (SELECT max(v) FROM t1) IN (SELECT c FROM t2)

On Spark Connect, SparkSession.newDataset collects subquery plan references via node.collect { case _: SubqueryExpression => ... }, which recurses through children. Because children was empty, the inner scalar subquery's plan was never added to WithRelations, while the proto converter still emitted a planId reference to it. The server then failed resolving that planId:

[CONNECT_INVALID_PLAN.ASSERTION_FAILURE] Missing relation in WithRelations: (N) not in (M)

Spark Classic is unaffected: its converter embeds the subquery's logical plan directly and does not rely on children. The missing normalize() override is a separate, lower-impact inconsistency: two Columns built from the same ds at different call sites compared unequal purely due to retained origin.

Does this PR introduce any user-facing change?

Yes. On Spark Connect, a nested subquery inside isin(ds) -- e.g. df1.scalar().isin(df2) -- previously failed with an internal Missing relation in WithRelations error. After this fix it works, matching Spark Classic and the equivalent SQL. This is a bug fix relative to released 4.1.x.

How was this patch tested?

New cases in DataFrameSubquerySuite (sql/core Classic + sql/connect/client/jvm Connect) exercise a scalar subquery nested inside an IN subquery's values. Verified reproduce-then-fix: the Connect case fails with Missing relation in WithRelations on the unfixed tree and passes after the fix; both full suites pass with no regressions. Direct normalization coverage for SubqueryExpression (SCALAR / EXISTS / IN(values)) was added to ColumnNodeSuite, verifying origin stripping and recursive normalization of IN(values).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.8)

…bquery plans in IN values SubqueryExpression returned children = Seq.empty even for SubqueryType.IN(values), so ColumnNode.collect/foreach never descended into a nested subquery inside isin(ds). On Spark Connect this left the inner subquery's plan out of WithRelations while the proto converter still referenced its planId, failing with "Missing relation in WithRelations". Expose IN.values as children and override normalize() to mirror sibling nodes.

LuciferYang · 2026-07-17T03:26:45Z

cc @cloud-fan

cloud-fan

0 blocking, 1 non-blocking, 0 nits.
The runtime fix is well-scoped; add direct coverage for the separately introduced normalization contract.

Correctness (1)

sql/api/src/main/scala/org/apache/spark/sql/internal/columnNodes.scala:669: Add direct coverage for SubqueryExpression normalization -- see inline

cloud-fan · 2026-07-18T00:17:09Z

    subqueryType: SubqueryType,
    override val origin: Origin = CurrentOrigin.get)
    extends ColumnNode {
+  override private[internal] def normalize(): SubqueryExpression = copy(


Please add this node to the normalization contract coverage in ColumnNodeSuite. The new execution tests exercise children, but they do not verify origin stripping or recursive normalization of IN(values), so this separately introduced equality/hash behavior could regress unnoticed.

…ation coverage Adds direct coverage of the SubqueryExpression normalization contract to ColumnNodeSuite, addressing review feedback: the existing execution tests exercise `children` but not origin stripping or the recursive normalization of IN(values). Covers SCALAR, EXISTS, and IN(values).

uros-b

Thank you @LuciferYang and @cloud-fan!

…case into its own test Address review feedback on PR apache#57308: - Move the SPARK-58165 "scalar subquery nested in IN subquery values" case out of `test("IN predicate subquery")` into a dedicated test in both the Connect and classic `DataFrameSubquerySuite`, so the regression has a self-contained failure signal that can't be masked by an earlier checkAnswer in the same block. - Add a correlated variant where the nested scalar subquery references the outer row via `outer()`, broadening coverage beyond the non-correlated (constant) shape. No production code change; test-only.

…lated test The correlated variant added in the previous commit wrote the nested subquery as a single-line method chain, which fails the scalafmt check that `dev/lint-scala` enforces on the sql/connect modules. Break the chain across lines to satisfy scalafmt. Formatting only; no behavior change.

LuciferYang · 2026-07-21T06:22:21Z

friendly ping @cloud-fan

dongjoon-hyun

+1, LGTM.

…bquery plans in IN values ### What changes were proposed in this pull request? `SubqueryExpression` (in `sql/api`'s `internal/columnNodes.scala`) was the only `ColumnNode` that returned `children = Seq.empty` even when its `SubqueryType` was `IN(values)`, whose `values: Seq[ColumnNode]` can hold nested column nodes -- including another `SubqueryExpression`. It also did not override `normalize()`, unlike every sibling node. This PR makes `SubqueryExpression` consistent with the other nodes: - `children` now exposes `IN.values` (and stays empty for `SCALAR`/`EXISTS`). The subquery plan `ds` is intentionally not a child -- it is a DataFrame plan, not a column node, and the converters handle it separately. - `normalize()` is overridden to strip `origin` and normalize the `IN.values`, mirroring sibling nodes. ### Why are the changes needed? A nested subquery is reachable through the public 4.1 API, e.g. ```scala t1.select(max($"v")).scalar().isin(t2.select($"c")) // WHERE (SELECT max(v) FROM t1) IN (SELECT c FROM t2) ``` On Spark Connect, `SparkSession.newDataset` collects subquery plan references via `node.collect { case _: SubqueryExpression => ... }`, which recurses through `children`. Because `children` was empty, the inner scalar subquery's plan was never added to `WithRelations`, while the proto converter still emitted a planId reference to it. The server then failed resolving that planId: ``` [CONNECT_INVALID_PLAN.ASSERTION_FAILURE] Missing relation in WithRelations: (N) not in (M) ``` Spark Classic is unaffected: its converter embeds the subquery's logical plan directly and does not rely on `children`. The missing `normalize()` override is a separate, lower-impact inconsistency: two Columns built from the same `ds` at different call sites compared unequal purely due to retained `origin`. ### Does this PR introduce _any_ user-facing change? Yes. On Spark Connect, a nested subquery inside `isin(ds)` -- e.g. `df1.scalar().isin(df2)` -- previously failed with an internal `Missing relation in WithRelations` error. After this fix it works, matching Spark Classic and the equivalent SQL. This is a bug fix relative to released 4.1.x. ### How was this patch tested? New cases in `DataFrameSubquerySuite` (`sql/core` Classic + `sql/connect/client/jvm` Connect) exercise a scalar subquery nested inside an IN subquery's values. Verified reproduce-then-fix: the Connect case fails with `Missing relation in WithRelations` on the unfixed tree and passes after the fix; both full suites pass with no regressions. Direct normalization coverage for `SubqueryExpression` (SCALAR / EXISTS / IN(values)) was added to `ColumnNodeSuite`, verifying origin stripping and recursive normalization of `IN(values)`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes #57308 from LuciferYang/SPARK-subquery-columnnode. Authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 5994d4f) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2026-07-22T19:42:14Z

Merge Summary:

merged into master 5994d4f
merged into branch-4.x 9c0f7e5
merged into branch-4.2 c19664f
merged into branch-4.1 d52a8e1

Posted by merge_spark_pr.py

LuciferYang · 2026-07-23T02:35:39Z

Thank you @dongjoon-hyun @uros-b @cloud-fan

cloud-fan reviewed Jul 18, 2026

View reviewed changes

uros-b reviewed Jul 20, 2026

View reviewed changes

Comment thread sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameSubquerySuite.scala

uros-b reviewed Jul 20, 2026

View reviewed changes

Comment thread sql/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameSubquerySuite.scala

uros-b approved these changes Jul 20, 2026

View reviewed changes

LuciferYang added 2 commits July 20, 2026 22:57

dongjoon-hyun approved these changes Jul 22, 2026

View reviewed changes

LuciferYang requested a review from cloud-fan July 22, 2026 09:10

dongjoon-hyun closed this in 5994d4f Jul 22, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-58165][SQL][CONNECT] Fix SubqueryExpression dropping nested subquery plans in IN values#57308

[SPARK-58165][SQL][CONNECT] Fix SubqueryExpression dropping nested subquery plans in IN values#57308
LuciferYang wants to merge 4 commits into
apache:masterfrom
LuciferYang:SPARK-subquery-columnnode

LuciferYang commented Jul 16, 2026 •

edited

Loading

Uh oh!

LuciferYang commented Jul 17, 2026

Uh oh!

cloud-fan left a comment

Uh oh!

cloud-fan Jul 18, 2026

Uh oh!

LuciferYang Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

uros-b left a comment

Uh oh!

LuciferYang commented Jul 21, 2026

Uh oh!

dongjoon-hyun left a comment

Uh oh!

dongjoon-hyun commented Jul 22, 2026

Uh oh!

LuciferYang commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

LuciferYang commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

LuciferYang commented Jul 17, 2026

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Correctness (1)

Uh oh!

cloud-fan Jul 18, 2026

Choose a reason for hiding this comment

Uh oh!

LuciferYang Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uros-b left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Jul 21, 2026

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 22, 2026

Uh oh!

LuciferYang commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

LuciferYang commented Jul 16, 2026 •

edited

Loading