chore: drop Spark 3.4 support

[manuzhang]

Which issue does this PR close?

Closes #4329.

Rationale for this change

Spark 3.4 was deprecated in Comet 1.0.0 and is scheduled for removal in 1.1.0. Dropping it reduces the supported build/test matrix and removes compatibility paths that are no longer needed now that Spark 3.5 is the oldest supported release.

What changes are included in this PR?

• Remove the Spark 3.4 Maven profile, source/test shims, Spark SQL patch, CI jobs, release artifacts, generated compatibility page, and documentation references.
• Simplify shared code and tests to assume Spark 3.5 or newer, including removing the Spark 3.4 AQE dynamic partition-pruning fallback.
• Promote the Spark 3.5 plan-stability overrides into the base golden directories and keep Spark 4.x fallback chains intact.
• Retarget the Iceberg 1.8 compatibility job to Spark 3.5.9 and JDK 17.

How are these changes tested?

• make format PROFILES=-Pspark-3.5
• ./mvnw clean test-compile -DskipTests -Pspark-4.0
• ./mvnw clean test-compile -DskipTests -Pspark-4.1
• ./mvnw clean test-compile -DskipTests -Pspark-4.2
• Focused Spark 3.5 runtime coverage for AQE DPP and hash joins
• CometTPCDSV1_4_PlanStabilitySuite on Spark 3.5 (97 tests)
• CometTPCDSV2_7_PlanStabilitySuite on Spark 3.5 (32 tests)
• YAML, shell syntax, Python bytecode, active-reference, and git diff --check validation

[kazuyukitanimura]

Is this used?

[manuzhang]

Yes, CometExecCompatibilitySuite is renamed from CometExec3_4PlusSuite and used by all supported Spark versions.

[manuzhang]

@andygrove @coderfender @comphead I'd appreciate your review here.

[coderfender]

Sure! Thank you for the pr @manuzhang

[comphead]

Thanks @manuzhang

this is def on the table, however lets keep this PR opened for now.

There are some pending use cases still using 3.4.3, I would propose doing this in parts.
Currently it would make sense to:

• remove 3.4 from CI
• Add a warn message saying 3.4 support would be dropped in 1.1.0
• update roadmap

[manuzhang]

@comphead It's already announced in 1.0.0 release blog.

Apache Spark 3.4 is deprecated and scheduled for removal in Comet 1.1.0.

Where else do we need to add a warn message and update?

[comphead]

@comphead It's already announced in 1.0.0 release blog.

Apache Spark 3.4 is deprecated and scheduled for removal in Comet 1.1.0.

Where else do we need to add a warn message and update?

I was thinking to add a warn runtime message that Spark 3.4 support is sunsetting. Not sure if it makes sense though.

But still I would propose doing a soft drop:

• disable CI first to save resources

Keep the code for couple of more weeks and drop it closer to the dedicated release date

[parthchandra]

What does the community think about retaining the code for 3.4 (but maybe relaxed ci and backporting of fixes)? We have a ton of users (and even more production jobs) still on 3.4.