ci: automate SynapseML releases for Spark 3.5 and 4.1

[Rana Singh (ranadeepsingh)]

Summary

Automate repeatable public SynapseML releases, defaulting to master / Spark
3.5 and spark4.1. Spark 4.0 remains an explicit opt-in and is not required
for a default release.

What this automates

• Prepare version changes, release branches and versioned documentation while
  preserving older guides.
• Bind release plans to reviewed source commits, validate selected candidates,
  create immutable tags and coordinate publication with a durable ledger.
• Verify publication evidence and generate release notes for the selected
  runtimes. Existing approved three-runtime plans remain valid.
• Support an explicitly approved pre-merge release path, no-write dry runs and
  recovery after partial or ambiguous submissions.
• Keep public plans and evidence separate from optional private configuration.
  Private integrations are not prerequisites for the public release.
• Leave an existing GitHub Release untouched and create one only after a
  completed HTTP 404 lookup. API, authentication and transport errors stop
  release-note publication.

Default version bumps keep Spark 4.0 examples on its last published version.
The operator guide
documents opt-in, readiness checks, dry runs and recovery. Plan approval,
signing approval and PR merges remain human actions.

Validation and current status

Status checked on 2026-09-24, for head 778bda108f, rebased onto master
384f27a5d0. The rebase preserved the existing PR patch before the targeted
release-lookup fix was added.

• Current-head targeted validation: 122 workflow, recovery and
  public-documentation checks passed, plus pinned Python formatting.
  The new lookup regression covers 13 cases. Before the fix, 11 error cases
  incorrectly proceeded to publication; all now pass.
• The confirmed release-lookup finding is fixed and its thread has a reply
  with the regression evidence. The
  fresh Copilot review
  of this commit reports Findings: None and marks that finding resolved.
  There are no unresolved threads or suppressed current-head findings.
  Its overview still calls for closer review because CI and release validation
  remain outstanding.
• Azure PR build 237500013 is running against verified merge commit
  8cdb5e6a4814601f6c387921367cd2c6428fe3a1. Build 237472805 completed
  with 65 successful jobs after retries, but that result belongs to the
  previous head and does not validate this commit.
• The Spark 4.1 PR compatibility replay failure is advisory. This does not
  waive validation of an actual Spark 4.1 release candidate.

The implementation is committed, but current-head CI and human approval
are not yet complete. Review evidence and its limits are recorded in
reviews/pr-2628.

Before publishing 1.2.0

1.2.0 has not been published by this work. Keep production publication
blocked until the Python distribution works on both selected runtimes.

The primary PyPI wheel contains master's Python code, while spark4.1 has
additional compatibility fixes. Testing port-built wrappers does not validate
that primary wheel; an image-conversion case reproduces the difference.

Before release approval, select and validate a cross-runtime wheel or matching
runtime-specific wheels, then update the artifact inventory and consumer
instructions. This PR does not introduce another publication destination.
Inspect the actual produced packages before approving publication.

Keep the PR unmerged until the requested pre-merge release trial is complete.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Pull request overview

This PR adds release-engineering automation to make SynapseML-to-Fabric releases more deterministic and reviewable, including a GitHub Actions “prepare → tag merged commit → publish notes” flow plus supporting Python tooling to derive/verify release identifiers and safely bump BBC-VHD pins.

Changes:

• Introduces a release matrix generator (release_matrix.py) and an end-to-end verifier (verify_release.py) with accompanying tests.
• Adds bump_bbcvhd.py (plus tests) to safely update BBC-VHD component pins while preserving line endings and rolling back on failure.
• Adds/updates GitHub workflows for release prepare/tagging and manual, artifact-gated release notes; enables workflow_dispatch for PR validation so release PRs created via GITHUB_TOKEN can be validated.

Show a summary per file

File	Description
scripts/test_bump_version.py	Extends bump-version tests for denylisted repo-relative paths; normalizes path keys to POSIX.
scripts/bump-version.py	Improves Windows console encoding robustness; adds path-based denylist support; normalizes path handling via as_posix().
scripts/release/release_matrix.py	Adds a single-source-of-truth release matrix for tags and artifact versions + CLI rendering.
scripts/release/verify_release.py	Adds live verification of GitHub/ADO tags and Maven/PyPI/UPack/Azure Artifacts presence.
scripts/release/bump_bbcvhd.py	Adds deterministic, rollback-safe BBC-VHD component pin updater with CRLF/LF preservation.
scripts/release/README.md	Documents how to use the new release tooling and where it fits in the guide.
scripts/release/test_release_matrix.py	Adds regression/contract tests for the release matrix derivations and CLI validation.
scripts/release/test_verify_release.py	Adds unit tests for verifier networking/error handling and run plan behavior.
scripts/release/test_bump_bbcvhd.py	Adds tests for BBC-VHD bump idempotency, rollback, and newline preservation.
scripts/release/test_release_workflows.py	Adds tests asserting key workflow contract properties (manual gating, dispatch behavior).
scripts/release/test_prev_tag.sh	Adds a repo-taglist regression script for “previous primary tag” selection logic.
.github/workflows/release-prepare.yml	Adds a reviewed release PR generator and a merged-commit tagger that dispatches downstream orchestration.
.github/workflows/release-notes.yml	Adds a manual, artifact-gated GitHub Release publisher with explicit previous-tag selection.
.github/workflows/pr-validation.yml	Enables workflow_dispatch so validations can be dispatched for bot-opened release PR branches.

Review details

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

scripts/release/verify_release.py:260

• --skip internal does not currently skip the internal PyPI feed checks (the synapseml_internal package); it only skips internal git tag checks. This makes --skip internal behave inconsistently compared with --skip public.

    def pip(self, package: str, version: str) -> str:
        if "pip" in self.skip or "ado" in self.skip:
            return SKIPPED
        # Azure Artifacts normalises pypi names: synapseml_internal -> synapseml-internal
        return (
            OK
            if version
            in self._feed_versions("Synapse-Conda", "pypi", package.replace("_", "-"))
            else MISSING
        )

• Files reviewed: 14/14 changed files
• Comments generated: 2
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:75

• On Windows, subprocess.run(..., shell=True) expects a command string, but this code passes a list. That combination can fail (or behave differently) on win32, which would break the fallback ADO auth path even when az is installed. Build a platform-appropriate command (string for shell=True, list otherwise).

    out = subprocess.run(
        [
            "az",
            "account",
            "get-access-token",
            "--resource",
            ADO_RESOURCE,
            "--query",
            "accessToken",
            "-o",
            "tsv",
        ],
        capture_output=True,
        text=True,
        shell=(sys.platform == "win32"),
    )

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

Addressed the current-head suppressed Windows auth finding in 6e80f27.

• Windows now gives subprocess.run(..., shell=True) a command string; non-Windows retains an argument list with shell=False.
• Parameterized regression coverage locks both command shapes.
• A token-less Windows fallback replay authenticated through �z and found all 12 selected v1.1.3 master artifacts; the complete release-tooling suite is 279 passed.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

scripts/release/verify_release.py:313

• The Maven publication gate here verifies only synapseml-core_{scala} (via public_maven()), but the workflow-generated release notes advertise com.microsoft.azure:synapseml_2.12:${VERSION} (see .github/workflows/release-notes.yml:137) and the docs use the synapseml_2.12 coordinate (website/doctest.py:25). This can let verify_release.py report COMPLETE even if the primary install coordinate is missing (or vice versa). Consider aligning the verification to check the same Maven artifact(s) that users are instructed to consume (e.g., verify both synapseml_{scala} and synapseml-core_{scala}, or switch to the canonical one).

            "maven",
            tp.key,
            f"synapseml-core_{tp.scala}",
            tp.oss_maven_version,
            c.public_maven(tp.scala, tp.oss_maven_version),

• Files reviewed: 14/14 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved release publication scoping and dependency issues, plus workflow-test coverage gaps, block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 · 1 · 4

Open (8)

• Maven-only plans trigger unauthorized non-Maven publication tasks · New
• Required staging arguments break the Spark 4.1 release pipeline
• Reject stale versioned files during ESRP staging
• Optional skipped dependencies can prevent release publication · New
• Review artifact uses a session-workspace path · New
• Review artifact exposes an absolute local session path · New
• Review artifact contains an absolute machine-local path · New
• Review artifact contains checkout-specific commands · New

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved release-safety findings remain in release-note error handling and immutable-container validation.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Fail closed for unknown blob containers · New

Resolved since last review (8)

• Maven-only plans trigger unauthorized non-Maven publication tasks
• Required staging arguments break the Spark 4.1 release pipeline
• Reject stale versioned files during ESRP staging
• Optional skipped dependencies can prevent release publication
• Review artifact contains checkout-specific commands
• Review artifact contains an absolute machine-local path
• Review artifact exposes an absolute local session path
• Review artifact uses a session-workspace path

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Three critical workflow issues and one moderate test-coverage issue remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3

Open (3)

• Empty tag list aborts release-note generation · New
• Unverified SBT launcher enables arbitrary code execution · New
• Primary tag check does not authorize merged release content · New

Resolved since last review (1)

• Fail closed for unknown blob containers

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

Address the two moderate release-test coverage gaps before approval.

Review effort: Lite
Findings: None

Resolved since last review (3)

• Primary tag check does not authorize merged release content
• Unverified SBT launcher enables arbitrary code execution
• Empty tag list aborts release-note generation

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved release-workflow guard issues remain, and production publication is explicitly blocked pending Python wheel validation.

Review effort: Lite
Findings: 1

Open (1)

• Check all derivative tags before creating a new preparation PR · New

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

Unresolved moderate release-safety findings remain in the guarded publication and documentation paths.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Check all derivative tags before creating a new preparation PR

Previously missed (1)

In code that hasn't changed since last review

Distinguish missing releases from GitHub errors

.github/​workflows/​release-notes.yml:125

gh release view returns nonzero for both a confirmed missing release and transport/authentication failures. This if treats every failure as found=false, so a transient GitHub error can fall through to note generation and a write attempt instead of failing closed (and can race with an existing release that could not be read). Distinguish a confirmed 404 from other failures and stop on all other errors before publishing.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🟡 Changes recommended

Release-note integration, failure handling, and rerun idempotence issues remain unresolved, alongside the documented wheel-validation release blocker.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

• Accept squash/rebase merge commits in release verification · New

[Copilot]

Copilot review overview

🔵 Needs a closer look

Moderate unresolved findings remain, CI is not green, and final human review is required.

Review effort: Lite
Findings: 1

Open (1)

• Treating all release lookup failures as release-not-found · New

Resolved since last review (1)

• Accept squash/rebase merge commits in release verification

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Copilot review overview

🔵 Needs a closer look

CI is not green, and the release trial and cross-runtime publication validation remain outstanding.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Treating all release lookup failures as release-not-found