fix(ci): repair publishing and Cognitive service test credentials - #2691
Merged
Rana Singh (ranadeepsingh) merged 6 commits intoSep 4, 2026
Merged
Conversation
## Summary Revalidate restored SBT dependencies with the existing bounded retry helper before artifact packaging starts, and add a pipeline contract regression that locks recovery ahead of the first publishing SBT command. ## Prompting Intent Triage Azure Pipelines run 234106471 on master, separate repository-actionable failures from external live-service failures, and fix the pipeline without weakening service assertions or hiding missing evidence. ## Linked Sources - Failed build: https://msdata.visualstudio.com/A365/_build/results?buildId=234106471&view=results - Existing cache hardening: microsoft@7c72c49 ## Rationale The cache task reported three exact hits, but packagePython repeatedly failed to resolve hadoop-bare-naked-local-fs, proving that an exact Azure cache restore does not guarantee usable local state. Azure task retries reused that state unchanged. Running sbt_retry.sh update once before packaging validates the cache and selectively evicts unresolved coordinates. This avoids retrying publishBlob uploads and avoids imposing an extra validation process on every SBT job. The unrelated 401/403 live-service failures remain visible because the test resources have local authentication disabled; skipping them would create a false green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 476d113f-dd35-40c6-bc79-005dcccd7b79
Rana Singh (ranadeepsingh)
requested a review
from Brendan Walsh (BrendanWalsh)
as a code owner
September 2, 2026 10:26
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Hey Rana Singh (@ranadeepsingh) 👋! We use semantic commit messages to streamline the release process. Examples of commit messages with semantic prefixes:
To test your commit locally, please follow our guild on building from source. |
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
September 2, 2026 10:27
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The change is small, targeted, and backed by a concrete contract test that enforces the intended publish-step ordering.
Review tier: Lite
Findings: None
What changed in this PR
This PR hardens the Azure Pipelines publish step against “poisoned” restored SBT/Ivy/Coursier caches by running the existing sbt_retry.sh dependency recovery/eviction flow before packagePython, and adds a regression test to enforce the ordering contract.
Changes:
- Run
SBT_SETUP_MAX_STAGGER_SECONDS=0 bash tools/ci/sbt_retry.sh updateintemplates/publish.ymlbeforesbt packagePython. - Add a pipeline contract test to ensure dependency recovery precedes both
packagePythonandpublishBlob.
| File | Description |
|---|---|
| tools/ci/tests/test_pipeline_yaml.py | Adds a regression contract asserting the publish template runs cache recovery before packaging and blob publish. |
| templates/publish.yml | Inserts a pre-packaging dependency recovery step using the existing SBT retry/eviction helper. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
## Summary Migrate Cognitive Services, website samples, and the failing Databricks Forms notebook from disabled local-key authentication to short-lived Microsoft Entra tokens. Add Speech SDK AAD support and redact notebook request and response bodies from retry diagnostics. ## Prompting Intent Triage and fix every actionable failure in Azure Pipelines run 234106471 on master. Follow the SynapseML PR loop, preserve key-based public behavior and JVM compatibility, avoid credential exposure, and prove the replacement authentication path. ## Linked Sources - Failed pipeline: https://msdata.visualstudio.com/A365/_build/results?buildId=234106471&view=results - Pull request: microsoft#2691 - Azure Policy evidence: affected Cognitive Services accounts enforce disableLocalAuth=true ## Rationale Azure Policy prevents restoring local keys, so the tests now use the pipeline service connection and resource-scoped Cognitive Services User assignments. Shared helpers keep resource routing consistent. Speech retains subscription-key fallback because its SDK requires a service-specific AAD credential format. Databricks receives a fresh token only for the affected notebook, and body redaction prevents retry logs from printing it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 476d113f-dd35-40c6-bc79-005dcccd7b79
Rana Singh (ranadeepsingh)
changed the title
Collaborator
Author
|
/azp run |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
September 2, 2026 17:58
View session
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
A couple of verified issues in changed code paths can lead to corrupted CI artifacts on exceptions and to error-reporting failures masking the real HTTP failure context.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/RESTHelpers.scala — In the non-redacted path, requestBodyOpt uses Try(...).get, so if reading the request entity… |
Suppressed comments (1)
website/doctest.py:133
add_python_helper_to_markdowntruncates the markdown file before applying transformations. Ifuse_aad_for_ci_samplesor the regex replacement raises, the file will be left empty/corrupted, which can mask the real failure and make CI harder to debug. Prefer writing the updated content first, then truncating after a successful write (and rely on the context manager instead of callingclose()explicitly).
with io.open(os.path.join(folder, md), "r+", encoding="utf-8") as f:
content = f.read()
f.truncate(0)
content = use_aad_for_ci_samples(content, md)
content = re.sub("<!--pytest-codeblocks:cont-->", replacement, content)
f.seek(0, 0)
f.write(content)
f.close()
## Summary Complete the Entra migration for the remaining website, AI Foundry, Face-adjacent, and Speech validation paths. Preserve HTTP diagnostics when request bodies are unreadable, make documentation updates failure-safe, retire the obsolete key-only custom Speech fixture in favor of deterministic endpoint configuration coverage, and apply the pinned notebook formatting. ## Prompting Intent The engineer asked to triage and fix every failure from the latest master CI/CD run using the full SynapseML PR loop, then keep iterating on exact-head validation and automated review findings until the change is merge-ready without exposing credentials or weakening meaningful coverage. ## Linked Sources - Original Azure Pipelines failure: https://msdata.visualstudio.com/A365/_build/results?buildId=234106471&view=results - First exact-head validation: https://msdata.visualstudio.com/A365/_build/results?buildId=234189609&view=results - Pull request: microsoft#2691 - REST diagnostics review finding: microsoft#2691 (comment) ## Rationale Rewrite authentication independently from fluent setter adjacency so all executable Python samples move to the correct Entra endpoint shape, and route the one-off AI Foundry stage through the shared test helper. The legacy custom Speech endpoint has no discoverable policy-compliant owning resource, so replace its brittle key-backed cloud assertion with a deterministic check that SynapseML applies the endpoint ID to the Speech SDK while retaining the suite's live Entra recognition coverage. Render diagnostics and transformed markdown defensively so secondary read or transformation failures cannot hide the original error or truncate source files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 476d113f-dd35-40c6-bc79-005dcccd7b79
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
September 2, 2026 20:43
View session
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The Speech SDK AAD path currently passes a formatted aad#resourceId#token string into SpeechConfig.setAuthorizationToken, which is likely invalid for the underlying SDK and risks breaking runtime authentication.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Lite
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
 |
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/speech/SpeechToTextSDK.scala — SpeechConfig.setAuthorizationToken expects the raw Entra access token (JWT). Here the code passes… |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
core/src/main/scala/com/microsoft/azure/synapse/ml/io/http/RESTHelpers.scala — In the non-redacted path, requestBodyOpt uses Try(...).get, so if reading the request entity… View resolved comment |
## Summary Move the IdentifyFacesSuite fixture from subscription-exhausted PersonGroup storage to LargePersonGroup while preserving the live end-to-end identification assertions. Restrict stale fixture cleanup to timestamped test-owned groups and apply the formatter output required by the hosted Style job. ## Prompting Intent The engineer asked to triage and fix every actionable failure from the latest master CI validation using the SynapseML PR loop, preserve meaningful service coverage, and distinguish product defects from hosted-agent infrastructure faults. ## Linked Sources - Original failed build: https://msdata.visualstudio.com/A365/_build/results?buildId=234106471&view=results - Exact-head validation: https://msdata.visualstudio.com/A365/_build/results?buildId=234213031&view=results - Pull request: microsoft#2691 - Face service quotas: https://learn.microsoft.com/en-us/azure/ai-services/face/identity-quotas-limits ## Rationale Direct AAD probes showed that standard PersonGroup person creation returns subscription-level QuotaExceeded even for an empty new group, while LargePersonGroup creation, person enrollment, training, and identification succeed on the same S0 resource. Using the transformer's existing largePersonGroupId support retains real service validation without deleting data in other resources, requesting unnecessary capacity, or weakening the test to a mock or skip. The Python change follows the exact hosted formatter diff rather than changing pipeline tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 476d113f-dd35-40c6-bc79-005dcccd7b79
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
September 2, 2026 22:53
View session
Collaborator
Author
|
/azp run |
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
September 3, 2026 09:17
View session
Rana Singh (ranadeepsingh)
changed the title
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
It changes CI publishing flow and live credential/region routing across many suites, which is high-impact and best confirmed with end-to-end CI evidence before approval.
Review tier: Lite
Findings: None
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #2691 +/- ##
==========================================
+ Coverage 86.89% 87.41% +0.52%
==========================================
Files 341 341
Lines 20742 20742
Branches 2166 2166
==========================================
+ Hits 18023 18132 +109
+ Misses 2719 2610 -109 🚀 New features to boost your workflow:
|
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Collaborator
Author
Final readiness evidence
Remaining blocker: repository-required human approval ( |
## Summary Restore subscription-key authentication for the custom entity recognition and multi-label classification transformer tests, and remove the test-only AAD token helper that is no longer used. ## Prompting Intent The engineer asked why the dedicated mmlspark-cs-language resource was used and required reusable transformer examples to configure .setSubscriptionKey(languageApiKey) rather than replacing key authentication with AAD. ## Linked Sources - Pipeline failure: https://msdata.visualstudio.com/A365/_build/results?buildId=234106471&view=results - Pull request: microsoft#2691 - Language resource setup: https://learn.microsoft.com/azure/ai-services/language-service/custom-named-entity-recognition/how-to/create-project - Regional support: https://learn.microsoft.com/azure/ai-services/language-service/concepts/regional-support ## Rationale The two custom projects and trained deployments are owned by the dedicated East US Language resource. Central US is prediction-only for these features, and the existing multi-service resource rejects Language authoring requests, so it cannot replace the authoring resource. Re-enabling local authentication under the resource's existing policy exemption preserves the explicit key-based examples and avoids an unnecessary AAD-only test dependency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 476d113f-dd35-40c6-bc79-005dcccd7b79
Copilot started reviewing on behalf of
Rana Singh (ranadeepsingh)
September 4, 2026 00:43
View session
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Brendan Walsh (BrendanWalsh)
approved these changes
Sep 4, 2026
Rana Singh (ranadeepsingh)
disabled auto-merge
September 4, 2026 00:52
Rana Singh (ranadeepsingh)
enabled auto-merge (squash)
September 4, 2026 00:53
Rana Singh (ranadeepsingh)
disabled auto-merge
September 4, 2026 00:57
Rana Singh (ranadeepsingh)
merged commit Sep 4, 2026
2581574
into
microsoft:master
16 of 80 checks passed
5 tasks
Rana Singh (ranadeepsingh)
added a commit
to ranadeepsingh/SynapseML
that referenced
this pull request
Sep 4, 2026
## Summary Require the trusted Azure Pipelines trigger helper and completeness summary to accept only the exact current-head safe verdict. Align the maintainer workflow documentation and mocked trigger behavior with that fail-closed rule. ## Prompting Intent The engineer asked for Copilot review to determine whether /azp run is safe before maintainers authorize credential-bearing Azure Pipelines. The current-head review found that a missing machine-readable verdict could still pass the trusted helper, so this change closes that authorization gap without allowing maintainer SHA confirmation to substitute for safety evidence. ## Linked Sources - Copilot review: microsoft#2692 (review) - High-severity finding: microsoft#2692 (comment) - Follow-up to PR microsoft#2691: microsoft#2691 ## Rationale Requiring the exact safe marker is simpler and safer than treating an absent marker as implicit approval. The separate maintainer SHA attestation remains necessary but cannot establish facts the review did not emit; instruction-changing bootstrap PRs continue to require independent review and manual authorization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 476d113f-dd35-40c6-bc79-005dcccd7b79
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
mmlspark-cs-centralresource incentralus..setSubscriptionKey(...)and.setLocation(...)calls so test configurations remain reusable examples.cognitive-api-key-centralKey Vault secret without changing existing eastus secrets.openai-api-key-3/synapseml-openai-3resource.Root cause
Azure Pipelines build 234106471 had 15 failing jobs:
packagePythonbefore validating the restored dependencies.disableLocalAuth=true, producing HTTP 401 and 403 responses.The blocked shared resource was
mmlspark-csin resource groupmarhamil-mmlspark, regioneastus. This PR does not weaken that policy. Ordinary tests instead usemmlspark-cs-central, whose local authentication is enabled, with the matchingcentralusendpoint. The central key returns HTTP 401 against the old eastus endpoint, so key and region are changed together.Compatibility and security
mmlspark-cs-central; the temporary Face Contributor role used for diagnosis was removed.Evidence
af6dc5208e490ab9ccc98314e1288f7127c29330: 64 succeeded, andUnitTests lightgbm4succeeded with issues only because three non-gating Azure cache restores hit an external SSL error. The build published 3,435 tests with 3,413 passed, 22 not applicable, and zero failed..setSubscriptionKey(languageApiKey)against the matching East US resource.azure-c-sharedplatform initialization error 2153; REST Speech with the same central key succeeds. Hosted Azure validation is required for that SDK path.Current exact head:
c8c5ddde6b29c683a6a8c92e742aca9f05882098Status: exact-head automated review and the 65-job Azure pipeline are pending after restoring custom Language key authentication.