Skip to content

{AKS} Stabilize AKS live test scenarios - #33805

Merged
Julie Zhu (yanzhudd) merged 3 commits into
Azure:devfrom
FumingZhang:fix/aks-live-test-stability
Aug 4, 2026
Merged

{AKS} Stabilize AKS live test scenarios#33805
Julie Zhu (yanzhudd) merged 3 commits into
Azure:devfrom
FumingZhang:fix/aks-live-test-stability

Conversation

@FumingZhang

@FumingZhang FumingZhang commented Jul 28, 2026

Copy link
Copy Markdown
Member

🤖 PR Validation — ️✔️ All clear

Breaking Changes Tests
️✔️ None ️✔️ 130/130

Related command

az aks

Description

Harden AKS live tests against the failure modes observed in CLI Runner:

  • Normalize quoted-empty service principal environment values and skip credential-dependent scenarios when credentials are unavailable.
  • Poll non-terminal cluster and agent-pool responses, then re-fetch exact cluster/agent-pool resources through native aks show commands before running checks.
  • Retry only the known transient AKS serialization conflicts (Another operation is in progress and PutExtensionAddonHandler.PUT) with bounded exponential backoff. Other errors and expected-failure scenarios still fail immediately.
  • Preserve feature-required regions for zone, Automatic SKU, Azure Container Storage, LocalDNS, and network-isolated tests instead of applying the runner-wide region override.
  • Select currently supported non-LTS Kubernetes versions for LocalDNS and network-isolated live tests.
  • Explicitly set the IPv4 managed outbound IP count in the dual-stack scenario.
  • Skip obsolete Basic Load Balancer scenarios because Basic LB retired on September 30, 2025.

Current dev also contains the newer AKS SDK/API version that resolves the maintenance and artifact-streaming payload mismatches seen in the original run. Azure Monitor LRO poller leaks are handled separately by #33734.

This PR changes test infrastructure only; customer-facing command behavior is unchanged.

Testing Guide

python -m pytest -q \
  src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_aks_provisioning_retry.py \
  src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_custom_preparers.py

Result: 27 passed, 7 subtests passed.

Focused scenario replay validation:

2 passed, 6 skipped

The skips are the intentionally live-only or retired scenarios. All changed files also pass python -m py_compile and git diff --check.

History Notes

None (test infrastructure only).


This checklist is used to make sure that common guidelines for a pull request are followed.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).

@FumingZhang
FumingZhang force-pushed the fix/aks-live-test-stability branch from 94f4e95 to fed9ff5 Compare July 28, 2026 06:32
@FumingZhang FumingZhang changed the title {AKS} Harden live test stabilization helpers {AKS} Stabilize AKS live test scenarios Jul 28, 2026
@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).

@a0x1ab

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).

FumingZhang and others added 3 commits August 4, 2026 03:23
Use the settled resource response after provisioning retries and treat quoted-empty injected service principal credentials as missing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Retry only known transient AKS operation conflicts, preserve required test regions, select supported Kubernetes versions, and retire Basic Load Balancer scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Distinguish transient operation-conflict retries from the native settled-resource refetch used after provisioning polling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@FumingZhang
FumingZhang force-pushed the fix/aks-live-test-stability branch from 67f9357 to 33bb099 Compare August 4, 2026 03:24
@FumingZhang
FumingZhang marked this pull request as ready for review August 4, 2026 03:45
@FumingZhang
FumingZhang requested a review from a team as a code owner August 4, 2026 03:45
Copilot AI lite review requested due to automatic review settings August 4, 2026 03:45
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens the AKS command module’s live-test scaffolding to reduce flaky failures in CI runners by normalizing missing credential inputs, adding bounded retries for known transient AKS conflicts, and ensuring assertions run against “settled” AKS resources re-fetched via native aks show/aks nodepool show.

Changes:

  • Add normalization for optional live-test environment settings (including quoted-empty placeholders) and ensure SP-dependent scenarios are skipped when credentials are unavailable.
  • Enhance the live-only ScenarioTest.cmd() adapter to retry known transient AKS operation conflicts and, after polling to Succeeded, re-fetch the exact cluster/agentpool resource for stable checks/return values.
  • Add/extend unit tests covering the new normalization behavior, cmd dispatch behavior when checks=None, transient-conflict retry behavior, and settled-result re-fetch behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_custom_preparers.py Adds unit tests validating normalization of optional settings and SP preparer skipping when credentials are quoted-empty.
src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_aks_provisioning_retry.py Expands unit coverage for live retry dispatch, settled-result refetching, and transient conflict retry logic.
src/azure-cli/azure/cli/command_modules/acs/tests/latest/test_aks_commands.py Updates the live-only cmd adapter to include transient conflict retries and to re-fetch settled AKS resources after polling.
src/azure-cli/azure/cli/command_modules/acs/tests/latest/custom_preparers.py Introduces _normalize_optional_live_test_setting and applies it to SP preparer settings to avoid false “present” credentials.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@FumingZhang

Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 3 pipeline(s).

@yanzhudd
Julie Zhu (yanzhudd) merged commit 49eed4e into Azure:dev Aug 4, 2026
50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

act-observability-squad AKS az aks/acs/openshift Auto-Assign Auto assign by bot

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants