fix(e2e): don't fail scriptless phase 3 on VHD-sourced GPU env skew

[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

Problem

Any PR that bumps a GPU driver version in parts/common/components.json currently cannot pass Agentbaker GPU E2E, and cannot be fixed before merge.

ValidateScriptlessPhase3 requires zero env var differences between two paths that have different sources of truth:

path	source of GPU_DRIVER_VERSION
provision-config	/opt/azure/components.json, baked into the VHD by vhdbuilder/packer/install-dependencies.sh
nbc-cmd	generated from the AgentBaker source under test

These only agree when the VHD was built from that same source. E2E deliberately does not do that — e2e/config/config.go:

SIGVersionTagValue string `env:"SIG_VERSION_TAG_VALUE" envDefault:"refs/heads/main"`

So a PR's code always runs against the newest VHD built from main. A PR that changes a GPU driver version therefore always produces:

env var differences (2): differs: GPU_DRIVER_VERSION; differs: GPU_IMAGE_SHA

The matching VHD can only be built after merge, so this is a deadlock rather than a regression in the change under test.

Evidence

Hit by the aks-gpu-grid 570.237 bump (#9229), GPU E2E build 176997071, scenario Test_Ubuntu2204_GPUA10 — failed identically on the initial run and both re-runs.

The node was otherwise completely healthy:

• CSE completed with exit code 0
• pulled and ran mcr.microsoft.com/aks/aks-gpu-grid:570.237-20260817204535
• nvidia-modprobe enabled and started successfully
• exactly 2 differing vars, both derived from the same components.json version string — every other env var matched
• Test_Ubuntu2404_GPUA10 passed on the very same new driver (it does not run the phase 3 comparison)

No GPU driver bump had landed since phase 3 was enabled in #8453 (2026-06-11), which is why this is the first PR to hit it.

Fix

Tolerate differences confined to the VHD-sourced GPU vars, and log them. Everything else still fails:

• any other differing var still fails, and the failure now names precisely which vars were unexpected
• the "no differences found" marker is still required to be absent before any tolerance applies
• if that marker is missing and no diff entries can be parsed, that now fails explicitly — this path is stricter than before, not weaker

Production behaviour is unchanged. aks-node-controller's compareEnvs still computes and reports the full diff — including these vars — as a guest agent event for Kusto.

I deliberately did not add these vars to isExpectedDiffCSEVar in aks-node-controller/app.go, since that would suppress the production observability signal. The artificial condition belongs to E2E, so the tolerance belongs in E2E.

Tests

New e2e/validators_envcompare_test.go, built around the exact log line from build 176997071:

• parses all three diff kinds (differs, only-in-pc, only-in-nbc)
• de-duplicates repeated log lines
• ignores unrelated log noise
• a non-GPU var mixed in with a GPU var is still reported as unexpected
• end-to-end: the real GPU-bump diff is tolerated

All pass, and go vet ./... is clean in e2e/.

Follow-up

Unblocks #9229 (aks-gpu-grid 570.237 bump), which will be rebased on this once merged.

[github-actions]

Windows Unit Test Results

  3 files   12 suites   47s ⏱️
389 tests 389 ✅ 0 💤 0 ❌
392 runs  392 ✅ 0 💤 0 ❌

Results for commit c466e61.

[Copilot]

Pull request overview

Adjusts the scriptless phase 3 E2E env-var comparison so GPU driver/image metadata that is intentionally VHD-sourced (from /opt/azure/components.json) no longer deadlocks GPU driver bump PRs, while keeping strict failures for any other unexpected env drift.

Changes:

• Adds parsing for compareEnvs diff markers and tolerates diffs confined to GPU_DRIVER_VERSION and GPU_IMAGE_SHA.
• Tightens the failure mode when the “no differences found” marker is absent but no diff entries can be parsed (explicit fail).
• Adds unit tests covering diff parsing, de-duplication, noise handling, and the real GPU-bump log line.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
e2e/validators.go	Parses env-compare diff entries and tolerates VHD-sourced GPU env skew while still failing on any other diffs.
e2e/validators_envcompare_test.go	Adds focused tests to validate parsing and ensure GPU-bump diffs are tolerated end-to-end.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

CI status: the two red checks are pre-existing infra flakes, not caused by this change

This PR changes only e2e/validators.go (ValidateScriptlessPhase3) plus a new unit test file. Both remaining failures are unrelated, and both fail on main independently.

Agentbaker GPU E2E — 1 failing test, infra

Build 177162394: the only failure is Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU, which never reached validation — the bastion tunnel failed on all 5 attempts, on all 3 runs:

ssh: handshake failed: failed to get reader: context deadline exceeded

For comparison, main build 177145051 failed 18 GPU tests with the same bastion signature, including this one. This PR failed exactly 1.

Two points confirm this change is inert here:

• the build logged zero env var differences, so it took the pre-existing "no differences found" path and the new tolerance branch never executed
• Test_Ubuntu2204_GPUA10 — the scenario this PR affects, and the one that fails on chore(gpu): bump aks-gpu-grid to 570.237-20260817204535 #9229 — passed

Agentbaker Windows E2E — chronic flake

Build 177162382: Test_Windows2022_AzureNetwork failed inside ValidateCollectWindowsLogsScript, where collect-windows-logs.ps1 failed to produce a zip:

New-Object System.IO.BinaryReader ... MethodInvocationException
collect-windows-logs.ps1 did not create a zip file

That pipeline has failed on refs/heads/main on 11 of the last 12 daily runs (2026-08-07 through 2026-08-17), with varying signatures. This PR touches no Windows code.

Green

Agentbaker E2E passed, as did all 20 other checks including go-test and lint.

[Ganeshkumar Ashokavardhanan (ganeshkumarashok)]

MIG_MultiGPU failure is a pre-existing ordering bug in ConfigureAndCreateVMSS — not this PR

This PR's actual change is working: the A10 env var differences (GPU_DRIVER_VERSION / GPU_IMAGE_SHA) skew failures are gone. The remaining Test_Ubuntu2404_NvidiaDevicePluginRunning_MIG_MultiGPU failure is unrelated to scriptless phase 3.

Evidence

Reproduced 3/3 in two independent builds (177162394 and the prior run), each at a near-identical ~257s:

vmss.go:460:      [61.748s]  VMSS ... created
bastionssh.go:304 [61.749s]  Attempt 1/5 establishing SSH over bastion to 10.89.128.5
bastionssh.go:323 [92.652s]  Attempt 1/5 SSH handshake failed: ssh: handshake failed: failed to get reader: context deadline exceeded
... (attempts 2-5, same) ...
bastionssh.go:323 [256.418s] Attempt 5/5 SSH handshake failed: ...
test_helpers.go:356 🔴 FAIL: failed to start bastion tunnel

The deterministic ~257s and the 3/3 reproduction rule out random bastion flake — this is a budget being exhausted.

Root cause

In e2e/vmss.go, the bastion dial runs before the VM is confirmed running:

line	step
418-424	DialSSHOverBastion(...) ← fixed budget: 5 attempts x 30s timeout + 4 x 10s backoff = 190s
425-427	create-operation error check
429-432	// Wait for VM to be in "Running" power state before proceeding waitForVMRunningState(...) ← waits up to 3 min

So we SSH into a VM whose power state we haven't yet waited for, and the hard-coded 190s dial budget expires before the patient waitForVMRunningState on the next line ever runs.

This scenario uses Standard_NC48ads_A100_v4 (2x A100, 48 vCPU). Boot on that SKU routinely exceeds 190s from VMSS-create, so the dial exhausts and the test hard-fails. failed to get reader: context deadline exceeded is exactly the signature of the bastion websocket opening while the target's sshd is not yet listening.

Note the scenario already declares WaitForSSHAfterReboot: 5 * time.Minute — but that only applies to RebootVMAndWaitForSSH (validators.go:328), not to this initial dial. The author's "this VM is slow" intent isn't honored on first connect.

Not a regression from #9165 or this PR

git blame shows the dial was inserted at these lines on 2026-01-20, placed above the pre-existing waitForVMRunningState (2025-11-07). #9165 fixed a different bastion fault (403 / EnableIPConnect on the shared bastion). This one is latent and only bites the slowest SKUs, which is why it looks intermittent — suli's runs show MIG_MultiGPU passing and MIG_Mixed failing, i.e. right at the margin.

Suggested fix (separate PR)

Reorder so waitForVMRunningState precedes the dial, and drive DialSSHOverBastion from a deadline rather than a fixed attempt count so slow GPU SKUs get a realistic budget.

Retrying this PR's GPU gate won't clear it — the fix has to land separately. Recommend not blocking this PR on that test.