fix: align containerd v2 configs with 2.3 schema

[Sylvain Boily (djsly)]

Summary

Fix the containerd 2.3.x / NVIDIA GPU Operator config-version mismatch by making AgentBaker-owned containerd 2.3 root configs use containerd config schema v4. This prevents NVIDIA-generated v4 drop-ins from being rejected under a lower-version root config while preserving containerd 2.0-2.2 schema-v2 compatibility.

Changes

• Updated legacy CSE containerd templates to choose schema v4 only for containerd >= 2.3.0, and to keep split-plugin schema v2 for containerd 2.0-2.2.
• Updated aks-node-controller/scriptless containerd templates to use the same schema-selection behavior, with explicit ContainerdVersion taking precedence and local detection as fallback.
• Added omitted-version fallback for known containerd-v2 distros so RP/custom-data that omits ContainerdVersion still renders the correct split-plugin config.
• Updated the Ubuntu GB static NVIDIA containerd config to schema v4 and split CRI plugin namespaces.
• Added NVIDIA CDI refresh repair after GPU driver readiness, plus a VHD-side guard to avoid early nvidia-cdi-refresh start-limit failures before the driver is available.
• Updated E2E parity conversion/defaulting so scriptful and scriptless containerd configs compare semantically across schema-v2 and schema-v4 paths.

Testing

• GOPROXY=https://proxy.golang.org,direct go test ./pkg/agent/...
• cd aks-node-controller && GOPROXY=https://proxy.golang.org,direct go test ./parser
• cd aks-node-controller && GOPROXY=https://proxy.golang.org,direct go test ./...
• GOPROXY=https://proxy.golang.org,direct make generate (generated agent testdata matched; command stopped during validate-shell on pre-existing SC3014 warnings in unrelated files)

Related

• AB#38808465

🤖 Generated by GitHub Copilot

[Copilot]

Pull request overview

Aligns AgentBaker-managed containerd v2 (Ubuntu 24.04) configuration with the containerd 2.3 config schema v4 so NVIDIA GPU Operator–generated v4 drop-ins are accepted, and updates runtime/plugin namespaces accordingly.

Changes:

• Updated Ubuntu 24.04 containerd v2 root config templates in pkg/agent to declare version = 4.
• Migrated Kata runtime blocks in the v2 templates to the split io.containerd.cri.v1.runtime plugin namespace.
• Updated the Ubuntu 24.04 GB static NVIDIA containerd config to schema v4 and the split images/runtime plugin namespaces.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
pkg/agent/baker.go	Updates Ubuntu 24.04 containerd v2 template schema to v4 and moves Kata runtime blocks to the split CRI runtime plugin namespace.
parts/linux/cloud-init/artifacts/ubuntu/gb/containerd-nvidia.toml	Updates the Ubuntu 24.04 GB NVIDIA containerd config to schema v4 and split CRI plugin namespaces.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

aks-node-controller/parser/templates/containerd.toml.gtpl:27

• When rendering config schema v4 (containerd >= 2.3), snapshotter / disable_snapshot_annotations should be configured under the images plugin (io.containerd.cri.v1.images) rather than under the runtime plugin’s .containerd section. Keeping these fields under [plugins."{{$runtimePlugin}}".containerd] means overlaybd artifact streaming (and potentially Kata’s snapshot-annotation behavior) may be ignored on v4 configs.

    disable_snapshot_annotations = false
    {{- end}}
    {{- if .GetEnableArtifactStreaming }}
    snapshotter = "overlaybd"
    disable_snapshot_annotations = false

aks-node-controller/parser/templates/containerd_no_GPU.toml.gtpl:23

• For config schema v4 (containerd >= 2.3), snapshotter / disable_snapshot_annotations need to be set on the images plugin (io.containerd.cri.v1.images), not under the runtime plugin’s .containerd table. As-is, the v4 path will still emit these fields under [plugins."{{$runtimePlugin}}".containerd], which containerd v2.3’s split plugins are unlikely to consume.

    {{- if .GetIsKata }}
    disable_snapshot_annotations = false
    {{- end}}
    {{- if .GetEnableArtifactStreaming }}
    snapshotter = "overlaybd"

[github-actions]

Windows Unit Test Results

  3 files   13 suites   43s ⏱️
404 tests 404 ✅ 0 💤 0 ❌
407 runs  407 ✅ 0 💤 0 ❌

Results for commit 9a4711d.

♻️ This comment has been updated with latest results.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

pkg/agent/baker.go:2002

• containerdSemverCore is now implemented in multiple places (also appears under aks-node-controller/parser/helper.go). Duplicated parsing logic is easy to drift over time (especially around edge cases like epochs and Debian ~ prerelease). Suggestion (optional but recommended): centralize this parsing in a shared package/util, or add shared test vectors to ensure both implementations stay behaviorally identical.

func containerdSemverCore(version string) string {
	version = strings.TrimSpace(version)
	if idx := strings.Index(version, ":"); idx > 0 {
		epoch := version[:idx]
		validEpoch := true
		for _, c := range epoch {
			if c < '0' || c > '9' {
				validEpoch = false
				break
			}
		}
		if validEpoch {
			version = version[idx+1:]
		}
	}
	version = strings.TrimPrefix(version, "v")
	if idx := strings.IndexAny(version, "-+~"); idx > 0 {
		version = version[:idx]
	}
	parts := strings.Split(version, ".")
	if len(parts) != 3 {
		return ""
	}
	for _, part := range parts {
		if part == "" {
			return ""
		}
		for _, c := range part {
			if c < '0' || c > '9' {
				return ""
			}
		}
	}
	return version
}

vhdbuilder/packer/install-dependencies.sh:841

• This function unconditionally runs systemctl --version. In build environments where systemctl isn't available/functional (e.g., image build context changes, minimal bases), this can fail the VHD build even though the guard is only needed at runtime. Suggestion: add a command -v systemctl check (and/or default systemd_version safely) before calling systemctl, so the script degrades gracefully when systemd tooling isn't present.

configureNvidiaCDIRefreshPreDriverReadinessGuard() {
  if [ "$OS" = "$UBUNTU_OS_NAME" ] && [ "$(isARM64)" -ne 1 ]; then # No ARM64 SKU with GPU now
    local systemd_version

    systemd_version=$(systemctl --version | awk 'NR == 1 { print $2 }')
    mkdir -p /etc/systemd/system/nvidia-cdi-refresh.service.d
    cat << 'EOF' > /etc/systemd/system/nvidia-cdi-refresh.service.d/aks-driver-ready.conf
[Unit]
ConditionPathExists=/proc/driver/nvidia/version
EOF

    if [ "${systemd_version:-0}" -ge 243 ]; then
      cat << 'EOF' >> /etc/systemd/system/nvidia-cdi-refresh.service.d/aks-driver-ready.conf
[Service]
ExecCondition=/bin/sh -c 'command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi -L >/dev/null 2>&1'
EOF
    fi
  fi
}

parts/linux/cloud-init/artifacts/cse_config.sh:1498

• There’s a new failure branch for the nvidia-cdi-refresh.path unit restart, but the added shellspec coverage only exercises the service failure path. Please add a test case that sets up NVIDIA_CDI_UNITS to include nvidia-cdi-refresh.path and makes systemctl start nvidia-cdi-refresh.path fail, asserting the function returns failure and emits the new error message.

    if nvidiaCDIRefreshUnitExists "${path_unit}"; then
        systemctl reset-failed "${path_unit}" || true
        if ! systemctl start "${path_unit}"; then
            echo "Failed to restart NVIDIA CDI refresh path unit after GPU driver is ready" >&2
            return $ERR_GPU_DRIVERS_START_FAIL
        fi
    fi

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Suppressed comments (3)

pkg/agent/baker.go:2164

• This second [proxy_plugins] declaration will collide with the earlier [proxy_plugins] emitted by the IsArtifactStreamingEnabled block when both are enabled, producing invalid TOML. Using a direct subtable header avoids redeclaring the parent table.

[proxy_plugins]
  [proxy_plugins.tardev]
    type = "snapshot"
    address = "/run/containerd/tardev-snapshotter.sock"

aks-node-controller/parser/templates/containerd.toml.gtpl:99

• This [proxy_plugins] header is a second declaration of the same table when artifact streaming is enabled earlier in the template, which can produce invalid TOML. Use a direct subtable header so the parent table is not redeclared.

[proxy_plugins]
  [proxy_plugins.tardev]
    type = "snapshot"
    address = "/run/containerd/tardev-snapshotter.sock"

aks-node-controller/parser/templates/containerd_no_GPU.toml.gtpl:81

• This [proxy_plugins] table header can duplicate the earlier [proxy_plugins] header emitted for artifact streaming, producing invalid TOML. Prefer a direct subtable header here.

[proxy_plugins]
  [proxy_plugins.tardev]
    type = "snapshot"
    address = "/run/containerd/tardev-snapshotter.sock"

[Xu Xue (xuexu6666)]

fixed

[Xu Xue (xuexu6666)]

fixed

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated no new comments.

Suppressed comments (2)

pkg/agent/baker.go:1921

• containerdV2ConfigTemplate / containerdV2NoGPUConfigTemplate now emit version = 4 (containerd config schema v4), so the constant names are misleading and make the selection logic harder to reason about (they look like the “containerd v2 schema” path used for pre-2.3). Consider renaming these templates to reflect schema v4 (e.g., containerdV4ConfigTemplate / containerdV4NoGPUConfigTemplate) and updating the callers accordingly to reduce future maintenance risk.

func containerdConfigTemplateForVersion(config *datamodel.NodeBootstrappingConfiguration, noGPU bool) ContainerdConfigTemplate {
	if shouldUseContainerdV4Config(config) {
		if noGPU {
			return containerdV2NoGPUConfigTemplate
		}
		return containerdV2ConfigTemplate
	}

aks-node-controller/app.go:478

• The doc comment for containerdConfigContentEqual says it ignores “whitespace differences”, but the implementation only trims leading/trailing whitespace (and strips TOML comments). Internal whitespace differences (e.g., within arrays) will still cause mismatches. Please adjust the comment to match the actual comparison behavior so callers don’t assume broader normalization than is implemented.

// containerdConfigContentEqual base64-decodes containerd TOML configs and compares
// table-scoped key/value pairs while ignoring harmless ordering and whitespace differences.
func containerdConfigContentEqual(a, b string) bool {

[Copilot]

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

[Xu Xue (xuexu6666)]

this is a behavior change and should not be in the current pr

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

e2e/validators_kata.go:112

• The updated comment says the containerd config schema AgentBaker renders for Kata is decided by the node’s containerd version, but in the scriptful path it’s actually driven primarily by the explicit NodeBootstrappingConfiguration.ContainerdVersion (and only falls back to distro inference when it’s empty). Since AgentPoolProfile.IsContainerdV2Distro() explicitly returns false for Kata distros, a Kata node with an omitted ContainerdVersion will still render the legacy grpc.v1.cri template even if the VHD boots containerd 2.x. Suggest clarifying this to avoid implying the behavior is purely runtime-version driven in all cases.

// containerd major version on the node is decided by the image, while the schema AgentBaker
// renders is decided by the node's containerd version. AgentBaker now hands each Kata distro a
// config that matches its containerd major (grpc.v1.cri for 1.x, the split cri.v1.runtime for
// 2.x), so AzureLinux V3 Kata boots containerd 2.x with a native 2.x config rather than relying
// on containerd's legacy migration of the "io.containerd.grpc.v1.cri" paths.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

aks-node-controller/app.go:481

• The doc comment block immediately above containerdConfigDiffDetail() starts by describing containerdConfigContentEqual(), but the next symbol is containerdConfigDiffDetail(). This is misleading (and breaks GoDoc association if someone navigates by comments). Consider updating/removing the first two lines so the comment accurately documents containerdConfigDiffDetail().

// containerdConfigContentEqual base64-decodes containerd TOML configs and compares
// table-scoped key/value pairs while ignoring harmless ordering and whitespace differences.
// containerdConfigDiffDetail returns a compact description of the canonical containerd TOML
// entries that differ between the provision-config (pc) and nbc-cmd values, so that
// scriptful/scriptless containerd parity drift is actionable from the logs instead of just
// reporting that the opaque base64 blob "differs". Returns "" for non-containerd keys.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.

Suppressed comments (1)

aks-node-controller/app.go:551

• containerdConfigContentEqual() only attempts base64.StdEncoding decoding on the raw env var values. When the nbc-cmd env comes from parsing a shell command it can be wrapped in quotes (KEY="") or omit padding (raw base64), which you already normalize for diagnostics in containerdConfigDiffDetail(). If decoding fails here, the function falls back to envValsEqual(), which prevents semantic comparison (order/whitespace-insensitive) and can report false diffs.

Consider reusing the same normalization (trim/strip quotes + RawStdEncoding fallback) in containerdConfigContentEqual so the parity check stays accurate even when the env value is quoted or unpadded.

func containerdConfigContentEqual(a, b string) bool {
	aDecoded, errA := base64.StdEncoding.DecodeString(a)
	bDecoded, errB := base64.StdEncoding.DecodeString(b)
	if errA != nil || errB != nil {
		return envValsEqual(a, b)

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.