feat: enable ABUpdate for ACL - #28
Open
bfjelds (bfjelds) wants to merge 27 commits into
Open
Conversation
bfjelds (bfjelds)
changed the base branch from
user/bfjelds/ab-trident-verity-hash-partitions-20260708
to
aclmain
July 13, 2026 20:38
bfjelds (bfjelds)
changed the title
There was a problem hiding this comment.
Pull request overview
This PR enables Trident-driven A/B updates for Azure Container Linux (ACL) images by making /usr dm-verity boot slot-agnostic (primary UKI path via systemd-stub addons) and by updating the GRUB fallback cmdline construction to avoid hard-coding partition identity/inline hash offsets. It also adds Trident to the image and enables its socket.
Changes:
- Introduce dedicated
/usrdm-verity hash partitions (HASH-A / HASH-B) and plumb averity_hashcross-reference throughdisk_util+ the UKI disk layout. - Switch UKI/systemd-boot to a slot-independent main cmdline plus per-slot
verity.addon.efiaddons, and switch GRUB to UUID-basedsystemd.verity_usr_*parameters injected at build time. - Add
tridentto the RPM package catalog and enabletridentd.socketin the built image.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| build_library/rpm/uki_install.sh | Reworks UKI cmdline to be slot-agnostic and adds generation of per-slot verity addons. |
| build_library/rpm/rpm_install.sh | Fixes punctuation in a log message. |
| build_library/rpm/package_catalog.yaml | Adds the trident RPM to the package set. |
| build_library/rpm/grub.cfg | Updates GRUB verity cmdline template to use injected FS/verity UUIDs instead of inline hash-offset logic. |
| build_library/rpm/grub_install.sh | Injects FS UUID + verity UUID into GRUB template during provisioning. |
| build_library/rpm/build_image_util.sh | Enables tridentd.socket via a sockets.target wants symlink. |
| build_library/disk_util | Adds dps-usr-verity type resolution, supports separate hash partitions, and emits FS/verity UUIDs. |
| build_library/disk_layout_uki.json | Adds HASH-A/HASH-B partitions and links USR-A/USR-B to them via verity_hash. |
| build_library/build_image_util.sh | Plumbs new UUID output files into bootloader install arguments. |
| acl/docs/architecture.md | Updates architecture docs for dedicated hash partitions / slot-agnostic verity boot. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
bfjelds (bfjelds)
added a commit
that referenced
this pull request
Jul 14, 2026
Copilot re-review follow-ups on PR #28: - disk_util: the --fs_uuid output was only written in separate-hash-partition mode, so legacy inline-verity layouts produced no FS UUID file and now hard- fail in grub.cfg generation. Collapse the two near-identical verity branches into a single contextlib.ExitStack path that reads the FS UUID in both modes, removing the duplicated veritysetup/blkid logic. - rpm/uki_install.sh: correct the stale UUID-validation comment (grub_install.sh now dies rather than warns; uki_install does not pass UUIDs through itself). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bfjelds (bfjelds)
added a commit
that referenced
this pull request
Jul 20, 2026
disk_util: normalize root-hash/verity-uuid block indentation and use `is not None` for root_hash. uki_install.sh: validate UUID file content (non-empty), not just existence, so a blank UUID file fails fast. grub_install.sh: hard-fail when verity is enabled but no hash file is provided, matching the sibling FS/verity UUID checks (a warn + empty substitution produces an unbootable grub.cfg). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
disk_util: normalize root-hash/verity-uuid block indentation and use `is not None` for root_hash. uki_install.sh: validate UUID file content (non-empty), not just existence, so a blank UUID file fails fast. grub_install.sh: hard-fail when verity is enabled but no hash file is provided, matching the sibling FS/verity UUID checks (a warn + empty substitution produces an unbootable grub.cfg). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
The "verity enabled but no hash file" else-branch (warn + empty @@USRHASH@@ substitution) predates this PR and is a working fallback. Changing it to a hard die altered pre-existing behavior that was not introduced by this change, so restore the original warn. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
uki_install.sh runs under switch_to_strict_mode (set -e + ERR trap). With jq -e, a missing/null .uuid makes jq exit 1, tripping the ERR trap with a generic error before the explicit [[ -z || == null ]] check can emit the clear "Partition has no uuid field" die message. Dropping -e lets jq emit null (exit 0) so the explicit check produces the intended message. Also matches the sibling matches=jq-r lookup which omits -e. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
Install a UKI addon setting systemd.default_device_timeout_sec=120 on the arm64 kola test image only (gated on ARCH=arm64 and INJECT_DOCKER_SYSEXT=true). Under kola's parallel QEMU-TCG emulation on aarch64, CPU contention can keep udev from initialising the ESP/OEM/usr-verity devices within the default initrd device timeout, dropping the VM to an emergency shell. Raising the timeout via a test-image ESP addon fixes this. Verified against a parallel=24 repro: 18/18 boot-to-login vs 18/18 emergency shells at baseline. Scoped to the test image, so production and amd64 boot behaviour is unchanged; the failure is a CI-only emulation artefact that never occurs on real hardware. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
- install_uki_timeout_addon: require exactly one UKI on the ESP (die on multiple) instead of silently picking one via sort -V | head, so the addon can never land in the wrong <uki>.efi.extra.d. Mirrors the single-UKI rule in rpm/uki_install.sh. - RPM-mode tridentd.socket enable: hard-fail if tridentd.socket is absent. trident is required for ACL images, so a missing unit is a broken image; fail the build rather than leave a dangling wants-symlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
A present-but-empty verity_hash (e.g. "verity_hash": "") previously fell through to legacy inline-verity mode instead of separate-hash-partition mode, producing an invalid layout that is hard to debug. Strip and raise InvalidLayout when the value is empty, consistent with the other malformed- layout guards in this function. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
…ID wording Under switch_to_strict_mode (set -e), a non-zero exit from `sudo ukify build` aborted the script before the following existence check could emit the intended die message or clean up the temp dir. Wrap every ukify invocation (main UKI + firstboot/fips/kdump/debug/verity addons in uki_install.sh and the timeout addon in vm_image_util.sh) in an explicit `if ! ...; then rm -rf <tmp>; die; fi`, and clean up the temp dir on the produce-check path too. Failures now surface with a clear message and no leaked temp dirs. Also reword disk_util Verity's fs_uuid comment/exception from "btrfs FS UUID" to "filesystem UUID": the blkid read is filesystem-agnostic. (ACL only verities btrfs today, so this is wording-only.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
sudo ukify build can leave root-owned artifacts in the mktemp temp dirs, so a plain `rm -rf` could fail and (under strict mode) abort before the intended die. Use `sudo rm -rf` for every temp-dir cleanup in the UKI/addon builders (uki_install.sh main UKI + firstboot/fips/kdump/debug/verity addons, and the timeout addon in vm_image_util.sh), matching the existing sudo rm -rf convention used throughout build_library. Each temp dir is assigned via mktemp -d immediately before use, so the rm target is always a real path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 72f4b84a-8f5c-4df6-9f8e-fb4850c520e2
GRUB's disk_layout.json previously used the legacy single-partition inline-hash verity scheme. The UKI-side changes in this branch switched grub.cfg's verity cmdline construction to UUID=-based hash identification (dropping hash-offset=), which only works when the hash tree lives on its own independently-discoverable partition. Mirror disk_layout_uki.json's structure: add dedicated HASH-A/HASH-B partitions (dps-usr-verity type) cross-referenced via verity_hash, so blkid/udev can resolve systemd.verity_usr_hash=UUID=<verity_uuid> the same way UKI does. ACL-GRUB still cannot do Trident A/B updates (no addon-swap equivalent), but the partitioning/verity mechanics are now consistent between the two boot paths instead of silently broken on GRUB. ESP stays at 128MiB (no AB-staging need on GRUB). BIOS-BOOT partition is kept (shared with Portage/flatcar builds; RPM/GRUB mode already skips installing to it).
install_uki_timeout_addon already works around a CI-only QEMU-TCG flake on arm64 kola test images: heavy parallel-VM CPU contention can prevent udev from initialising ESP/OEM/usr-verity devices within the default initrd device timeout, dropping the VM to an emergency shell. That fix was UKI-only (built as a signed .efi addon dropped into <uki>.efi.extra.d/). GRUB has no addon mechanism, so add install_grub_timeout_override: since grub.cfg is plain text (not signed/embedded like a UKI), patch the already-written grub.cfg copies on the test image's ESP directly via sed, appending systemd.default_device_timeout_sec=120 to the shared linux_cmdline. Same scoping as the UKI addon: arm64 test image only (INJECT_DOCKER_SYSEXT=true), never applied to production images or amd64.
Addresses Copilot review comment on PR #28: install_grub_timeout_override logged \ patched\ unconditionally even if the sed substitution silently did not match (e.g. linux_cmdline format changed), leaving the timeout override ineffective without any signal. Now verifies the substitution took effect and dies with a clear message if not.
…check virsh net-info default | grep -q could report a false pipeline failure under set -o pipefail (inherited via sourcing from validate_common.sh): grep -q exits as soon as it matches, which can SIGPIPE virsh while it is still writing, making the whole pipeline non-zero even though the pattern matched. Capture output first, then grep the captured string.
trident-selinux pulls in policycoreutils-python-utils (for SELinux module install tooling), which transitively depends on the audit package. Azure Linux audit RPM enables+starts auditd via its %post systemd preset, unlike portage-mode builds which ship a blanket disable-all preset. Remove the enable symlink so RPM-mode images match upstream ACL behavior: auditd installed but disabled by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ACL images need trident-acl-agent (triggers AB-updates), not the full trident package. Keep the auditd.service disable workaround in build_image_util.sh regardless, since trident-acl-agent still depends on trident, and defense-in-depth is warranted independent of the trident.spec fix (microsoft/trident#734). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bfjelds (bfjelds)
force-pushed
the
user/bfjelds/ab-verity-partuuid-uki-addons-20260708
branch
from
August 14, 2026 14:52
4777b5e to
a4ac985
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (2)
acl/validate/validate_qemu.sh:57
validate_common.shenablesset -euo pipefail, sonet_info="$(virsh net-info ...)"will cause the script to exit immediately ifvirsh net-inforeturns non-zero (e.g., transient libvirt error), instead of falling back to the existing start-network logic. Also,net_infoshould belocalto avoid leaking a global into other sourced modules.
# Capture output first instead of piping directly into grep: under `set -o pipefail`
# (inherited from validate_common.sh, since this file is sourced not executed),
# grep -q can exit as soon as it matches while virsh is still writing, causing a
# SIGPIPE in virsh and a false pipeline failure even though the pattern matched.
net_info="$(virsh net-info default 2>/dev/null)"
if ! grep -q 'Active:.*yes' <<< "$net_info"; then
build_library/rpm/build_image_util.sh:602
- The build failure message references a missing “trident RPM”, but this PR adds
trident-acl-agenttopackage_catalog.yaml. Updating the message to name the expected RPM/package makes failures easier to debug.
# Enable tridentd.socket - listens for trident API requests.
# trident is a required component of ACL images: a missing unit means the
# trident RPM was not installed, i.e. a broken image. Fail the build now
# rather than leave a dangling wants-symlink and ship an image whose
# provisioning socket never activates.
if [[ ! -f "${root_fs_dir}/usr/lib/systemd/system/tridentd.socket" ]]; then
die "tridentd.socket not found in image - trident RPM missing (trident is required for ACL)"
fi
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
acl/validate/validate_qemu.sh:56
net_infois assigned withoutlocalinsideensure_libvirt_network(). Because this file is sourced (not executed), that can leaknet_infointo the caller’s/global scope and cause hard-to-trace variable collisions later in the validation run.
net_info="$(virsh net-info default 2>/dev/null)"
build_library/disk_layout_uki.json:13
- The ESP resize comment says "128 MiB -> 256 MiB", but this file’s previous value was already 192 MiB (393216 blocks). Updating the comment to match the actual delta avoids confusion when auditing layout history.
"_comment": "128 MiB -> 256 MiB (blocks are 512-byte sectors). UKI boot keeps kernel+initramfs on the ESP; the acl-t UKI rebuilt by Image Customizer and Trident A/B update addons both need more headroom than stock.",
Member
Author
|
it looks like github-acl-pr-validation doesn't use mantle ref provided (which would have fixes for verity failures). github-mantle-pr-validation has both mantle and acr fixes. azldevel has both mantle and acr fixes as well. |
- generate_grub_hashes.py: fix PCR policy allowlist entry for the OEM partition, which moved from gpt6 to gpt8 in this PR's renumbered disk_layout.json. The expected measured GRUB command is the OEM search result substituted into "source (hd0,gptN)/grub.cfg"; it must track the actual partition number or secure/measured boot policy checks will fail against real hardware measurements. - prod_image_util.sh: create_prod_tar() resolve ROOT and USR-A by GPT PARTLABEL via blkid instead of hardcoded partition numbers (p9/p3), which broke when ROOT moved from partition 9 to 11 in this PR. - disk_util / disk_layout.json: add an explicit legacy_verity_offset opt-in flag. When set on a verity_hash-bearing partition, Verity() now writes the hash tree to both the dedicated hash partition and the legacy inline offset within the data partition (same salt, so both copies are byte-identical and share one root hash). This keeps Portage/bootengine boots working via the existing hardcoded inline hash-offset activation, without requiring any bootengine or GRUB cmdline changes, while RPM/UKI-style consumers of the dedicated HASH-A/HASH-B partitions are unaffected. Verified end-to-end with veritysetup that both write targets produce an identical root hash and both independently pass veritysetup open. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 16 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
build_library/disk_layout_uki.json:13
- This layout history and sizing rationale are inaccurate: the previous value was 393216 sectors (192 MiB), not 128 MiB, and the A/B space pressure comes from staging a second arm64 UKI rather than from the comparatively small addon files. Please correct the comment so future ESP sizing changes are based on the actual baseline.
"_comment": "128 MiB -> 256 MiB (blocks are 512-byte sectors). UKI boot keeps kernel+initramfs on the ESP; the acl-t UKI rebuilt by Image Customizer and Trident A/B update addons both need more headroom than stock.",
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables Trident-driven A/B updates on ACL images by giving
/usrdm-verity a slot-agnostic boot path, installing/enabling Trident in the image, and expanding the ESP to support A/B UKI staging on arm64.Changes
Verity: dedicated hash partitions + slot-agnostic boot
disk_layout_uki.json: add dedicated verity hash partitions HASH-A and HASH-B (typedps-usr-verity) paired to USR-A/USR-B via a newverity_hashcross-reference; renumber OEM/ROOT accordingly.disk_util: adddps-usr-veritypartition-type GUID resolution and theverity_hashlayout key; add a separate-hash-partition verity mode (hash tree written to the dedicated partition instead of an inline hash-offset); emit the btrfs filesystem UUID and verity superblock UUID via new--fs_uuid/--verity_uuidflags; only the active (prioritized) slot is formatted at build time.build_image_util.sh: capture the new UUID outputs and pass them to bootloader install.UKI / systemd-boot (the boot path for ACL images).
rpm/uki_install.sh:mount.usr=/dev/mapper/usr,root=LABEL=ROOT, ...).verity.addon.efiin<uki>.efi.extra.d/) built per slot withsystemd.verity_usr_data=PARTUUID=<usr>andsystemd.verity_usr_hash=PARTUUID=<hash>plus the slot root hash. Slot A is active by default; Trident switches slots by swapping the active addon. This lets one UKI boot either slot without re-signing.GRUB (secondary bootloader path).
grub_install.sh(declare--fs_uuid/--verity_uuid),rpm/grub_install.sh(read and inject the values), andrpm/grub.cfg(template@@FSUUID@@/@@VERITYUUID@@): build the verity cmdline fromsystemd.verity_usr_data=UUID=<fs>andsystemd.verity_usr_hash=UUID=<verity>instead of a hard-coded PARTUUID + hash-offset.ESP capacity for A/B UKI staging
disk_layout_uki.json: increase the EFI System Partition from 128 MiB (262144 blocks) to 256 MiB (524288 blocks).Trident enablement
rpm/package_catalog.yaml: add thetridentpackage.rpm/build_image_util.sh: enabletridentd.socket.arm64 test-image boot reliability
vm_image_util.sh: addinstall_uki_timeout_addon, a UKI addon that raisessystemd.default_device_timeout_sec=120. Scoped to the arm64 kola test image only; it fixes a CI-only QEMU-TCG emulation flake where udev cannot initialise the ESP/OEM/usr-verity devices within the default device timeout and the VM drops to an emergency shell. Production and amd64 boot behaviour are unchanged.Docs
acl/docs/architecture.md: document the dedicated hash partition and slot-agnostic verity boot.Required PRs
Validation