fix: ARC/DinD pre-create mount dirs, install docker compose, and pull build-tools image#42906
Conversation
AWF validates that --mount host paths exist before starting containers.
On ARC/DinD, the compiler emits mounts for ${RUNNER_TEMP}/gh-aw/home and
${RUNNER_TEMP}/gh-aw/sandbox/agent which may not exist yet (only the parent
${RUNNER_TEMP}/gh-aw/ is created by actions/setup).
Add mkdir -p for these directories in the generated shell script, right
after the DOCKER_HOST detection probe and before the AWF invocation.
Error seen in canary run 28566765653:
Invalid volume mount: ...gh-aw/home:...gh-aw/home:rw
Reason: Host path does not exist: ...gh-aw/home
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
ARC/DinD runners may not have the Docker Compose v2 CLI plugin pre-installed. AWF requires 'docker compose' to orchestrate its containers (squid-proxy, agent, api-proxy). Add a generated step that downloads and installs docker-compose v2.36.2 into $DOCKER_CONFIG/cli-plugins/ before the AWF invocation. Error seen in canary: 'unknown shorthand flag: -d in -d' when AWF tried to run 'docker compose up -d'. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
AWF uses --pull never when starting containers, so all images must be pre-downloaded. The build-tools image (used as the sysroot-stage init container on arc-dind topology) was missing from the download list, causing: 'No such image: ghcr.io/github/gh-aw-firewall/build-tools:0.27.21' Add build-tools to collectDockerImages() when arc-dind topology is active. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR improves ARC/DinD runner topology support by ensuring required host mount directories exist before AWF runs, guaranteeing Docker Compose v2 availability for AWF orchestration, and pre-pulling the AWF build-tools image that ARC/DinD needs when --pull never is used.
Changes:
- Pre-create ARC/DinD bind-mount source directories before invoking AWF.
- Install the Docker Compose v2 CLI plugin on ARC/DinD runners.
- Include the AWF
build-toolsimage in the docker image pre-download set for ARC/DinD.
Show a summary per file
| File | Description |
|---|---|
| pkg/workflow/nodejs.go | Adds ARC/DinD-specific Docker Compose plugin install step to the AWF-injected npm engine install sequence. |
| pkg/workflow/docker.go | Ensures the AWF build-tools image is included in collected images for ARC/DinD + firewall-enabled workflows. |
| pkg/workflow/copilot_engine_installation.go | Introduces a workflow step generator that installs the Docker Compose v2 CLI plugin. |
| pkg/workflow/awf_helpers.go | Creates required ARC/DinD mount directories before AWF invocation. |
| pkg/workflow/awf_helpers_test.go | Adds a unit test asserting ARC/DinD mount directories are pre-created in the generated AWF command. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 5/5 changed files
- Comments generated: 3
- Review effort level: Low
| // Add build-tools sysroot image for ARC/DinD topology. | ||
| // AWF uses this as an init container to populate the sysroot volume with | ||
| // system binaries (gcc, make, libraries) that are invisible on the DinD daemon's FS. | ||
| if isArcDindTopology(workflowData) { | ||
| buildToolsImage := constants.DefaultFirewallRegistry + "/build-tools:" + awfImageTag | ||
| if !setutil.Contains(imageSet, buildToolsImage) { | ||
| images = append(images, buildToolsImage) | ||
| imageSet[buildToolsImage] = struct { | ||
| }{} | ||
| dockerLog.Printf("Added AWF build-tools sysroot container for arc-dind: %s", buildToolsImage) | ||
| } | ||
| } |
There was a problem hiding this comment.
Added TestCollectDockerImages_BuildToolsForArcDind in pkg/workflow/docker_build_tools_test.go (commit $(git rev-parse --short HEAD)). It covers four cases: image included when arc-dind + firewall enabled, excluded without arc-dind topology, excluded when firewall is disabled, and correct tag used with a custom AWF version.
There was a problem hiding this comment.
Added TestCollectDockerImages_BuildToolsForArcDind in pkg/workflow/docker_build_tools_test.go (18d8420). Covers: image included when arc-dind + firewall enabled, excluded without arc-dind topology, excluded when firewall is disabled, and correct tag used with a custom AWF version.
| // Add build-tools sysroot image for ARC/DinD topology. | ||
| // AWF uses this as an init container to populate the sysroot volume with | ||
| // system binaries (gcc, make, libraries) that are invisible on the DinD daemon's FS. | ||
| if isArcDindTopology(workflowData) { | ||
| buildToolsImage := constants.DefaultFirewallRegistry + "/build-tools:" + awfImageTag | ||
| if !setutil.Contains(imageSet, buildToolsImage) { | ||
| images = append(images, buildToolsImage) | ||
| imageSet[buildToolsImage] = struct { | ||
| }{} | ||
| dockerLog.Printf("Added AWF build-tools sysroot container for arc-dind: %s", buildToolsImage) | ||
| } |
There was a problem hiding this comment.
Valid observation. The embedded action_pins.json is synced from .github/aw/actions-lock.json via make sync-action-pins, which is populated during the gh aw update / release cycle. The build-tools image has never been included in that cycle because it's a new image type — the existing pinned versions only cover agent, squid, api-proxy, cli-proxy, and agent-act. The release workflow (release.md) validates that all container images referenced by compiled workflows have SHA pins before releasing, so the first release that includes this will require the pins to be present in actions-lock.json. Adding the digest pins requires registry access to fetch them — that needs to happen as part of the gh aw update pin-refresh process, which is the appropriate path for all firewall images. Flagging this as a prerequisite for the first release that ships arc-dind support.
…RC/DinD
On ARC runners, setup-node may find a pre-cached node at the original
tool cache (/home/runner/_work/_tool/) which is NOT under RUNNER_TEMP
and therefore not bind-mounted into the AWF container. Add a step after
runtime setup that copies node to the redirected tool cache if needed.
Also add a step to copy the Copilot CLI binary from /usr/local/bin
(where install_copilot_cli.sh places it) to ${RUNNER_TEMP}/gh-aw/bin/
which is the daemon-visible path the AWF command references.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ARC/DinD
On ARC/DinD, /tmp/gh-aw/ is on the runner's filesystem but NOT visible to
the Docker daemon. The activation job writes prompt files to
/tmp/gh-aw/aw-prompts/, but AWF can only access daemon-visible paths.
Two fixes:
1. Copy /tmp/gh-aw/aw-prompts/ to ${RUNNER_TEMP}/gh-aw/aw-prompts/ before
AWF invocation (in the arcDindDockerHostProbe preamble)
2. Change --prompt-file path in copilot engine from /tmp/gh-aw/... to
${RUNNER_TEMP}/gh-aw/... when arc-dind topology is active
Other engines (Claude, Codex) have the same hardcoded /tmp/ path and will
need similar fixes for arc-dind support.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@copilot run pr-finisher skill |
… arc-dind Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
The post-run steps (firewall log parsing, artifact upload) used hardcoded
/tmp/gh-aw/sandbox/firewall/ paths. On ARC/DinD, AWF writes logs to
${RUNNER_TEMP}/gh-aw/sandbox/firewall/ (daemon-visible). This mismatch
caused 'No log sources found' errors and empty artifact uploads.
Fix generateFirewallLogParsingStep(), generateSquidLogsUploadStep(), and
the unified artifact path list to use ${RUNNER_TEMP} paths when arc-dind
topology is active.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@copilot please run the
|
Actions taken: No code changes needed in this pass; all three merge-ready conditions were already satisfied by prior commits. |
|
Hey @lpcox 👋 — great work tracking down and fixing these three interrelated ARC/DinD pain points! The pre-create of rw mount directories, Docker Compose plugin installation, and inclusion of the The PR is well-scoped — all changes live in
|
|
@copilot please run the Unresolved review threads to address (newest first):
|
Actions taken: No code changes in this pass. Branch is up to date and all local checks are green. |
…ith blocks
${RUNNER_TEMP} is a shell variable that only expands in `run:` blocks.
In `env:` and `with:` blocks, GitHub Actions only expands ${{ }}
expressions. The firewall log summary step and artifact upload paths
were using ${RUNNER_TEMP} which was passed as a literal string, causing
'No log sources found' even though AWF_LOGS_DIR was set.
Fix: use ${{ runner.temp }} in env/with contexts for ARC/DinD topology.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot please run the
|
Actions taken: No code changes. Branch is current and local validation is green. |
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Picks up the pre-shutdown squid log chmod fix (#5817) which resolves artifact permission errors on ARC/DinD runners.
|
@copilot resolve the merge conflicts in this pull request |
…unt-paths # Conflicts: # .github/workflows/plan.lock.yml Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Resolved. The only conflict was in |
|
🎉 This pull request is included in a new release. Release: |
Summary\n\nFixes ARC/DinD (Actions Runner Controller + Docker-in-Docker) topology failures by addressing path visibility constraints: on ARC/DinD,
/tmp/gh-aw/is not accessible to the Docker daemon, so all mounts, binaries, prompts, and log paths must reside under${RUNNER_TEMP}/gh-aw/(a bind-mounted, daemon-visible location). Also bumpsDefaultFirewallVersiontov0.27.22.\n\n### Changes\n\n#### Runtime / mount fixups (pkg/workflow/awf_helpers.go)\n\n- Pre-creates rw mount source directories (${RUNNER_TEMP}/gh-aw/home,${RUNNER_TEMP}/gh-aw/sandbox/agent) before AWF invocation. AWF validates that mount source paths exist before starting containers; the parent dir is created byactions/setupbut the subdirectories may not exist.\n- Copies prompt files from/tmp/gh-aw/aw-prompts/to${RUNNER_TEMP}/gh-aw/aw-prompts/so they are visible to the Docker daemon.\n\n#### Node.js daemon-visibility (pkg/workflow/compiler_yaml_main_job.go)\n\n- Inserts "Ensure Node.js is at daemon-visible path" setup step for ARC/DinD.setup-nodemay cache node under/home/runner/_work/_tool/node/..., which is not bind-mounted into the AWF container. The step copies node to${RUNNER_TEMP}/gh-aw/tool-cache/nodeif needed and setsGH_AW_NODE_BIN.\n-collectArtifactPaths: for ARC/DinD with firewall, uses${{ runner.temp }}/gh-aw/...(Actions expressions) instead of/tmp/gh-aw/...constants, becauseactions/upload-artifact'swith:block does not expand shell variables.\n\n#### Copilot CLI daemon-visibility (pkg/workflow/nodejs.go,pkg/workflow/copilot_engine_execution.go)\n\n- Adds "Copy Copilot CLI to daemon-visible path" step for ARC/DinD + firewall: copies/usr/local/bin/copilotto${RUNNER_TEMP}/gh-aw/bin/copilot.\n- Prompt file path inbuildCopilotBaseCommandnow switches to${RUNNER_TEMP}/gh-aw/aw-prompts/prompt.txtfor ARC/DinD (both sandbox and non-sandbox modes).\n\n#### Docker Compose installation (pkg/workflow/copilot_engine_installation.go,pkg/workflow/nodejs.go)\n\n- NewgenerateDockerComposeInstallStep(): downloads Docker Compose CLI plugin v2.36.2 for the runner's arch (x86_64 or aarch64). Injected into the install sequence for ARC/DinD runners, which may not have it pre-installed. AWF requires Docker Compose to orchestrate the squid-proxy, agent, and api-proxy containers.\n\n#### Build-tools sysroot image (pkg/workflow/docker.go)\n\n-collectDockerImages: for ARC/DinD + firewall, appendsbuild-tools:<awfImageTag>to the image pull list. AWF uses this as an init container to populate the sysroot volume with system binaries (gcc, make, libraries) that are not visible on the DinD daemon's filesystem.\n\n#### Firewall log paths (pkg/workflow/engine_firewall_support.go)\n\n-generateSquidLogsUploadStep: now acceptsworkflowData; uses${{ runner.temp }}/gh-aw/sandbox/firewall/logs/for ARC/DinD in thewith:block.\n-generateFirewallLogParsingStep: uses${RUNNER_TEMP}/gh-aw/sandbox/firewall/logsinrun:and${{ runner.temp }}/gh-aw/sandbox/firewall/logsinenv:for ARC/DinD (Actions expressions are required inenv:blocks, shell vars inrun:blocks).\n\n#### Version bump (pkg/constants/version_constants.go)\n\n-DefaultFirewallVersion:v0.27.20→v0.27.22.\n\n### Tests\n\n-pkg/workflow/awf_helpers_test.go:TestBuildAWFCommand_ArcDindPreCreatesMountDirs— verifiesmkdir -pfor mount source dirs and--mountflags for rw paths.\n-pkg/workflow/docker_build_tools_test.go(new):TestCollectDockerImages_BuildToolsForArcDindwith 4 sub-tests covering inclusion/exclusion ofbuild-toolsimage across topology and firewall combinations.\n-pkg/workflow/testdata/TestWasmGolden_*/: all golden files updated to reflect version bump and new ARC/DinD steps.\n\n### Affected topologies\n\nAll changes are guarded byisArcDindTopology(). Standard (non-ARC/DinD) runner behaviour is unchanged.