STAC-25142 Move the agent CI to GitHub Actions and retire the GitLab pipelines - #444
STAC-25142 Move the agent CI to GitHub Actions and retire the GitLab pipelines#444LouisParkin wants to merge 52 commits into
Conversation
3cd1a54 to
7308812
Compare
76349d9 to
c2e9da2
Compare
…ration phase 1) Ports filename_linting + unbranded/branded unit_tests from .gitlab-ci-agent.yml to GitHub Actions, tracking the active stackstate-7.78.2 pipeline. - filename-linting on `docker` (cheap gate; only needs `git ls-files`). - unbranded/branded unit tests on `xlarge` inside the datadog_build image (tag 7af9194f), pulled via the read-only quay proxy. - 7.78.2 deltas vs 7.71.2: `--major-version` dropped from agent.build/inv test (removed upstream in DD 7.78), and branded runs scripts/verify_branding_literals.sh after fix_branding.sh. Test-execution policy (architect directive — minimize redundant runs): each suite runs once. filename-lint on every PR; the two heavy ~2h suites on pull_request, deduped via concurrency cancel-in-progress. They move to merge_group (run once at land) at phase 6; master/version-branch pushes then build+publish without re-running tests. Security: minimal contents: read, SHA-pinned checkout, persist-credentials false, no PATs, registry-proxy secret only via container.credentials, fork PRs skipped. Zizmor clean; actionlint clean apart from self-hosted labels. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…; conda needs bash)
…ds the 3.x version tag
…on phase 2) Port build_binaries and build_cluster_agent from .gitlab-ci-agent.yml to .github/workflows/build-binaries.yml: build the branded (StackState) production agent (+ rtloader) and cluster-agent in the datadog_build container on public xlarge runners, once per change (pull_request), deduped via concurrency. Carry over the GitLab pipeline speedups: self-contained jobs with no shared deps prerequisite (the Go module cache is untransportable between jobs and shared vendoring breaks the go.mod<->vendor reconcile), version.txt generated in-job from git, and a runner-local persistent-cache follow-up noted for the real warm-cache win. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e image Speeds up the migrated stackstate-agent GitHub Actions build/test jobs by replacing each job's cold `go clean -modcache` + `inv deps` (go mod download + tidy) reconcile with a warm GOMODCACHE delivered as a prebuilt CI image, keyed by a content hash of the module graph. Mirrors the StackGraph ci-metadata / build-ci-image pattern. - .github/scripts/agent-godeps-cache-metadata.sh: content-addressed image tag from every go.mod/go.sum + go.work + modules.yml plus the Dockerfile/script themselves, so any graph or mechanism change rotates the tag. - .github/docker/godeps-cache/Dockerfile: FROM datadog_build, bakes external modules into GOMODCACHE via a per-module `go mod download` loop. - .github/workflows/godeps-cache.yml: reusable workflow (tag lookup + build only when missing); pushes to quay.io/stackstate (private), pulled by consumers through the registry.tooling quay proxy with the existing REGISTRY_* creds. - build-binaries.yml / lint-and-unit-tests.yml: add the godeps-cache job, repoint the 4 heavy jobs to the cache image, drop the per-job modcache reset + inv deps (keep `go work sync` / `go work vendor` against the warm cache). Zizmor clean (narrow unpinned-images ignore on the content-addressed container refs, which are effectively digest-pinned but computed per run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c2e9da2 to
c07d0c8
Compare
RUN ran under dash so the read -d '' loop never iterated; -mod=mod is rejected in workspace mode.
STAC-25429 Warm the Go module cache with a prebuilt, content-addressed CI image
STAC-25459 Wire inv check-mod-tidy into CI, gated on the godeps cache hash
… STAC-25142-agent-omnibus-deb
STAC-25142 Add omnibus DEB build and container image workflows
The Cerberus Lambda already knows which channel to post to, so passing one
from the workflow is redundant. `notify.go` resolves the channel as:
Channel: util.GetOrDefault(req.Context, "channel", s.Channel)
and `util.GetOrDefault` (internal/util/context.go:27) treats an empty or
whitespace value as absent, falling back to the Lambda's own SLACK_CHANNEL
environment variable. Since this repo does not hold the secret, the previous
code was sending `channel: ""` on every call and hitting that fallback
anyway -- the field never did anything.
Dropping it halves what pulumi-infra has to provision for this repo
(StackVista/pulumi-infra#258 now asks for CERBERUS_LAMBDA_URL alone), which
matters because org-level secrets default to visibility=private and are
therefore invisible to this PUBLIC repo.
The `channel` key is removed from the payload entirely rather than sent
empty; GetOrDefault makes the two equivalent, and omitting it states the
intent. A comment records how to reintroduce it if agent failures should
ever go somewhere other than the shared CI channel.
Verified: actionlint clean (only the pre-existing self-hosted runner-label
warnings), zizmor clean, and the jq payload still emits valid JSON in the
shape Cerberus expects.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
STAC-25461 Drop SLACK_CI_REPORT_CHANNEL from the Cerberus notify path
Added commit: Cerberus bearer token (STAC-25536)Heads-up for anyone who already reviewed — one commit added on top: StackVista/cerberus#4 (STAC-24889) adds authentication to the Cerberus Lambda. The endpoint was previously completely unauthenticated — anyone who knew the URL could lock/unlock a protected branch or post as the Cerberus Slack bot. Every non-Slack request must now carry That PR is still open, so Sending the header now is forward-compatible (the deployed Lambda ignores unknown headers), so this is correct either side of that deploy and needs no coordination. Fixing it here rather than as a follow-up PR because this branch hasn't merged yet. Changes
Validation
The identical fix shipped for |
Cerberus now requires CI callers to present a shared bearer token. Send it from the notify script. Suspend xtrace around the curl and drop --verbose: the script runs under set -exuo pipefail, so either would echo the token into the job log.
StackVista/cerberus#4 (STAC-24889) adds authentication to the Cerberus Lambda: every non-Slack request must carry Authorization: Bearer <ssm:/cerberus/api-token> compared constant-time in internal/auth/auth.go verifyBearer. Before it, the endpoint was completely unauthenticated -- anyone who knew the URL could lock or unlock a protected branch or post as the Cerberus Slack bot. That PR is still open, so the workflow added in STAC-25461 would work as written today. The moment cerberus#4 deploys, the unauthenticated POST gets a 401, curl --fail turns the notify job red, and no Slack message is sent -- exactly the silent-failure condition the job exists to remove. Sending the header is forward-compatible: the deployed Lambda ignores unknown headers, so this is correct either side of that deploy and needs no coordination. CERBERUS_API_TOKEN already exists as a repo-level secret alongside CERBERUS_LAMBDA_URL, so no pulumi-infra change is needed. It is declared `required: false` for the same reason as the URL -- passing `${{ secrets.X }}` for a secret the repo lacks yields an empty string, which GitHub rejects against a *required* secret and fails the call before the guard can warn. The guard now covers both, so a missing secret still warns and exits 0 rather than adding a second red job to an already-failed run. Also drops `curl --verbose`, which echoes request headers -- the Authorization header now carries the shared token. GitHub would mask it, but not emitting it beats relying on masking. The same fix shipped for stackstate-process-agent as STAC-25529. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ed1f889 to
689909a
Compare
STAC-24889: authenticate Cerberus calls
|
Removed the whole-line comments from the five workflow YAML files in this PR, per house convention. Shell comments inside The change is behaviour-neutral by construction. I parsed every file before and after the removal and confirmed the loaded YAML documents are structurally identical, and re-ran actionlint and zizmor (no findings). Where a comment was carrying information a reader genuinely needs, the surrounding YAML already expresses it, for example through |
House convention is that we do not comment YAML files. The ported workflows carried a large block of explanatory commentary from the GitLab originals. Only whole-line YAML comments are removed. Shell comments inside run: blocks are script content and stay, as do the SHA-pin tag comments that make pinned action versions readable and the zizmor ignore directives, which are both functional rather than commentary. Verified by parsing each file before and after: the loaded YAML documents are structurally identical, so this is a pure comment removal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ports the publishing half of pre_release_main_agent_image and pre_release_cluster_agent_image, plus merge_docker_manifest_main_agent and merge_docker_manifest_cluster_agent, from GitLab to GitHub Actions. PR #446 landed the build-and-verify halves; this adds the push. Each image gets a per-arch publish job that needs: the existing image build job, and a manifest-merge job that assembles the multi-arch tag from the two single-arch ones. Gated on `github.event_name == 'push'` alone: the workflow's push filter only carries the release branch, so the event check is the whole gate, and a workflow_dispatch on an arbitrary branch cannot publish. This is the same reasoning the cerberus-notify gate already uses. Uses the StackVista/image-pipeline composite actions rather than an open-coded docker push, matching stackstate-process-agent and the STAC-24837 direction for product repos. Over publish_image.sh that adds cosign signatures in both v2 and v3 bundle formats, SBOM and max-mode provenance attestations, canonical SUSE Observability OCI labels, an entrypoint ELF-architecture check that catches arch-mismatched images before they are signed, and refusal to overwrite an existing tag. Neither Dockerfile declares `ARG BASE_IMAGE`, so base-name is passed explicitly; both final stages are registry.suse.com/bci/bci-micro. Tag is the 8-character short SHA, matching GitLab's CI_COMMIT_SHORT_SHA. The `<branch>-<arch>` tag publish_image.sh also pushed is deliberately dropped: helm-charts-internal pins the agent and cluster-agent images to an 8-character SHA (currently 9516cb4, the stackstate-7.78.2 HEAD), and beest receives the tag as AGENT_HASH_UNDER_TEST, so nothing consumes a branch-name tag. An org-wide code search for stackstate-k8s-agent:master, :stackstate-7*, :$CI_COMMIT_REF_SLUG and the cluster-agent equivalents returns no hits, verified against a positive control so an empty result is not a false negative. Both publish jobs need id-token: write for keyless cosign signing, and are added to each workflow's cerberus-notify needs list so a failed publish on the release branch still reaches Slack. Requires STAC-25541 (terraform-infra #75): the stackstate+agent robot behind QUAY_USER has no write grant on stackstate-k8s-agent or stackstate-k8s-cluster-agent, since GitLab publishes them with the legacy gitlabci robot instead. Without it these jobs fail exactly as process-agent did in STAC-25510. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…roxy BCI bases Address review feedback on #454. Gate publication on the image scan: build-agent-image and build-cluster-agent-image now run image-pipeline's scan-image in `mode: gate` at the house severity (UNKNOWN..CRITICAL, Grype enabled) directly after the smoke test. The publish jobs already depend on the build jobs, so a failing scan now blocks publication instead of only informing. Because build-agent-image also runs for same-repo pull requests, the gate executes on this PR's own CI. Make publication depend on all verification jobs: publish-agent-image now needs test-deb-renaming as well as build-agent-image, so a .deb that still carries DataDog branding cannot reach quay.io. test-deb-renaming lives in this workflow only, so the cluster-agent lane is unaffected. Pull the BCI bases through the SUSE proxy: both Dockerfiles take BCI_IMAGE_REGISTRY as a build argument, defaulting to registry.tooling.stackstate.io/suse/bci. This follows the existing BASE_IMAGE_REGISTRY convention in the same files, so every build path picks it up -- the GitHub workflows, the GitLab jobs, and build_images.sh -- without each caller having to pass a build argument. The apply-oci-labels base-name input stays on registry.suse.com so the provenance label keeps the upstream identity that VEX data is keyed on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tions The image scan gate added in this branch surfaced a real backlog: 24 findings in the agent image (embedded CPython 3.13.13, cryptography 48.0.1, two Go advisories) and 2 in the cluster-agent image. Secret scanning is clean on both. Keep the gate at mode: gate and declare every finding explicitly instead of weakening the severity filter, so nothing new can slip in unnoticed. Each exception carries a <=14-day expiry per the CVE remediation SLA, so the gate starts failing again if the underlying bumps stall. The omnibus runtime bumps (CPython 3.13.14, cryptography) are tracked in STAC-25556; they are software-definition changes, not workflow changes. LOCAL_IMAGE is renamed to the quay path the publish jobs already use because the evaluator keys exceptions on the normalised image name, which strips only the tag or digest and not the registry or namespace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…itHub Closes the last two GitLab jobs with no GitHub counterpart: sign_deb and pre_release_deb (.gitlab-ci-agent.yml lines 604 and 635). Security prerequisite, not cleanup ---------------------------------- sign_debian_package.sh called printenv unconditionally. On GitLab that dumped the GPG private key and its passphrase into the job log; stackstate-agent is a PUBLIC repo, so on GitHub that log is world-readable. It also left the exported private key in the checkout as gpg_private.key, where any later artifact upload would collect it. Both are removed before any signing secret is wired in. Key setup now happens in an ephemeral GNUPGHOME created with mktemp and mode 700, removed by an EXIT trap that also kills the gpg-agent so a preset passphrase cannot outlive the job on a reused runner. The passphrase reaches gpg through a mode-600 file inside that directory rather than argv. Fixes a latent signing bug -------------------------- The old preset step interpolated an unquoted command substitution into a single gpg-preset-passphrase call. A key exposes one keygrip per primary and subkey, so with a signing subkey the second keygrip was passed as a stray argument and never presetted, leaving signing able to block on a pinentry prompt no CI runner can answer. Each keygrip is now presetted individually. Verified against a throwaway key: two keygrips, both presetted. Shared setup ------------ sign_debian_package.sh and publish_package.sh need the same key in the same state but are separate processes, and on GitHub may be separate steps, so neither can rely on a keyring the other left behind. The setup moves into gpg_signing_setup.sh, sourced by both. gpg-preset-passphrase is auto-detected across /usr/lib/gnupg2, /usr/lib/gnupg, /usr/libexec and PATH, and the script fails loudly rather than silently skipping the preset when it is absent. Both scripts now fall back from CI_PROJECT_DIR to GITHUB_WORKSPACE, and publish_package.sh falls back from CI_COMMIT_REF_NAME to GITHUB_REF_NAME, so the apt codename stays the release branch exactly as it was on GitLab. Workflow -------- sign-and-publish-deb downloads both architecture artifacts and runs a single deb-s3 upload. GitLab fanned this out per architecture, so two jobs rewrote the same apt index concurrently; collecting both first removes that race. The install script is split in two. generate-install-script runs inv release.generate-install -t inside the build container, because the task collection imports python-gitlab and the GitHub and Datadog API helpers and only loads in the conda environment, and asserts the rendered script contains no None.s3.amazonaws.com from an unset bucket variable. publish-install-script then uploads it, so the container job never holds a credential. Both publishing jobs are gated on push and bound to the agent-pre-release environment, which carries the deployment branch rule, the signing secrets and the AWS role from STAC-25545. Pull requests cannot reach them. Validated: shellcheck -x clean on all three scripts; gpg_signing_setup exercised against a generated throwaway key covering explicit override, PATH auto-detection and the missing-binary failure path; actionlint clean apart from the pre-existing self-hosted runner-label notices; zizmor reports no findings. Blocked until the agent-pre-release environment, its four SIGNING_* secrets and AGENT_PRERELEASE_ROLE_ARN exist. Refs STAC-25546, STAC-25545 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion Addresses review feedback on PR #455. [P1] The signing job installed the publisher with `gem install deb-s3`, resolving the latest code at run time into a job that then executes it with the package signing key and the pre-release AWS credentials in scope. GitLab never did this: `sign_deb` ran in a pinned image with deb-s3 already baked in, so the runtime resolve was a regression introduced by the port. deb-s3 and its full runtime dependency tree are now pinned in `.github/deb-s3-gems.sha256` and installed by `.github/scripts/install-deb-s3.sh`, which fetches each gem at its exact version and verifies it against the SHA256 RubyGems publishes for that release before anything is installed or executed. The manifest covers nine gems. base64, bigdecimal and logger are deliberately excluded: aws-sdk-core requires them at ">= 0" and they are Ruby default gems supplied by the distribution's own ruby package, so pinning them would force a native build for no supply-chain gain. The script also links the canonical executable when RubyGems installs a versioned binstub, since publish_package.sh invokes deb-s3 by bare name, and ends with a `deb-s3 help` smoke check that activates the whole pinned set so a missing or incompatible gem fails at install time rather than midway through publishing. [P2] publish-install-script depended only on generate-install-script, so it could overwrite the public install.sh even when the build, signing or apt upload had failed. GitLab's pre_release_deb required sign_deb. It now requires both generate-install-script and sign-and-publish-deb, restoring that release boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolving the rebase conflict against the rewritten base took this branch's whole copy of build-deb.yml, which predated the exception work on STAC-25457. That silently reverted two lines, so the gate ran here with no exceptions loaded and reported all 24 findings as unmanaged. Restores the quay LOCAL_IMAGE name the evaluator matches exceptions on, and the exceptions-path input. The workflow diff against the base branch is now additions only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
d7e06f4 to
ca28691
Compare
STAC-25500 Port the DEB signing and pre-release publishing lane to GitHub Actions
STAC-25457 Publish and sign the agent and cluster-agent images from GitHub Actions
|
Do not merge yet — the DEB signing credentials are only half-provisioned. pulumi-infra #283 merged and its apply partially succeeded:
Safe to merge once the permission is granted and the pulumi apply is re-run green. |
|
Hold cleared — credentials are provisioned. `agent-pre-release` environment on this repo now has all four `SIGNING_*` secrets, with a protected-branches-only deployment policy, and `AGENT_PRERELEASE_ROLE_ARN` is set. pulumi-infra #283 is merged and applied (run 31400655652). Remaining blocker on this PR is review only. |
Branch protection required 'Build agent binary (branded / StackState)' and the cluster-agent equivalent, but the build lanes are an arch matrix, so the emitted contexts carry a ', amd64' / ', arm64' suffix and those two required contexts could never report. Every pull request to stackstate-7.78.2 was BLOCKED. Enumerating the suffixed names would work until the next matrix or rename change breaks it again, silently. Instead adopt the aggregate already used by stackstate-agent-integrations: one job per workflow that evaluates its needs and reports a single stable context. The aggregate fails on a skipped upstream job rather than passing. Fork pull requests skip every build and test job, so treating skipped as success would let a fork merge on a pipeline that ran nothing. mod-tidy is the one documented exception, skipped when the cache image already covers the module graph. Publish, signing and manifest jobs are excluded: they are push-only and can never report on a pull request. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Unblocked — this now needs only a review. pulumi-infra#286 merged and applied, so the release-branch protection reads: All three report |
GitLab pipeline schedule 20682 runs the agent pipeline on the release branch every weekday evening. Archiving the GitLab project without an equivalent here would end nightly builds silently, so this adds one. Those scheduled runs are now the only pipelines still executing over there. That schedule is 22:00 in Europe/Amsterdam. GitHub cron is UTC with no timezone support, so this uses 20:00 UTC to match during CEST and accepts an hour's drift in winter, a few minutes off the hour because scheduled runs are delayed at popular times. The schedule covers the build, DEB and image workflows but not the unit suites. Those are two ~2h jobs on self-hosted xlarge runners and they already run on the pull request and again on the push, so a third nightly run buys no signal. Publication had to be widened as well. Every publishing job tested for a push event, and a scheduled run is not one, so the nightly would have built the packages and images and then discarded them. The Cerberus conditions carried the same test, which would have left a nightly failure reported to nobody. Image tags are the commit SHA, so a nightly on an unchanged branch republishes an identical tag. The point is drift in the base images and fetched dependencies, not fresh artifacts. Keying concurrency on the event name stops a push from cancelling a nightly that is already running. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2a3ce7e to
05d578f
Compare
The GitHub workflows cover the pipeline, so the GitLab definitions are dead weight. Only the four StackState-owned files go: the parent pipeline, the agent child pipeline, the manifest-merge pipeline, and the runner-image pipeline whose trigger has been commented out for a long time. Deliberately left in place: - `.gitlab/`, 129 upstream files that STS pipelines never included but that around fifteen upstream `tasks/*.py` modules read. Deleting them would break upstream tooling and guarantee conflicts on the next merge. - `.gitlab-scripts/setup_artifact_registry.sh`, which the GitHub DEB build, `tasks/omnibus.py` and an omnibus software definition all call. The name is now misleading but the file is live. - `.ci-builders/`, no longer built by anything but not pipeline definitions. `tasks/buildimages.py` and two Renovate regex managers read `.gitlab-ci.yml` for `CI_IMAGE_*` variables. The STS file never defined any, so both were already no-ops and are left untouched rather than diverging further from upstream. Documentation is repointed at the workflows. Two claims went stale rather than moving: the `[cluster-agent]` commit-message optimisation, which neither pipeline ever implemented, and `AGENT_HELM_CHART_VERSION`, an override the GitHub lane does not carry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
What
Ports the
stackstate-agentGitLab pipeline to GitHub Actions, for amd64 and arm64.inv check-mod-tidy, branded + unbranded unit suitesEach top-level workflow ends in a
CI success (...)aggregate, and those three are the required checks on the base branch. A matrix job cannot be required directly by name, so the aggregate is what branch protection can pin.Nightly
GitLab schedule 20682 builds the release branch every weekday evening (22:00 Europe/Amsterdam; GitHub cron is UTC-only, so the trigger uses 20:00 UTC and drifts an hour in winter). The build and DEB workflows carry an equivalent
schedule:trigger so archiving GitLab does not end nightly builds silently. The unit suites are excluded — two ~2h jobs that already run on the PR and on the push.Publication is widened to
schedulealongsidepush, otherwise the nightly would build and discard everything, and a nightly failure would reach nobody. Image tags are the commit SHA, so an unchanged branch republishes an identical tag; the signal is drift in base images and dependencies, not fresh artifacts.GitLab pipelines are removed
The four StackState-owned GitLab pipeline files are deleted, since these workflows replace them. Scheduled runs were the only pipelines GitLab had executed since 27 July, so the nightly above is what was keeping it alive.
Left in place deliberately:
.gitlab/(129 upstream files that ~15 upstreamtasks/*.pymodules read),.gitlab-scripts/setup_artifact_registry.sh(called by the DEB build here,tasks/omnibus.pyand an omnibus software definition), and.ci-builders/. Docs are repointed at the workflows.Publication is push-only
Every publishing job is gated on
github.event_name == 'push'. A PR builds, smoke-tests and scans the images and the DEB, then throws them away. Jobs needing registry credentials skip on fork PRs.The 256 go.mod / go.sum files
One commit, STAC-25466.
inv check-mod-tidyis now a gate and the workspace manifests did not survive it. No version was picked by hand; this is whatgo mod tidyproduces.Validation
All three required aggregates green on
3e2a23fe, including the first full arm64 run. Branch is up to date with the base.Jira: https://stackstate.atlassian.net/browse/STAC-25142
Sub-tasks merged into this branch: STAC-24889, 25429, 25457, 25459, 25461, 25466, 25494, 25498, 25500, 25536, 25575, 25583