feat(ci): run integration suites under an ES→OS migration phase (#36320)

[fabrizzio-dotCMS]

Proposed Changes

Makes it possible to run the full integration battery under a chosen ES→OS migration phase — on a single phase or swept across all three — locally, per-PR, on-demand, and automatically on a schedule. Contributes to #36320 ("Make it possible to run MainSuite Tests in all 3 phases").

The opensearch-phase-suite Maven profile already existed (two clusters ES @9207 + OS 3.x @9201, sets DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE, inherits the base suite includes). This PR wires that capability into every layer of the tooling. It does not change any product code, and default (phase 0) behavior is untouched everywhere.

Four ways to run a phased battery

Path	Trigger	Scope
Local	just test-integration-phase <N> / test-integration-phase-per-suite <N>	single phase
Per-PR (opt-in)	PR label OpenSearch:Phase-<N>	single phase
On-demand	Nightly workflow_dispatch → opensearch_phase dropdown (none/1/2/3/1,2/1,3/2,3/all)	single phase, subset, or sweep
Automatic	New weekly scheduled workflow (Sun 06:00 UTC)	sweep (all 3)

Local (justfile)

• test-integration-phase <N> [TEST] — full battery in a single JVM (one docker bring-up), or a narrowed run with a test arg.
• test-integration-phase-per-suite <N> — iterates the MainSuite/Junit5 suites one at a time (forkCount=1) under phase <N>, giving attributable per-suite pass/fail plus a summary (non-zero exit if any suite fails). This is the local triage tool for de-ES-ifying the ITs. (renamed from test-integration-phase-all — the sweep is per-suite, not per-phase.)

CI — shared engine (cicd_comp_test-phase.yml)

• New opensearch_phase input now accepts a single phase (2), a comma-list subset (1,2, 1,3, 2,3), the full list (1,2,3), or the alias all (= 1,2,3). The parser expands all, splits on commas, keeps values matching ^[1-3]$, dedupes and sorts.
• The matrix generator fans each integration MainSuite/Junit5 entry out once per requested phase, appending -Dopensearch.phase=<N> and suffixing name / stage_name per phase so job labels and the resulting build-reports-<stage_name> artifacts stay unique. The OpenSearch Upgrade Suite is never phased (it already drives OS via -Dopensearch.upgrade.test; combining the two profiles would collide their endpoint config).
• fail-fast is conditional — derived from "phase off" ('' / none / 0), not an exact-match list — so ANY active run (single phase, a subset like 1,2, or all) disables fast-fail and every suite/phase runs to completion with per-phase attribution. (Fixes a latent mismatch where a comma-list subset would otherwise have left fast-fail on.)
• No-op when off: empty / none / 0 yields an empty phase list → the matrix is byte-for-byte identical to today. Verified against the other callers that don't opt in (cicd_2-merge-queue, cicd_3-trunk, cicd_5-lts).

CI — per-PR (cicd_1-pr.yml)

• A PR carrying an OpenSearch:Phase-<N> label runs the integration battery under that phase (mirrors the existing PR: docker image label pattern). No label → phase 0 (current behavior). (The label family was aligned with the repo's OpenSearch area label — see Repo labels below.)

CI — on-demand (cicd_4-nightly.yml)

• workflow_dispatch opensearch_phase dropdown: default none (clearer than 0), options none/1/2/3/1,2/1,3/2,3/all.
• Any active value (single phase, subset, or all) skips the deployment job — a build exercised under a non-default migration phase is never promoted. The skip guard is derived from "phase off" so subsets are covered. The scheduled nightly (03:18) is unaffected: it runs with none → phase 0 + deploy as today.

CI — automatic (cicd_scheduled_opensearch-phase-sweep.yml, new)

• Weekly (Sun 06:00 UTC, off-peak and clear of the daily nightly): sweeps the integration battery across all 3 phases automatically, so migration-flow regressions are caught without any developer opting a PR in.
• Test-only: initialize → build → test → finalize → report. No deploy, no CLI build. Only the integration test type is enabled (other suites don't vary by phase). workflow_dispatch also allows picking a single phase or a subset (1,2/1,3/2,3).
• Serialized via a concurrency group (cancel-in-progress: false): only one sweep runs at a time — a manual dispatch fired while the weekly run is going is queued rather than run in parallel (a sweep is expensive and its result matters).
• Scheduled triggers only fire from the default branch, so it stays dormant until merged to main.

Repo labels (created out-of-band)

• Renamed the area label OpenSearch-Migration → OpenSearch (assignments preserved; nothing references it in code).
• Created the CI-trigger family OpenSearch:Phase-1 / -2 / -3 used by the per-PR trigger.

Checklist

• YAML validated; matrix-generation logic simulated against the real test-matrix.yml: none/''/0 → 7 entries / 0 phased (unchanged); 2 → 6 phased + upgrade untouched; 1,2 → 12 phased; all → 18 phased + upgrade, all 19 stage_names unique (no artifact collision). fail-fast and deploy-skip guards verified across all values.
• Shared-engine change confirmed no-op for callers that don't pass opensearch_phase (merge-queue, trunk, lts).
• Smoke-verified locally: just test-integration-phase 1 ContentletIndexAPIImplTest brings up both clusters, bootstraps OS under PHASE_1_DUAL_WRITE_ES_READS, and runs the suite.
• No product code changed (CI/tooling only).

Notes for reviewers

• The OpenSearch:Phase-<N> labels already exist in the repo (created alongside this PR) so the per-PR trigger is selectable.
• Impact of the new scheduled workflow on other jobs is minimal: it's independent and test-only (no shared state, no deploy, per-run artifacts); the only shared surface is cicd_comp_test-phase.yml, whose change is a verified no-op when the phase is off. The material cost is ~19 long integration jobs once a week, off-peak Sunday (no cron collision with the pentest/rolling-tags/nightly schedules).

Related: #36320, PR #36268 / #36266 (added the opensearch-phase-suite profile).

This PR fixes: #36320

[claude]

Claude finished @fabrizzio-dotCMS's task in 6m 19s —— View job

---

Code Review — phased integration suite (CI/tooling only)

• Read the full diff against origin/main
• Cross-checked the notify-slack action contract, test-matrix.yml, and the other callers of cicd_comp_test-phase.yml
• Reviewed matrix fan-out, fail-fast/deploy-skip guards, and the new sweep workflow

New Issues

• 🟡 Medium: .github/workflows/cicd_scheduled_opensearch-phase-sweep.yml:128-130 — The notify-scout payload is wrapped in literal double quotes. The notify-slack action (with the default json: 'false') renders the payload as "text": ${{ toJSON(inputs.payload) }} — it re-encodes the string to JSON itself. Every other caller in the repo (cicd_6-release.yml:431, cicd_comp_deployment-phase.yml:341, etc.) passes raw markdown with no surrounding quotes. Here the leading " and trailing " are part of the payload text, so toJSON escapes them and the Slack message renders with stray " characters at the start and end (":x: *OpenSearch Phase Sweep failed* … run_id>"). Drop the two " so it matches the convention:

  payload: |
    :x: *OpenSearch Phase Sweep failed* — phase(s) `${{ inputs.opensearch_phase || 'all' }}`
    *Run:* <https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}|${{ github.run_id }}>

  Fix this →

Everything else checks out. Specifically verified:

• No-op when off: rawPhase empty/none/0 → phaseList empty → matrix identical to today; the else branch pushes the unmodified testEntry. Callers that never pass opensearch_phase (merge-queue, trunk, lts) get the '' default → unchanged.
• Upgrade suite exclusion is correct: phaseable is gated on test_type === 'integration' and !maven_args.includes('opensearch.upgrade.test'); the upgrade suite's base_maven_args carries -Dopensearch.upgrade.test=true (test-matrix.yml:93), so it's never phased.
• fail-fast guard (cicd_comp_test-phase.yml:235) and deploy-skip guard (cicd_4-nightly.yml:191) are both derived from "phase off" ('' | none | 0), so comma-list subsets like 1,2 correctly disable fast-fail and skip deploy. The == '' branch also covers the scheduled-cron case where inputs is empty (defaults aren't applied on schedule), so the nightly still deploys.
• Script-injection safe: opensearch_phase reaches the github-script step via the OPENSEARCH_PHASE env var, not string interpolation.
• notify-scout firing: if: failure() with needs: [initialize, build, test] — a skipped build (artifacts reused) is not a failure, and a failing matrix job propagates through the reusable test job, so it triggers as intended.

The \ No newline at end of file on the new workflow is cosmetic — not worth blocking.
· issue-36320-phased-integration-suite

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request