rules_go: make instrumented artifacts deterministic

.github/workflows/shared-validation.yml

@@ -256,15 +256,18 @@ jobs:
         run: python3 tools/dev/check_release_archive_contents.py
 
   rules-go-fork-profile-shard:
-    name: rules-go-fork-profile-shard (${{ matrix.upstream }})
+    name: rules-go-fork-profile-shard (${{ matrix.upstream }}, ${{ matrix.runner }})
     needs: rules-go-fork-drift-matrix
     if: ${{ !inputs.docs_only && inputs.run_rules_go_fork_drift }}
     strategy:
       fail-fast: false
       max-parallel: 4
       matrix:
         upstream: ${{ fromJSON(needs.rules-go-fork-drift-matrix.outputs.upstreams) }}
-    runs-on: ubuntu-latest
+        runner:
+          - ubuntu-latest
+          - ubuntu-24.04-arm
+    runs-on: ${{ matrix.runner }}
     steps:
       - name: Checkout
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

AGENTS.md

@@ -74,7 +74,10 @@ The sync rule creates `@test_optimization_data//` containing:
 - **Sanitization**: module names are converted into Bazel-safe labels using `sanitize_label_fragment()` (lowercase, `[a-z0-9_]` only, deterministic suffixes).
 - **Go importpath inference**: `topt_go_payloads_selector` mirrors rules_go importpath logic (explicit `importpath` > `embed` provider > fallback `<module>/<package>`).
 - **Vendored rules_go forks for root workflows**: the repository root pins `rules_go` as a dev-only dependency and redirects it to `third_party/rgo/v0_60_0/base` with `local_path_override(...)`; consumer-facing core usage remains rules_go-free.
-- **Cross-platform uploader**: Unix uses Bash/curl; Windows uses PowerShell and .NET `HttpClient`.
+- **Cross-platform uploader**: Python 3.10+ provides one implementation on
+  Linux, macOS, and Windows. Small Bash and PowerShell launchers locate Python
+  and Bazel runfiles. The former Bash/curl and PowerShell/.NET uploaders remain
+  available only through the explicit `use_python_uploader = False` rollback.
 
 ## Build, Test, and Development Commands
 - Canonical validation command matrix lives in `CONTRIBUTING.md`; keep this

CHANGELOG.md

@@ -70,6 +70,13 @@ versioning.
 - Config-disabled Go analysis now resolves stable empty Orchestrion repository
   targets before host-Go discovery or source fetching, so ordinary targets do
   not require Go to be installed merely because the integration is declared.
+- Generated-profile verification now checks deterministic output bytes for
+  `GoStdlib`, `GoSyntheticTestmainHelpers`, synthetic `GoCompilePkg`, and
+  `GoLink` actions across every maintained `rules_go` version. The builder
+  normalizes ephemeral execroot and Go work paths, CGO random seeds, source
+  trim paths, and copied helper build IDs while preserving relative CGO input
+  resolution. Ordinary Go actions cannot consume the instrumented stdlib
+  cache.
 
 ## [1.2.0] - 2026-06-03
 

CONTRIBUTING.md

@@ -70,6 +70,7 @@ This product includes software developed at Datadog
   - `shellcheck` (shell lint lane)
   - `buildifier` (Starlark formatting lane)
   - `jq` (integration harness payload/CODEOWNERS checks)
+  - `zstd` (rules_go compact execution-log verification)
 - Optional pre-commit setup:
   - `python3 -m pip install pre-commit && pre-commit install`
 - Optional Python syntax smoke check when editing tooling:
@@ -94,7 +95,7 @@ This product includes software developed at Datadog
     `--config=test-optimization` as the only user-facing switch. The shared
     config entry is
     `common:test-optimization --repo_env=DD_TEST_OPTIMIZATION_ENABLED=1`.
-  - Go additionally sets the existing `rules_go` Orchestrion `enabled=true`
+  - Go also sets the existing `rules_go` Orchestrion `enabled=true`
     build setting. Python-only consumers must not declare that Go-only label.
   - Omitting the config is the documented complete opt-out for Go and Python:
     metadata repositories use disabled stubs when `enabled_by_env = True`, Go
@@ -228,7 +229,10 @@ This product includes software developed at Datadog
 - Utility/lint lanes:
   - module version alignment check (`tools/dev/check_module_versions.py`)
   - `.bazelversion` parity check (`tools/dev/check_bazelversion_sync.py`)
-  - global fork drift checks plus one consumer patch-profile shard per supported `rules_go` upstream
+  - global fork drift checks plus x86-64 and ARM64 consumer patch-profile shards
+    for every supported `rules_go` upstream; each profile shard rebuilds the
+    cache-critical Test Optimization actions twice from isolated output roots
+    and requires identical action keys and output bytes
   - shell scripts, PowerShell, Buildifier, gofmt, schema sync checks, fixture JSON checks, and Python tooling tests
 - Workflow dependency pinning:
   - Keep GitHub Actions pinned by commit SHA and preserve the `# vX.Y.Z` comment.

README.md

@@ -556,6 +556,12 @@ Disabled or mismatched repositories fail analysis. This descriptor is an
 additive option; generated single-, multi-, and manifest-sync exports keep
 their existing behavior.
 
+The `.topt` clone preserves the source test's execution policy. A source tagged
+`no-remote-exec` therefore still runs its `TestRunner` locally, but that policy
+does not leak into deterministic build actions such as stdlib preparation,
+synthetic testmain helpers, compilation, or linking. Those actions remain
+eligible for normal Bazel caching and remote execution.
+
 Large consumers can warm the matching instrumented standard library in a
 dedicated cache-writing invocation:
 
@@ -572,8 +578,9 @@ bazel build //tools/test_optimization:go_stdlib_warmup \
 ```
 
 The warmup target applies the same target-scoped Go configuration transitions,
-enables Orchestrion in `test_optimization` mode, and materializes both the
-instrumented stdlib and its Go build cache. The consumer
+enables Orchestrion in `test_optimization` mode, and materializes the
+instrumented stdlib plus its deterministic declared archive cache. Ordinary Go
+actions neither publish nor consume that instrumented cache. The consumer
 remains responsible for restricting remote-cache writes to a trusted CI
 invocation.
 Without the named config, the target is a no-op so ordinary broad builds can
@@ -1472,14 +1479,26 @@ The uploader is a normal Bazel rule (not a test) that runs via `bazel run` after
 4. Then run the uploader via `bazel run`
 5. The uploader discovers all `test.outputs/` directories, waits for quiescence, uploads, and deletes files
 
-Before transport, the uploader enriches each test payload and splits any body
-larger than 4,500,000 bytes along its `events` array. Parts preserve event order
-and the original top-level envelope, are compressed independently when gzip is
-enabled, and use independent retries. The original payload is deleted only
-after every part uploads successfully. A terminal upload failure reports the
-HTTP status, a bounded response body, and uncompressed/compressed/transmitted
-sizes without requiring debug mode. A single event above the 5,000,000-byte
-intake limit is rejected because it cannot be split safely.
+The default Python runtime prepares CODEOWNERS, contexts, schemas, freshness,
+and telemetry plans once. It then gives each source file to one of eight
+independent workers. That worker owns enrichment, optional validation,
+preventive splitting, upload retries, and cleanup for the file, whether it
+contains test, coverage, or telemetry data. Starting several uploader processes
+is neither necessary nor supported.
+
+After enrichment, a worker compacts a test payload and splits it along its
+`events` array before any HTTP request when the encoded body would exceed
+`4_718_592` bytes (4.5 MiB). Chunks preserve event order and the original
+top-level envelope, and each chunk has its own retry lifecycle. HTTP `413` is a
+terminal contract error: the uploader has already applied the preventive limit,
+so retrying or splitting adaptively would hide a defect. A single event that
+cannot fit within the limit is rejected locally without spending a request.
+
+At the end of the run, the uploader prints deterministic totals for files,
+payload types, chunks, requests, retries, and cleanup. Use `--debug` for verbose,
+redacted diagnostics. Use `--dry-run --validate-enrichment` to exercise
+discovery, enrichment, validation, splitting, and request preparation without
+contacting the backend or deleting source files.
 
 Telemetry-specific notes:
 - Telemetry files must contain one raw top-level tracer telemetry request body per file.
@@ -1522,12 +1541,12 @@ $env:DD_SITE = "datadoghq.com"
   //...
 ```
 
-**IMPORTANT**: Always preserve the test exit code. When upload is enabled, the
+Always preserve the test exit code. When upload is enabled, the
 wrapper runs the doctor, then validates enrichment and uploads every available
 fresh valid payload in one uploader pass even if tests or doctor failed. The
 earliest test, doctor, or uploader failure remains the job result.
 
-### Important runtime requirements
+### Runtime requirements
 
 1. Use `bazel run` (not `bazel test`) for uploader execution.
 2. Use a single uploader target per workspace (do not run concurrent uploaders).
@@ -1720,13 +1739,12 @@ an `outputs.zip` archive to the requested `--output` path before
 the downloader contract; it does not ship credentials or a Datadog-internal CAS
 client.
 
-Artifact staging requires Python at uploader runtime. Bash resolves
-`DD_TEST_OPTIMIZATION_PYTHON`, then `PYTHON`, then `python3`, then `python`;
-PowerShell uses the same discovery order. Existing local-only uploader flows
-remain usable without Python except for support bundle generation and the
-pre-existing optional schema and telemetry helpers. Bash BEP freshness parsing
-still requires `jq` whenever BEP freshness validation is enabled in the Bash
-uploader path.
+The default uploader and artifact staging require Python 3.10 or newer. The
+small Bash and PowerShell launchers resolve `DD_TEST_OPTIMIZATION_PYTHON`, then
+`PYTHON`, then `python3`, then `python`. Only the explicit
+`use_python_uploader = False` rollback path can upload local files without
+Python. That legacy Bash path still requires `jq` for BEP freshness parsing,
+schema validation, enrichment, and oversized-payload splitting.
 
 ### Enrichment validation
 

docs/Configuration_Reference.md

@@ -211,7 +211,7 @@ Extension tag: `test_optimization_sync.test_optimization_sync(...)`
 | `test_management` | bool | `True` | Local switch for Test Management request. When `False`, request is skipped, a minimal stub is written, and settings are mutated to `test_management.enabled=false` |
 | `flaky_tests` | bool | `True` | Local switch for Flaky Tests request. When `False`, request is skipped, a minimal stub is written, and settings are mutated to `flaky_test_retries_enabled=false` |
 | `enabled` | bool | `True` | Hard enablement switch. When `False`, the repository emits the deterministic disabled interface and skips local Git discovery and metadata HTTP requests |
-| `enabled_by_env` | bool | `False` | When `True`, additionally gate enablement on `DD_TEST_OPTIMIZATION_ENABLED` (`1`, `true`, `yes`, or `on`, case-insensitive). Unset and false values emit the disabled interface. The public Go extension and Go WORKSPACE helper override this low-level default to `True` |
+| `enabled_by_env` | bool | `False` | When `True`, also gate enablement on `DD_TEST_OPTIMIZATION_ENABLED` (`1`, `true`, `yes`, or `on`, case-insensitive). Unset and false values emit the disabled interface. The public Go extension and Go WORKSPACE helper override this low-level default to `True` |
 | `require_git_metadata` | bool | `False` | Strict local/CI validation for settings-request Git metadata. When `True`, sync fails before HTTP if repository URL, branch or tag, and commit SHA cannot be resolved |
 | `debug` | bool | `False` | Enables verbose repository-rule logging |
 

docs/Initial_documentation.md

@@ -11,9 +11,9 @@ This product includes software developed at Datadog
 This document explains the current implementation architecture in this
 repository. For installation and day-to-day usage, start with `README.md`.
 
-> Last reviewed: 2026-07-27
+> Last reviewed: 2026-09-11
 
-## Approach Overview
+## Approach overview
 
 The integration uses a Bazel module extension and repository rule to
 materialize the Test Optimization repository during module/repo resolution, a
@@ -37,7 +37,7 @@ The steps are:
    - `@<repo>//:test_optimization_files` (core bundle, includes `cache/http/settings.json`)
    - `@<repo>//:test_optimization_context` (`context.json` plus
      `telemetry_facts.json`)
-   When an enabled response contains module data, the repository additionally
+   When an enabled response contains module data, the repository also
    exposes `@<repo>//:module_<sanitized>` bundles with
    `cache/http/settings.json` plus that module's known-tests, test-management,
    and flaky-tests files.
@@ -59,7 +59,16 @@ The steps are:
    targets, or payload instrumentation.
 
 3. **Payload validation and reporting**:
-   A single workspace-level doctor runs via `bazel run` after tests complete and validates local JSON payloads, Bazel target metadata, Git metadata, and invalid Go payload-selection states. A single workspace-level uploader then discovers all `test.outputs/` directories in `bazel-testlogs/`, waits for payloads to quiesce, enriches them with `context.json`, and uploads via agentless (`DD_API_KEY`, `DD_SITE`) or EVP proxy (`DD_TEST_OPTIMIZATION_AGENT_URL`).
+   A single workspace-level doctor runs via `bazel run` after tests complete and
+   validates local JSON payloads, Bazel target metadata, Git metadata, and
+   invalid payload-selection states. A single workspace-level uploader then
+   discovers local or BEP-staged `test.outputs/` directories and waits for
+   payloads to quiesce. Its coordinator prepares CODEOWNERS, contexts, schemas,
+   freshness, and telemetry plans once before starting a bounded pool of eight
+   workers by default. Each worker owns one source file through enrichment,
+   optional validation, preventive 4.5 MiB splitting for test payloads, upload
+   retries, and cleanup. A worker can process test, coverage, or telemetry data;
+   workers do not synchronize with one another.
    In mixed-runtime workspaces, the uploader can bundle multiple `context.json`
    files and select the matching one per payload using sibling
    `bazel_target_metadata.json` repo metadata instead of reusing one global
@@ -69,10 +78,26 @@ The steps are:
    can use root labels; large monorepos should use a lightweight package such
    as `//tools/test_optimization`.
    Usage: run `bazel test`, then the doctor target, then one uploader pass with
-   `--validate-enrichment`; add `--dry-run` only when upload is disabled.
-   Preserve the earliest failure while still processing every available fresh
+   `--validate-enrichment`; add `--dry-run` only when upload is disabled. The
+   uploader prints final file, type, split, request, retry, and cleanup totals.
+   `--debug` adds verbose redacted diagnostics. Preserve the earliest test,
+   doctor, or uploader failure while still processing every available fresh
    valid payload.
 
+   ```mermaid
+   flowchart LR
+     D[Discover fresh source files] --> P[Prepare shared context and CODEOWNERS]
+     P --> Q[Bounded file queue]
+     Q --> W1[Worker 1: enrich, validate, split, send, clean]
+     Q --> WN[Worker N: enrich, validate, split, send, clean]
+     W1 --> S[Aggregate final statistics]
+     WN --> S
+   ```
+
+   Each worker owns its file from dequeue to final result. There is no serial
+   split or upload stage after enrichment, and workers do not exchange payload
+   state.
+
 4. **Language macros (optional)**:
    Thin wrappers (for Go/Python/Java/NodeJS/.NET/Ruby) set up the right runfiles/env so test code can read the synced files and write payloads to `TEST_UNDECLARED_OUTPUTS_DIR`.
    - Core module (`datadog-rules-test-optimization`) stays runtime-agnostic.
@@ -281,16 +306,24 @@ generated per-service sync repository keys used by payload metadata.
 
 ## Runtime uploads and hermetic tests
 
-Tests remain hermetic with network blocked. They write payloads to Bazel's built-in `TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage}`, which is automatically collected to `bazel-testlogs/<target>/test.outputs/`. A single workspace-level doctor validates those local outputs before upload. A single workspace-level uploader (via `bazel run`) then:
+Tests remain hermetic with network blocked. They write payloads to Bazel's
+built-in `TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage,telemetry}`,
+which Bazel collects under `bazel-testlogs/<target>/test.outputs/`. A single
+workspace-level doctor validates those outputs before upload. One uploader
+process then:
 
 - Discovers all `test.outputs/` directories in `bazel-testlogs/`,
 - Waits for filesystem quiescence,
-- Enriches test payloads with `context.json` when present,
+- Prepares CODEOWNERS, contexts, schemas, freshness, and telemetry state once,
+- Assigns each source file to one of eight workers by default,
+- Lets that worker enrich, validate, split when required, upload with retries,
+  and clean up the file,
 - When multiple bundled contexts are present, matches them per payload using
   `bazel.test_optimization.repo_name` from sibling `bazel_target_metadata.json`,
 - Can dry-run the enrichment path without uploading or deleting files,
 - Uploads to Datadog using either `DD_API_KEY`/`DD_SITE` (agentless) or `DD_TEST_OPTIMIZATION_AGENT_URL` (EVP proxy),
-- Deletes successfully uploaded payloads.
+- Deletes successfully uploaded payloads,
+- Prints aggregate statistics after all workers finish.
 
 No secrets are written to disk; all credentials are passed via environment variables.
 
@@ -335,21 +368,24 @@ flowchart TD
   %% Test execution: hermetic, offline
   subgraph T[Test Execution (Hermetic)]
     T1[Tests (instrumented)]
-    P1[bazel-testlogs/.../test.outputs/\n  payloads/tests/*.json\n  payloads/coverage/*.json]
+    P1[bazel-testlogs/.../test.outputs/\n  payloads/tests/*.json\n  payloads/coverage/*.json\n  payloads/telemetry/*.json]
     T1 -->|read runfiles| A3
     T1 -->|write to TEST_UNDECLARED_OUTPUTS_DIR| P1
   end
 
   %% Validate/upload steps: bazel run after tests
   subgraph U[Validate and upload via bazel run]
     U0[Doctor rule]
-    U1[Uploader rule]
+    U1[Uploader coordinator]
+    UW[Bounded file workers\n enrich, validate, split, send, clean]
+    US[Final statistics]
     U0 -->|validate| P1
     U0 -->|validate context| A3
     U0 --> U1
-    U1 -->|enrich with| A3
-    U1 -->|upload tests| G1{Agentless?\n DD_API_KEY}
-    U1 -->|upload coverage| G1
+    U1 -->|prepare shared context| A3
+    U1 --> UW
+    UW -->|upload tests, coverage, telemetry| G1{Agentless?\n DD_API_KEY}
+    UW --> US
     G1 -- Yes --> I1[(citestcycle/citestcov\n intake on <DD_SITE>)]
     G1 -- No  --> I2[(EVP proxy\n ${DD_TEST_OPTIMIZATION_AGENT_URL})]
   end
@@ -387,13 +423,15 @@ Build Graph
 
 Test Execution (Hermetic)
   [tests (instrumented)] --read runfiles--> synced JSONs
-                         --write payloads--> TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage} (-> bazel-testlogs/.../test.outputs/)
+                         --write payloads--> TEST_UNDECLARED_OUTPUTS_DIR/payloads/{tests,coverage,telemetry} (-> bazel-testlogs/.../test.outputs/)
 
 Validate and upload (via bazel run)
   [doctor rule] --validate--> payload JSON, bazel_target_metadata.json, context.json
-  [uploader rule] --dry-run enrichment or upload--> context.json
-       |-- agentless (DD_API_KEY, DD_SITE) --> citestcycle/citestcov intake
+  [uploader coordinator] --prepare once--> context/CODEOWNERS/schema/freshness
+       |-- bounded file workers: enrich -> validate -> split -> send -> clean
+       |-- agentless (DD_API_KEY, DD_SITE) --> citestcycle/citestcov/telemetry intake
        |-- EVP proxy (DD_TEST_OPTIMIZATION_AGENT_URL) -> /evp_proxy/... endpoints
+       `-- aggregate final statistics
 
 Optional: Multi-service aggregator
   @test_optimization_data//:test_optimization_files_<service>