Register one UK full-build graph with all-geography calibration

[juaristi22]

UK population construction and calibration currently span source-spine, national and joint-local drivers, with transformations and acceptance evidence outside the executable graph. This change registers one full build from canonical raw FRS sources through geographic replication, target compilation, calibration, optional exact-count sizing, gates, diagnostics and checked export.

Stacked on #893 at 1cebe3547543bf2917a7af7362bfbb22a05bf964; review against microcosm-us-launch-integration-20260909.

• Calibrate all applicable geographic levels together by default. --target-geographies country explicitly filters the same graph. Pool copies K, exported household count k and target scope remain independent; failed support or fit checks never narrow scope automatically.
• Remove executable frs_hmrc_retained_leaves and hmrc_spi_income stages; retain the canonical spine replacements already present in Consolidate the US survey graph build and enrichment pipeline #893. Retire independent national/joint-local scientific drivers. Legacy command names route into the new full-build/readiness interfaces, but retired flags fail explicitly: these are entry-point shims, not compatibility with old invocation scripts. Use microcosm-build-uk / tools/build_uk_full.py for the maintained build interface.
• Reuse shared execution, typed artifacts, target selection, ordered sparse problem/result codecs, gate evidence and staged file publication. Add explicit same-kind weight-update contracts and correct pandas index alignment for graph writes on sampled populations.
• Preserve UK source/model boundaries, seeds, sampling before cloning, geographic draws, identifiers and solver/selection algorithms. Include the full national register, band edges, independent source pins and national/local checks. Chronicle census cells supply household targets; regional rows remain in calibration while country totals control country-wins reconciliation.
• Restore stage, fit, gate and diagnostic evidence from verified storage. Retain source-phase manifests after downstream exceptions. Checked H5 readback precedes an immutable candidate package and unsigned certification-readiness artifact with explicit missing comparisons.

The coverage manifest also drops the historical pre-spine hmrc_cgt_gains entry, which already declared hmrc_cgt_gains_spine as its replacement. Canonical CGT coverage remains required at build, with pinned source verification, capital-gains output/rewrite checks, the conserved-mass receipt and all 76 published facts fenced from calibration. This removes validation of the old certified-input-H5 route, not canonical CGT construction.

The context mutation guard now includes immutable frame metadata, mass records and declared column order. _context_digest checks for input mutation around kernel execution; it does not directly construct node keys. The shared implementation source changes deliberately change implementation and cache identities. The H1 parity refresh also resolves inherited stale implementation pins: direct input and numerical fixture bytes remain unchanged, and old/new node identities were independently reconstructed. The PR workflow now runs for non-main base branches so stacked PRs receive CI; that trigger remains after merge.

The latest parent adds supplied-parent US export verification and moves US atomic assignment after initial cloning, with a typed geography-gate artifact consumed before financial fitting. Its shared artifact is opt-in. The UK graph retains its existing ladder and assignment method; adopting shared atomic assignment and changing UK stage order remain subsequent work.

Validation includes direct-versus-graph target/matrix/solver parity, required replay, real preflight refusal before solving/export, CLI failure/export evidence, and UK source-spine H2 parity. UK regression runs covered 2,018 distinct cases; corrected fixture cases passed fresh reruns, and the inherited licensed-reference mismatch below remains explicit. Both UK engine and UK-spine jobs passed on the preceding pushed head.

On the merged source, all 521 source-provenance audit cases, 89 shared/UK adapter checks and nine previously failing survey/default/manifest cases pass. Python 3.13 and 3.14 produce identical 438 audit contracts and 1,071 references. All six wheels were rebuilt: 475 Python files and 155 resources match source, wheel and installation, and all 54 installed versions match the lock. The engine-free wheel environment passes 125 shared/UK/export checks; canonical CLI help retains the all-geography default.

All 69 affected parent US acceptance cases pass across seven files, including export comparison, post-clone identity, financial/population graph execution, age-calibration replay and origin-budget integrity checks.

The final ordered regression passes all 101 cases: the complete 58-case PUF file followed by all 43 affected survey/budget cases from CI. All cases execute without skips, errors or failures. An observer checks every teardown and confirms zero changes to sealed producer identities or source bytes.

The reviewed repairs and merged parent are pushed at 051fb972b19d319d58277bd63306d0d0e0947ce2. GitHub CI on this exact head is running; local acceptance does not substitute for its final result.

CI repairs cover issues introduced by this stack and failures in the moving parent:

• Keep synthetic context fixtures consistent with declared column-order metadata; register the shared weight-update implementation; update only independently reconstructed current identity expectations and lock admission.
• Admit eight exact public parameter CSVs needed by the locked US engine only after origin, version, installed-record and content checks. Preserve network, child-process and unlisted-data refusals. All 127 bootstrap controls and 43 maintained seed/loader/inventory checks pass; the complete guarded Linux diagnostic passes on 051fb972, with all 65 source stamps verified against that exact commit. Its output remains candidate-only. Align the expected module list with all 44 maintained calibration modules.
• Reconstruct four omitted solver-default options in independent survey diagnostics while retaining exact comparison and numerical expectations. Repair stale fixture/API expectations and check optional ACS dependencies at their actual fixture boundary; all 422 engine cases execute and pass.
• Classify current source/model preparation separately from population operators. Bind reviewed functions, callers and finite selector definitions; preserve the raw population-operator scan, exact gate-accessor restrictions and adversarial controls. The static audit documents its limits and adds no whole-module provenance exemption.
• Restore a mocked PUF resolver through pytest cleanup instead of reloading its whole production module. The previous teardown changed 82 sealed function identities and caused later budget checks to fail. The repaired predecessor leaves zero identity changes in source and installed wheels; production integrity checks remain intact.

One local test with a licensed cached eFRS file exposes an inherited reference mismatch: the frozen extraction uses PolicyEngine-UK 2.89.0, while both parent and current locks use 2.97.0. The unchanged parent generator reproduces the difference against the pinned H5. The historical reference remains unchanged.

A native f001 smoke reached the UC capital-coherence donor-support refusal, reproduced by a direct call to the maintained helper. The available ladder has the older digest. A full native run against the corrected ladder and native certification remain outstanding; this PR does not publish or certify a production dataset. The maintained command, operation inventory and migration boundaries are documented in docs/uk-full-build-graph.md.

[vahid-ahmadi]

Review at 22a4dbff (Claude Code, high effort; worktree checkout; scoped to pr893..22a4dbff — the three #901 commits — reviewed against microcosm-us-launch-integration-20260909 as the body asks; #893 is Max's and not reviewed here. Without the engine or the pinned feed: 848 passed, 35 skipped, 0 failed across the 54 touched build suites plus all of microcosm-graph and microcosm-calibrate; ruff check clean on all 105 changed files; both census tools, the coverage manifest and ci_test_groups --verify current. Draft, so a comment rather than a review.)

The structure is right and the two behavioural claims I could test hold: all geographies is the default with --target-geographies country filtering the same graph and K and k independent, and the pandas alignment fix is real — the new non-contiguous-label tests fail on #893's population.py and pass here. The stage removal is clean: sources.yaml and source_stages.json drop exactly the two stages in lockstep, every remaining stage is in _STAGE_CELLS and _STAGE_CONSUMES, and the only surviving references are negative assertions and historical build records. Nothing from main is lost: main is an ancestor of #893, the #889, #891/#892 and #877 code is untouched, and #900's Chronicle census bindings are ported into the graph.

Blocking

1. The US engine lanes break, and both causes are in this diff. us-p fails now; us-qs, us-am and engine-shared will follow.

• weight_update.py is a new module in microcosm.graph, but us_runtime/graph_implementation_inventory.json package_rosters["microcosm.graph"] was not updated (19 entries, no weight_update). _validate_package_roster raises, and survey_population_preparation.py swallows it into PREPARATION_ISSUANCE_REFUSED with the cause masked, which is every ERROR in the us-p log (the two puf55 survey suites). The same tests pass on Consolidate the US survey graph build and enrichment pipeline #893. Worth also checking the per-file contracts for the five modified graph modules, not only the roster.
• KernelContext.__post_init__ (kernel.py:357-369) now requires frame_column_order[entity] to equal the table's column set, so any dataclasses.replace(context, tables=…) that changes columns raises TypeError. That is test_us_puf_detail_transfer.py:1023 in CI, and the same pattern sits in test_us_current_survey_puf_host.py:302-316 and test_us_survey_age_artifact.py:83,98 in the lanes that have not run yet. The production site in graph_acs_housing_universe.py only adds entities, so it survives.

2. The uv.lock digest is not re-pinned. This diff changes uv.lock (adding microcosm-data to the uk extra); APPROVED_UV_LOCK_SHA256 in worker_identity.py is still #893's 751d5ef5… while the branch lock is 14bf10bf…. spec_engine_coverage --check fails on that, and the two spec-engine coverage suites error, so the shared-spec and rest lanes cannot go green. One line, but you did this same re-pin on #900 (0d978f20), so it should be in the branch.

Should-fix

3. The merge commit carries the Chronicle work. 22a4dbff is described as a merge preserving bindings, but it adds a 198-line new test file (test_uk_full_graph_admission.py), docs/uk-chronicle-feed-repin.md, and roughly a hundred net lines across graph_targets.py, full_targets.py, full_problem.py and graph_build.py — the codec registration, uk_census_household_uprating, the dispersion key, the identity.targets.chronicle receipts. That is the port of #900 into the graph and the most review-worthy part of the PR; it should be its own commit, not conflict resolution.

4. "Temporary redirects" overstates what the shims do. calibrate_uk_national_dataset.py and build_uk_rowwise_dataset.py SystemExit on the legacy flags (--staging-h5, --diagnostics-json, --allow-unpinned-feed, --crosswalk, …); build_uk_rowwise_candidate.py and certify_uk_release_cut.py forward to the new argument surface. docs/uk-full-build-graph.md:85 says this honestly; the body does not. Anyone with a script against the old CLIs gets a hard stop, which is a fine decision if stated.

5. The kernel parity re-pins absorb a stale base pin silently. kernel.py and executor.py change, so re-pinning calibrate, fit.qrf and simulate is expected — but test_h1_kernel_parity already fails on #893 with the old pin, so the new pins also close a drift you inherited, and neither the commits nor the body say what moved. The _context_digest change (folding frame metadata, the mass log and column order into node keys) is a deliberate identity change and belongs in the body.

Questions

• release_input_coverage_manifest.json drops the whole hmrc_cgt_gains family and the builder loses _cgt_family_coverage_contract (−246 lines). --check passes because it is self-consistent, but retiring the CGT coverage family three days after Align UK CGT calibration with FY2024-25 individual observations #889 is not mentioned anywhere. Intended?
• _numerical_dependency_versions() (graph.py:791-812) bakes the installed engine version, or "not-installed", into every spine node's params, and uk_spine.json now carries ["policyengine-uk","2.97.0"]. Node keys become environment-dependent by design; is an engine-less key meant to differ from the engine one, and is that what you want for cache reuse?
• Private cross-package imports: kernel.py:61 takes _freeze_metadata from microcosm.frame.bundle, executor.py:80 takes _encode_frame_metadata from .store.

Nits

• ruff format --check would reformat graph_population.py, test_uk_national_sampling.py and spec_engine/resolver.py; CI lint is ruff check only.
• efrs_parity_known_gaps.json re-escapes an em dash; serializer noise.
• .github/workflows/test.yml drops branches: [main] from pull_request so stacked PRs get CI — right call, and it will stay after merge, so say so.
• Body: "register canonical spine replacements" — the replacement stages already exist on Consolidate the US survey graph build and enrichment pipeline #893, this diff only deletes; "results will be added here" is still pending; the 730/39 test counts are not reproducible from the tree.
• The spec-seed-diagnostics failure is Consolidate the US survey graph build and enrichment pipeline #893's (identical on its own run), but item 2 means it would not pass here even once Max fixes it.

Could not verify

Engine-gated tests (h2 spine parity, full graph on fixture), the pinned-feed tests, the native f001 claim and the outstanding full-UK and wheel regressions.

---

Blocked on 1 and 2 until the US lanes and the spec-engine check are green; 3 is the one I would want before this leaves draft, since the Chronicle port is currently unreviewable as a merge hunk.

[juaristi22]

Follow-up to this review, against current head 051fb972:

1. The inventory/context blocker is fixed in e16acf20. This was introduced by this stack. weight_update.py is now in the package roster and all ten affected stage closures. The per-file import contracts were checked and updated too; resource-access and unbound-use hashes remain unchanged. Column-changing synthetic contexts now carry matching frame_column_order; production validation remains strict. Both US-p jobs passed on the preceding head, and the new run is still in progress.

2. The worker lock pin is fixed in e16acf20. The accepted digest is the actual 14bf10bf… lock. Checkout acceptance, altered-lock refusal, loader/inventory tests and the maintained coverage check passed locally. The coverage report's eight changed leaves were independently reconstructed identity digests, with all 42,156 fields and 41 checks accounted for. The complete guarded Linux diagnostic has now passed on 051fb972; its 65 source stamps match this exact head. Its artifact remains diagnostic-only, not release certification.

3. The Chronicle port is still in merge 22a4dbff; the requested commit split is not done. I have retained the published history. The merge-resolution view below exposes the graph port and admission test without treating the entire first-parent diff as the port:

git show --remerge-diff 22a4dbff -- \
  packages/microcosm-build/src/microcosm/build/uk_runtime/graph_targets.py \
  packages/microcosm-build/src/microcosm/build/uk_runtime/full_targets.py \
  packages/microcosm-build/src/microcosm/build/uk_runtime/full_problem.py \
  packages/microcosm-build/src/microcosm/build/uk_runtime/graph_build.py \
  packages/microcosm-build/tests/test_uk_full_graph_admission.py \
  docs/uk-chronicle-feed-repin.md

4. The PR body now states the CLI break explicitly. Former command names route to the new interfaces; retired flags such as --staging-h5, --allow-unpinned-feed and --crosswalk fail with migration guidance. Existing invocation scripts need migration. The body also now says the canonical replacement stages already existed on Consolidate the US survey graph build and enrichment pipeline #893, and documents that the stacked-PR workflow trigger remains after merge.

5. The inherited parity drift and deliberate identity change are now documented. The three H1 pins had inherited implementation drift. Their wrapper-specific source closures were audited, direct input and numerical fixture bytes remain unchanged, and the old/new node identities were independently reconstructed. Generic kernel.py/executor.py changes alone do not explain those wrapper hashes. One distinction: _context_digest runs before/after a kernel to detect input mutation. Adding metadata, mass records and column order strengthens that guard; it does not directly construct node keys. Source-hashed implementation changes invalidate cache identities.

On CGT: deleting the old hmrc_cgt_gains coverage entry was intentional. It already declared hmrc_cgt_gains_spine as its replacement. The canonical family remains required at build, including source pins verified before reading, capital-gains output/rewrite requirements, importance weights, its mass-conservation receipt and the same 76-fact calibration fence. What leaves active coverage is the historical adapter's prebuilt certified-H5/base-tier contract and old receipt, not canonical CGT construction. The frozen historical manifest/helper remain. This distinction is now in the body.

Engine-less and installed-engine identities intentionally differ: the dependency tuple records not-installed explicitly, so installing extras cannot reuse a result under a falsely matching numerical environment. Scientific spine stages bind that tuple and downstream identities inherit it. This is conservative and reduces cross-environment reuse. _freeze_metadata remains private cross-package coupling, reusing Frame's exact freezing semantics; that API cleanup is unresolved. _encode_frame_metadata is same-package internal reuse, keeping the mutation guard consistent with persistence's lossless typed encoding.

The three formatting nits remain; I have not changed them during this CI repair. The em-dash escape is serializer noise with identical decoded text; the two current_producer_stage additions in that JSON are substantive.

The old 730/39 counts were earlier scoped local runs; I removed them from the body rather than presenting them as a reproducible current-suite total. Current affected validation includes 521 provenance-guard cases, 69 parent US acceptance cases and an ordered 101-case PUF/survey regression with no skips or failures and no producer-identity drift. All 630 packaged files match the rebuilt wheels and installation. CI for 051fb972 is still running, so I am not marking the full CI result green. The PR remains draft; a completed native full build and native certification are still outstanding.