Bind UK census households from Chronicle and map NI Data Zones with NISRA's published lookup (#887)

[juaristi22]

Closes #887.

What this does

Every bound UK local calibration family now compiles from the pinned Chronicle consumer feed. The
last exception, census_households/households (650 constituencies + 361 local authorities), was
summed from the OA geography ladder and appended after the contract compile under the external:
prefix. It is now the 34th local contract target, ons.census.households, and the doors that let a
non-Chronicle family exist are closed: cross_grain.py refuses external: sides, every UK bridge
must name a contract: lower side, the pinned_in_ladder status and the 1,011 ladder-derived
battery exclusions are gone, and guard tests pin all of that. The OA ladder stays the
geography-assignment artifact and stage-one sampling weight; its household sums are diagnostics
only, scored against the Chronicle cells on every build.

Three parts, one PR (María's ruling):

1. Local Chronicle feed re-pin 6fb700e → ec7169b (chronicle main, 131,450 facts, facts sha
   4a50ee95…, manifest sha a95d0ee9…). Main already pins that artifact nationally (Align national UC targets with paid claims and explicit source windows #891); this PR
   moves the separately reviewed local pin to the same artifact. No national or local reference
   changed status or value; the four national UC family-type references now match 27 candidate facts
   instead of 9 because chronicle#251/qualified_passenger_vehicle_loan_interest is never imputed — OBBBA auto-loan deduction is structurally $0 #252's cross packages also satisfy their selectors (values
   unchanged; an exact dimensions pin is a follow-up). On the current base every national
   surface (references, membership, both parity receipts) regenerates byte-identical to main;
   nothing national moves in this PR.
2. Northern Ireland constituency mapping from NISRA's published lookup. The issue said NISRA
   publishes no DZ2021→PARLCON24 lookup; it does (Data Zone and Super Data Zone lookups v3, April
   2025, sheet DZ2021_Admin_geog_lookup, sha 8e2e1f6d…, now pinned in
   local_area_crosswalk.json beside the ONS and NRS lookups), and NISRA's own PARLCON24 census
   tables are built by aggregating Data Zones with it. The postcode-mode inference and both ONS
   postcode inputs are deleted. Rebuilt ladder: only constituency_code changes, at 10 Data Zones
   (2,075 households); NI sums vs the 18 published counts go from sum |Δ| 3,562 / max 694 to
   126 / 16 (mean 7), the same cell-key noise as England (7.4 / 29). ladder_vs_chronicle_household_dispersion
   receipts every build and refuses an NI regression (max > 50 or mean > 15).
3. Bind census households from Chronicle. ons.census.households selects
   record_set_spec_id: [households.by_constituency.v1, households.by_local_authority.v1]; each
   spec holds exactly one fact per area. Its measurement carries the occupied-household universe
   (uk.household.occupancy = occupied), so it pairs with the composition partition only through the
   declared bridge and never with dwelling stock. The A15 factor is now per grain from Chronicle sums:
   29,003,000 / 28,061,271 at constituency and / 28,061,277 at local-authority grain (both
   ≈ 1.03356, versus the ladder's 1.0335759). The A17 rule is unchanged; its factor moves to the
   local-authority value for all 1,436 tenure cells. The 1,011-cell before/after is in
   docs/evidence/uk-887/census-households-cells.json.

Review round 1 (Vahid, 2026-09-10) — what changed

• Manifest keys (blocker 1). New names say Chronicle: identity.targets.chronicle,
  census_household_uprating (per-grain grains, household_cells, tenure_cells),
  ladder_assignment_provenance, household_dispersion; candidate manifest and dry-run plans are
  schema_version 3. preflight_uk_local_release_candidate.py, assemble_uk_dense_release_dir.py,
  evaluate_uk_incumbent_surface.py and the microcosm-data dense-release contract read the new keys
  only (clean break; historical R16/R17 dirs already fail the current gate contract per the runbook),
  and the contract now names a missing identity instead of silently skipping the surface evaluation.
  Pre-existing --ledger-facts / load_ledger_consumer_artifact / pinned_in_ledger_facts names
  stay, per the chronicle_epoch "nothing renames on disk" rule.
• Rebase (blocker 2). Onto main 0afb1235 (Align national UC targets with paid claims and explicit source windows #891, Repair UC capital graph inputs and document calibration support gaps #892, UK dataset sizes: informed L0 and exact-count candidates (#355) #877, Align UK CGT calibration with FY2024-25 individual observations #889, Cancel outdated PR CI runs while preserving main pushes #899); national
  references, membership and parity receipts regenerate byte-identical to main; local surface
  unchanged (20,430 active); rebase-fallout fixes: main's PIPR composition tolerates an empty
  surface, size-selection arguments are validated before the mandatory Ledger arguments, main's two
  end-to-end size tests use the synthetic households-only inputs, main's Align national UC targets with paid claims and explicit source windows #891 seam-driver test
  asserts the reviewed local pin, and the workbook tests skip without openpyxl.
• Source rows (3). The three census source rows cite the artifacts Chronicle pinned: Nomis
  NM_2059_1 at TYPE172/TYPE424, NRS UV404 CSVs on the UKDS CKAN host, NISRA v=PARLCON24/v=LGD14.
• A17 (4). The receipt says "A17 rule unchanged, factor moves from 1.0335759 to the LA-grain
  1.0335595"; the receipts carry the old factor, the per-cell evidence file and per-country uprated
  deltas. The f100 re-measure is deferred by María's ruling; the f001 dry run is the pre-merge
  evidence.
• Micro-LA cells (5). local_area_support_exclusions.json is schema 3 with a signed
  bound_despite_support_floor.census_households block; the generator reads both the excluded areas
  and the binding families from the register (the hard-coded tuple and the code exemption are gone);
  the area-support gate window-checks both blocks.
• Signature question. Answered by construction: the occupancy filter separates the census cells
  from the unfiltered household-count signature, and the exact-signature skip added in the first
  cut is reverted (the shared cross_grain.py change is back to the external: refusal only).
  Fence and adjudication say so.
• Max's source review (same commit). (1) the unbound-bridge signature suppression is the defect
  above, fixed by construction; (2) the households-only scope now runs through
  uk_local_target_surface, so it applies the per-grain A15 factor and carries the uprating receipt
  (it previously bound raw compiled values); (3) openpyxl is declared in the microcosm-build[uk]
  extra with a named import error in the NISRA loader, and the workbook tests skip without it.
• Lock approval. Declaring openpyxl re-resolved uv.lock, and the US primary-QRF worker
  identity refuses an unapproved lock digest by design (first CI run on the rebased fixes failed only
  engine-us (us-qs) for that reason); APPROVED_UV_LOCK_SHA256 is re-pinned to the branch lock,
  the same one-line approval the merged Port post-freeze UK childcare/TFC spine inputs and calibration contract #834 tree needed.
• Nits. Touched files ruff formatted; _build_bound_problem joins target identities by area
  code; the NISRA sha is committed in the crosswalk register.

For María

• Two register entries carry your name and today's date pending your ratification at review:
  the re-worded census_disclosure_control_noise adjudication (approver/dates unchanged from
  2026-08-31 → 2026-11-30) and the new bound_despite_support_floor.census_households entry
  (2026-09-10 → 2026-12-10).
• The census_households: bind from the Chronicle feed and replace the inferred NI constituency mapping #887 issue text needs a correction comment (NISRA lookup exists; DZ codes are N20; 10 DZs, not
  ~9) — drafted, posted on your go.
• Follow-ups outside this PR: exact dimensions pins for the four UC family-type references; the
  imputation-anchor inventory in repos/uk-887-census-households-plan.md; the f100 re-measure.

Verification

• Receipts: experiments/887-census-households-receipts.md (Codex pass + independent re-run + review
  round) and docs/evidence/uk-887/census-households-cells.json.
• Known pre-existing failure, reproduces on main, local cache only:
  TestEfrsParityReference.test_cached_reference_regeneration_matches_committed_surface
  (89 vs 87 person input columns) — the deferred parity-reference re-mint from UK FRS stages evaluate policy parameters at the survey period: declare a policy-year rule (uk-data#475, uk-data#478) #862.
• CI groups: CI runs on this head (see the checks). An independent local re-run of the uk, shared-spec, spine-uk, rest, trade groups and the microcosm-data suite on the same tree is in progress and will be reported in a comment.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review pass at 0a8787dc (Claude Code, high effort; worktree checkout with the ec7169b5 feed rebuilt from chronicle main — facts 4a50ee95…, manifest a95d0ee9…, 131,450 rows, matching the new pin exactly, so the feed-dependent tests and regenerations ran here; no engine and no OA ladder artifact; the requested suites plus every changed test file: 662 passed, 2 skipped, 0 failed; ruff check clean; both census tools, the coverage manifest and ci_test_groups --verify current. GitHub reports the PR CONFLICTING, so no workflow has ever run on it.)

The data work is right

ons.census.households is declared as the 34th local contract target over both households specs; every one of the 1,011 compiled values equals the feed fact exactly, with one fact per area on each spec and Scotland's 32 councils coming from #253's record set (sum 2,509,275). The external: doors are closed in code — _contract_target_id raises on the prefix, _validate_rule refuses external sides, the cross-grain declarations require contract: — and the post-compile append loop is gone. The NISRA lookup is the published one: I downloaded it, the sha256 8e2e1f6d… matches the pin, the sheet maps 3,780 Data Zones each to exactly one of the 18 PARLCON2024 codes, and the ten Data Zones that moved carry exactly the codes the receipts list; the postcode inference and both ONS postcode inputs are deleted; the crosswalk rosters are unchanged. The per-grain A15 factors reproduce from the feed (650 cells / 28,061,271 and 361 / 28,061,277). Membership and references regenerate byte-stably with 1,011 rows added and 0 changed — the 241k/162k line count is a Myers-diff artefact, histogram gives +21k/−5 — and the dispersion receipt refuses an NI regression. It even repairs a stale signed-differences receipt on main: regenerating main's committed file with main's own tool and the new feed reproduces the PR's file minus the census rows exactly, though the body does not say so.

Findings

1. Blocking — the builder renames manifest keys that three untouched release tools still read, so every release candidate from this branch fails preflight and the assembler and incumbent evaluator crash. The candidate builder now writes census_household_uprating (build_uk_rowwise_candidate.py:791-814) and identity.targets.ledger (:1546, :2238). But preflight_uk_local_release_candidate.py:160 still reads manifest.get("ladder_household_uprating", {}) and fails on "A15 ladder household uprating not applied", assemble_uk_dense_release_dir.py:292 and :454 read identity.get("ledger") and exit on "must be a JSON object", :530 reads ladder_household_uprating, and evaluate_uk_incumbent_surface.py:169 indexes identity["ledger"]. The suite does not see it because the preflight and assembler tests fabricate manifests with the old keys and none of the PR's dry runs reach those tools. It fails closed, which is the right failure, but the release path is broken. Update the three tools and their test manifests, or write both keys.

2. Blocking — rebase; CI has never run. The PR was opened after #891 merged its own membership change and was born conflicting on target_reference_membership.json and test_uk_target_references.py. Resolve by regenerating the membership from the feed on top of #891 and #892, not by hand-merging the JSON, then let the hermetic regeneration test and CI run. Until then every count in the body is local-only.

3. Should-fix — the census source rows describe the OA-ladder inputs, not what Chronicle pinned. uk_local_target_census.json and local_target_census.py:283-336 still cite Nomis census2021-ts041.zip at OA grain, NRS's postcode-to-OA file and NISRA HOUSEHOLD&v=DZ21 as the family's sources. The feed's lineage is Nomis NM_2059_1 at constituency and LA types, NRS's UV404 CSVs on the UKDS host, and NISRA v=PARLCON24 / v=LGD14. The url and product fields should name the artifacts Chronicle actually pinned.

4. Should-fix — A17 moves, the body says "A17 unchanged", and the measurement is incomplete. The rule is unchanged but the factor is not: 1.0335759 on main, 1.0335595 at LA grain here, applied to all 1,436 census-tenure holds. The old factor is never stated, the 1,011-cell before/after is given only as per-country mean, max and net in the receipts, and #887's own step 7 — the f100 re-measure, which the issue predicts will worsen NI fit on first measurement — is deferred. That number belongs in the receipts before merge.

5. Should-fix — the generator exemption is an unsigned doctrine deviation. generate_uk_local_target_references.py:442-446 exempts ons.census.households from the A14 micro-LA support-floor deferral by code comment; local_area_support_exclusions.json is untouched, and A14's register says the excluded authorities' own LA cells are signed-deferred. The body flags it for María; it needs a register entry, not a comment.

Questions

• The exact-signature skip reaches further than the census rows. cross_grain.py:136-142, 164-168 compute bridge_is_unbound for any row, so the seven bound ons.household_composition.* country cells also lose their signature key; on main they could group with voa.council_tax_stock.total, which shares the unfiltered {household, uk.household.count} signature. #887 said no cross-grain rule change beyond the lower_side rewrite. It is disclosed in the body, but the fence text ("when a national same-concept control is bound, the standing rule rescales both grains; country wins") now describes a path the code does not take while the composition bridge is reviewed-unbound. Once the #757 cells land the census rows will be keyed with both the VOA total and the partition and should fail closed; say so.
• Should ons.census.households carry a distinguishing measurement (occupied-household universe) so the signature rule can never pair it with dwelling stock?

Nits

ruff format --check fails on ten files this PR touched (main fails on one); CI does not enforce it. _build_bound_problem overwrites target_frame["target_name"] positionally — correct today because the area-code order defaults to the frame's, but fragile. The NISRA sha lives only in the receipts and the uncommitted ladder metadata, not in a register.

Needs the licensed data

The OA ladder artifact (two skips), the old-versus-new per-cell values for the 1,011 census cells, the A17 before/after, and the f100 re-measure.

---

Not mergeable as filed: 1 breaks the release path downstream of the builder, and 2 means nothing has been run on the merge ref. Both are mechanical. The binding, the mapping and the regeneration discipline are the substance and they check out.

[juaristi22]

Thanks, Vahid — all five findings, both questions and the nits are addressed on the rebased branch
(6a3a8f23), plus Max's three source findings. The licensed f100 re-measure stays deferred by María's
ruling.

Findings

1. Release readers and renamed manifest keys. Candidate manifests are now schema 3 and write
   only identity.targets.chronicle; the dispersion API/key is
   ladder_vs_chronicle_household_dispersion / chronicle_households. The release preflight,
   dense assembler, incumbent-surface evaluator, and microcosm-data release contract read only
   the new shape. The data contract now names a missing identity.targets.chronicle explicitly,
   so a stale manifest fails loudly instead of skipping the incumbent-surface check. Coverage:
   test_uk_local_release_preflight.py, test_uk_dense_release_assembler.py,
   test_uk_incumbent_surface_evaluation.py, and test_uk_dense_release_contract.py.

2. Conflict / CI never ran. The branch is rebased onto main 0afb1235 (Align national UC targets with paid claims and explicit source windows #891 national/local
   feed split, Repair UC capital graph inputs and document calibration support gaps #892 UC diagnostics, UK dataset sizes: informed L0 and exact-count candidates (#355) #877 size selection, Align UK CGT calibration with FY2024-25 individual observations #889 CGT observed year, Cancel outdated PR CI runs while preserving main pushes #899 CI
   concurrency). National references, membership and parity receipts regenerate byte-identical to
   main's; the local surface is unchanged at 20,430 active. CI runs on the merge ref now; the first
   run's only failures were two tests fixed here (main's seam-driver test pinning the old local
   digests, and the workbook tests importing openpyxl before their skip).

3. Census source rows cited ladder inputs rather than Chronicle lineage. The three household
   source rows in uk_runtime/local_target_census.py now cite the exact pinned products: Nomis
   NM_2059_1 TS041 at TYPE172/TYPE424, the NRS UV404 constituency/council CSVs on UKDS CKAN, and
   NISRA HOUSEHOLD at PARLCON24/LGD14. uk_local_target_census.json was regenerated and tests pin
   those product/URL markers.

4. A17 factor moved and receipts understated it. The runtime adjudication and
   experiments/887-census-households-receipts.md now state that the rule is unchanged but all
   1,436 tenure cells move from the old ladder factor 1.0335759 to the LA-grain Chronicle factor
   1.0335595. docs/evidence/uk-887/census-households-cells.json contains all 1,011 source-only
   cell comparisons: ladder and Chronicle counts, old/new uprated values, and deltas, summarized
   by country and grain in the receipt. Per María's ruling, the licensed f100 re-measure is wholly
   deferred; the f001 dry run is the pre-merge evidence.

5. Unsigned micro-LA exemption. uk/local_area_support_exclusions.json is now schema 3 with
   María's signed bound_despite_support_floor.census_households approval (review window
   2026-09-10 through 2026-12-10). The schema-3 loader validates and returns both blocks; the gate
   window-checks both; and the reference generator derives both excluded area ids and exempt
   families from the register. Regeneration remained byte-identical: 20,430 active / 513 deferred.

Questions

1. How far did the signature skip reach? It no longer exists. I reverted the two generic
   bridge_is_unbound skips, so a reviewed-unbound bridge's rows retain their exact-signature keys;
   test_cross_grain.py pins that regression. The UK census case still produces no signature group
   because its measurement now has the occupied-household filter, while the composition cells do
   not. test_reviewed_household_composition_gap_leaves_census_contract_unbound pins groups == [].

2. What measurement distinguishes census households? ons.census.households.measurement now
   includes filters: [{"concept": "uk.household.occupancy", "equals": "occupied"}]. That
   expresses the published occupied-household universe and prevents accidental same-signature
   pairing with dwelling stock. The fence and signed adjudication say that these cells pair only
   through the household-composition bridge; once fully bound, the standing cross-grain rule
   applies, with country winning.

Nits

• Ruff formatting was applied only to the Python files in the PR/current touched-file set, not the
  repository.
• _build_bound_problem now joins target_name and contract_target_id by area code; it no
  longer overwrites them positionally.
• generate_uk_local_area_crosswalk.py now copies each ladder layer's per-country source, URL,
  sha256, and vintage into levels.<level>.ladder_layer_sources. The test pins NISRA sha
  8e2e1f6daaddd2f5b6887ccab1bf9d0bf179f6d0d2ba60ef8e0ebb495a3e1999.

Max's source review (0a8787d)

Max traced three findings without executing them; all three are in this push.

1. An unbound bridge also suppresses independent exact-signature reconciliation — the same defect
   as your question above; the skip is reverted and the occupancy filter carries the distinction.
2. --households-only drops the computed uprating — confirmed: _build_bound_problem bound the raw
   compiled census values and its manifest said "no cross-grain receipt". The households-only scope
   now goes through uk_local_target_surface with the same per-grain receipt as the joint scope, so
   the constituency cells take the A15 factor and the manifest carries census_household_uprating;
   test_households_only_targets_come_from_compiled_chronicle_registry pins raw values without a
   receipt and ×factor with one.
3. The NI loader needs openpyxl declared for installed microcosm-build — confirmed: only the
   workspace dev group carried it. openpyxl>=3.1 is now in the microcosm-build[uk] extra
   (uv.lock re-resolved), the loader raises a named ImportError pointing at the extra, and the
   workbook tests skip where the dependency is absent (the wheels lane).

[vahid-ahmadi]

Second pass at 0d978f20 (Claude Code, high effort; worktree checkout, the fix commit 6a3a8f23 and the lock approval read against the dispositions; the branch is three commits on main 0afb1235; 1,433 passed, 13 skipped, 0 failed across the 42 touched and named suites without the engine or the pinned feed; ruff check clean on all 93 changed files; both census tools, the coverage manifest, the spec-engine coverage check and ci_test_groups --verify current; CI green on the merge ref).

All five findings, both questions and the nits closed as described, with tests that fail on the previous head.

1. Verified. The builder writes schema 3 and identity.targets.chronicle in all three manifest paths, and every release reader — preflight, dense assembler, incumbent evaluator, and the microcosm-data release contract — reads only the new shape; no ladder_household_uprating or identity.ledger read remains anywhere in Python. The contract names the missing key and fails rather than skipping. With the new test files dropped onto 0a8787dc, 32 tests fail across all five suites, so the mismatch would have been caught. One residual: the contract indexes manifest_sha256 directly while provenance() sets it conditionally; it is inside the same try block so it fails closed, but the message would not name the key.

2. Verified. Merge-base is 0afb1235; the national references, membership, both signed-differences files, the national feed fixture and the graph parity pins are byte-identical to main; the local membership carries 20,430 active, 513 signed-deferred, 1,586 no-fact. CI has now run on the merge ref and passes.

3. Verified. The three source rows cite the pinned products and URLs (Nomis NM_2059_1 at TYPE172/TYPE424, NRS UV404 on UKDS CKAN, NISRA HOUSEHOLD at PARLCON24/LGD14), the test pins them, and census_uk_local_targets --check is current.

4. Verified and reconciled. The evidence file holds all 1,011 cells (839 England, 89 Scotland, 54 Wales, 29 Northern Ireland; 650 constituency, 361 local authority) with both counts, both uprated values and both deltas, and its sha matches the receipt. I recomputed all sixteen rows of the two receipt tables from the JSON and every value matches; the local-authority factor is exactly 29,003,000 / 28,061,277. The 1,436 tenure holds are 4 families × 359 authorities in the membership. The body and receipt now say the factor moves.

5. Verified, one test gap. Schema 3 with the signed bound_despite_support_floor.census_households entry; the loader requires both blocks and refuses schema 2; the generator derives the exempt families from the register and no census literal remains. The gate loops over both blocks, and when I moved an expired entry into the new block in a scratch copy of the test, the binding failed closed on bound_despite_support_floor/census_households. But no committed test does that — test_uk_battery_bindings.py:1123 exercises only the exclusions block with an empty second block. Worth one parametrised case so the window on the new block is pinned, not just believed.

Q1 verified. Both bridge_is_unbound skips are gone and test_reviewed_unbound_bridge_keeps_the_exact_signature_group fails on the old head. The groups == [] test dates from the first commit and passes on both heads, so it is a fence rather than a regression pin for this fix, which is fine.

Q2 verified. The occupancy filter is on the measurement in uk_population_targets.json.

Max's three and the nits verified. Households-only routes through the surface with the per-grain receipt and the test pins raw-without-receipt versus ×factor-with-receipt; openpyxl>=3.1 is in the [uk] extra with the lock re-resolved, the loader raises a named ImportError, and the workbook tests skip here. ruff format --check flags only main's test_spec_engine_field_usage.py, untouched by this PR.

---

Approving. The one thing I would add before merge is the expired-window case for the new register block; the manifest_sha256 message is a nit. The "For María" items — ratifying the two entries that carry her name and the #887 correction comment — are hers.