Skip to content

Publish calibration hierarchy schema 8 - #855

Merged
anth-volk merged 78 commits into
mainfrom
codex/schema7-geography-dimensions
Sep 11, 2026
Merged

Publish calibration hierarchy schema 8#855
anth-volk merged 78 commits into
mainfrom
codex/schema7-geography-dimensions

Conversation

@anth-volk

@anth-volk anth-volk commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • require every calibration target declaration to select a category from the same contract that defines provider and category labels
  • carry typed provider, category, geography, ordered dimension, and target identity through generated references, TargetSpec, registry serialization, runtime targets, and diagnostics
  • inherit geography, categorical coordinates, categorical value labels, and exact single-fact target labels from Chronicle, with deterministic Microcosm fallbacks when Chronicle cannot supply a label
  • publish complete hierarchy-bearing diagnostics as schema 8 while retaining schema 6 for generic calibrations that declare no hierarchy
  • require complete hierarchy metadata for every registry-backed diagnostics row, including Microcosm-generated UK local household controls
  • cover all current UK providers, including the Department for Education and Department for Transport, with non-empty labels
  • retain the target declarations currently present on main
  • document hierarchy ownership, propagation, and shared geography constants

Validation

  • uv run ruff check .
  • calibration diagnostics, provider-label, and category-label unit tests
  • UK diagnostics, Ledger-target, local-rowwise, and complete rowwise-candidate test modules
  • US fiscal diagnostics regression test
  • diagnostics schema 7 compatibility and schema 8 validation tests

Fixes #854

@anth-volk
anth-volk force-pushed the codex/schema7-geography-dimensions branch from 790766f to 7ca4a5d Compare September 9, 2026 16:06
@anth-volk
anth-volk marked this pull request as ready for review September 9, 2026 19:55
@anth-volk
anth-volk requested a review from juaristi22 September 9, 2026 19:55
juaristi22 and others added 2 commits September 9, 2026 23:00
Repair UC capital graph inputs and document calibration support gaps
…tialization (#355)

Attesting microcosm.calibrate.gates and microcosm.calibrate.initialization
in seeds.py added two resolved-binding fields (9,770 -> 9,772; ledger
42,154 -> 42,156; compiler_semantic 27,715 -> 27,717; no_generation0_effect
3,678 -> 3,680). field_usage.py and the coverage tool test were re-pinned in
c2ee134; this sibling test only runs in the rest, engine-shared and wheels
lanes and was missed. Values recomputed from the ledger, not transcribed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22

Copy link
Copy Markdown
Collaborator

Review: request changes

Reviewed head: ab66edc

C1 — Must fix: stack this work on #900 and preserve its Chronicle household contract

PR #855 independently migrates the census-household target away from runtime ladder injection, but retains the synthetic contract ID external:census_households/households. Its generic local-authority support-floor path then defers E06000053 and E09000001.

The generated artifacts show the resulting difference:

Branch Contract Total Constituencies Local authorities E06000053 / E09000001
#855 external:census_households/households 1,009 650 359 0
#900 ons.census.households 1,011 650 361 2

I recommend rebasing/updating #900 onto current main, then stacking #855 on #900. During conflict resolution, preserve #900's Chronicle contract and feed pin, NISRA mapping, 1,011-cell set, refusal of external target IDs, and diagnostic-only OA ladder. Then reapply the schema-8 hierarchy work and regenerate the reference, membership, parity, and golden artifacts.

In particular, replace the #855 test that approves the two household deferrals with an invariant asserting that all 1,011 cells and both named IDs remain present.

Relevant code: generic support-floor handling. The two branches currently have 19 conflicted paths, so this needs deliberate conflict resolution rather than a blind rebase.

A1 — Should address: fail closed for required source-owned labels

The schema-8 hierarchy direction is good, but the implementation humanizes stable identifiers when Chronicle labels are missing. The same fallback also masks conflicting labels: when selected facts do not produce exactly one unique nonblank label, the code substitutes identifier-derived display text.

That can make a broken upstream contract look valid. I suggest:

  1. Require Chronicle-owned fact, dimension, and dimension-value labels; reject missing or conflicting values.
  2. For multi-fact, restamped, or derived targets, require an explicit Microcosm-owned target label rather than borrowing a member-fact label or humanizing an ID.
  3. Require provider and category labels in one normalized Microcosm-owned catalog, without authoring-path humanization fallbacks.
  4. Add negative tests for missing labels, conflicting labels, multi-fact targets, and restamped targets.

Relevant code: hierarchy label fallback.

Geography

A narrow display-name fallback is reasonable only after an explicit valid geography identity has been supplied. For example, country/K02000001 may resolve to “United Kingdom” through a shared authoritative catalog.

I would not interpret missing geography as UK. “National” can mean UK, Great Britain, England and Wales, England, or another coverage universe. If a source adapter guarantees UK coverage, encode country/K02000001 explicitly at that boundary; unknown or absent geography identities should fail schema-8 validation.

Validation

  • git diff --check passed.
  • At review time, exact-head GitHub CI had 19 successful checks, no failures, and three jobs still running.
  • Local tests were not run because pytest was unavailable in the locked offline environment.
  • GitHub reported the branch as dirty against its current base.

Overall: the typed hierarchy model and all-or-none schema-8 validation are directionally strong, but I would not merge until #855 is stacked on #900 and the label fallback contract is tightened.

juaristi22 and others added 14 commits September 10, 2026 10:43
UK dataset sizes: informed L0 and exact-count candidates (#355)
Cancel outdated PR CI runs while preserving main pushes
Align UK CGT calibration with FY2024-25 individual observations
…ISRA's published lookup (#887)

- Re-pin the UK Chronicle consumer feed 6fb700e -> ec7169b (131,450 facts,
  facts sha 4a50ee95); no national or local reference changed status or value.
- Northern Ireland: replace the active-postcode modal DZ2021->PARLCON24
  inference with NISRA's published administrative lookup (v3, April 2025) and
  rebuild the OA ladder: only constituency_code moves, at 10 Data Zones; NI
  sums land within 16 households of NISRA's 18 published counts (was 694). A
  ladder-vs-Chronicle household dispersion receipt scores every build.
- Declare ons.census.households as the 34th UK local contract target (650
  constituency + 361 local-authority cells) compiled from the two households
  record-set specs; delete the ladder append loop; A15 uprating is now per
  grain from the Chronicle sums (29,003,000 / 28,061,271 and / 28,061,277);
  A17 tenure cells take the local-authority factor.
- Close the hatches: cross_grain refuses external: sides and skips the
  exact-signature key for a reviewed-unbound bridge; UK bridges must name a
  contract lower side; the pinned_in_ladder status and the 1,011
  ladder-derived battery exclusions are removed; guard tests pin all of it.
- Registers and docs: census disclosure fence and adjudication re-worded to
  per-area dispersion (approver and dates unchanged, pending ratification);
  receipts in experiments/887-census-households-receipts.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Vahid (CHANGES_REQUESTED at 0a8787d):
- Blocker 1: the candidate manifest keys this PR renamed are read under their
  new, Chronicle-named shape by the release preflight, the dense-release
  assembler, the incumbent-surface evaluator and the microcosm-data release
  contract: identity.targets.chronicle, census_household_uprating with
  per-grain factors, ladder_assignment_provenance, household_dispersion
  (ladder_vs_chronicle_household_dispersion / chronicle_households); manifest
  and dry-run plans are schema_version 3; the contract names a missing
  identity instead of silently skipping the surface evaluation. Frozen
  ledger-named surfaces stay per the chronicle_epoch rule.
- Blocker 2: rebased onto main 141bfac (previous commit); national
  references regenerate byte-identical to main, the local surface is
  unchanged; main's stale national parity receipts refresh on its own pin.
- Should-fix 3: the census source rows cite the artifacts Chronicle pinned
  (Nomis NM_2059_1 TYPE172/TYPE424, NRS UV404 CSVs on UKDS CKAN, NISRA
  PARLCON24/LGD14).
- Should-fix 4: the receipt states the A17 rule is unchanged while its factor
  moves from 1.0335759 to the LA-grain 1.0335595; all 1,011 cells before and
  after in docs/evidence/uk-887/census-households-cells.json. The f100
  re-measure is deferred by ruling.
- Should-fix 5: local_area_support_exclusions.json is schema 3 with a signed
  bound_despite_support_floor.census_households block; the generator reads
  excluded areas and binding families from the register.
- Questions: ons.census.households carries the occupied-household
  measurement filter and the exact-signature skip is reverted, so the census
  cells pair only through the composition bridge, never with dwelling stock;
  fence and adjudication say so.
- Nits: touched files formatted; households-only identities joined by area
  code; the NISRA lookup sha pinned in local_area_crosswalk.json.

Max (source review at 0a8787d):
- The households-only scope now runs through uk_local_target_surface, so it
  applies the per-grain A15 factor and carries the uprating receipt.
- openpyxl is declared in the microcosm-build[uk] extra with a named import
  error in the NISRA loader; the workbook tests skip without it.

CI on the rebased commit: main's #891 seam-driver test now asserts the
reviewed local pin, and the workbook tests importorskip openpyxl before the
loader imports it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ty (#887)

Declaring openpyxl in the microcosm-build[uk] extra re-resolved uv.lock,
and the primary-QRF worker identity refuses an unapproved lock digest by
design (engine-us us-qs: 13 failures + 324 errors, one cause). Same one-line
approval as 45833f1 for the merged #834 tree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@anth-volk
anth-volk merged commit 837326b into main Sep 11, 2026
23 checks passed
MaxGhenis added a commit that referenced this pull request Sep 11, 2026
Main's diagnostics schema 8 (#855) requires a complete hierarchy for every
registry-backed target, so the `national.demographic_calibration` node
failed on the merge ref: its S0101 registry carried none. Each activated
cell now declares the only hierarchy it may have — provider `census_acs`,
category `census_acs.population_by_age` (S0101) or
`census_acs.population_by_sex_and_age` (B01001), both labels added to the
reviewed US variable-label catalog, the activation's own national publisher
row (0100000US, "United States", level country), one `publisher_cell`
dimension whose value is the cell and whose label is the published one the
band binds, and the cell itself as the target so its id equals the spec
name. Both S0101 spec builders attach it; the registry declaration round-
trips it; the registry validator refuses a spec without exactly the
hierarchy rebuilt from its table, name, label and geography; the survey
diagnostics verifier's defensive pin moves from schema 6 to 8.

Controls: the invented seven-node graph's diagnostics are schema 8 with a
hierarchy on every row; the registry JSON round-trips every hierarchy;
missing, foreign-cell, foreign-geography and foreign-category hierarchies
are refused; the label catalog pin lists the two new rows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
juaristi22 added a commit that referenced this pull request Sep 11, 2026
#904)

Main merged the calibration target hierarchy (schema 8, #855) after
round 2. The eight #890 targets now declare a catalog category and a
label (four new providers: ORR, Welsh Government, DfI Northern Ireland,
NITHC; six new categories; the cars fuel duty under obr.efo_receipts),
and the committed references carry the contract-derived hierarchy
lookups the generator emits for the 196 contract targets with an active
reference.

What the rebase exposes and this commit does not fix: main's hierarchy
completion requires dimension labels on every selected Chronicle fact
(chronicle#261), which no artifact carries, so on main and here alike
the generators and compile_uk_target_registry return nothing against
the pinned feed (0 compiled / 424 unsupported). The reference surfaces
are the round-2 regenerations plus those lookups; the feed-backed
regeneration and compile tests skip in CI and fail locally until #261
or the interim fallback that issue announces. Vendored resources still
regenerate byte-identical.

The nine English regions join Microcosm's UK geography label catalog:
the London fact carries no geography name and the catalog had only
country codes, so the region leg would be refused at compile once labels
arrive; a test checks the national region roster is covered.

Conflicts: measure-exclusion counts to the merged truth (47 entries,
tranche 35); the UK spec digest re-pinned; the target-fit register and
its tests are main's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish schema 7 structured geography diagnostics for UK and US

3 participants