Skip to content

UK dataset sizes: informed L0 and exact-count candidates (#355) - #877

Draft
juaristi22 wants to merge 12 commits into
mainfrom
uk-dataset-sizes-355
Draft

UK dataset sizes: informed L0 and exact-count candidates (#355)#877
juaristi22 wants to merge 12 commits into
mainfrom
uk-dataset-sizes-355

Conversation

@juaristi22

@juaristi22 juaristi22 commented Sep 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

The UK rowwise driver currently exports every cloned household. This adds --dataset-households to request a compact candidate after the dense solve, using contribution-informed L0, exact-count sampling, and a frozen-support refit.

Based on main after #870 (uk-rowwise-candidate-762-b) merged as 2894377, so the diff contains only the size-selection increment. Refs #355 and #665; this does not close #355's certification and default-switch requirements.

Behavior

  • Keep pool generation separate from output size: the existing K=15, grain_equal, 1,500-epoch defaults and local gates remain.
  • Initialize L0 using target-contribution shares and protect each nonzero target's largest absolute weighted carrier. The prior initializes search; it is not a top-k selector.
  • Reuse exact-count Sampford sampling and the shared normalized Horvitz–Thompson refit baseline. The stretch multiplier remains 10 relative to that inclusion-adjusted baseline; the manifest explicitly records this reference.
  • Freeze full-pool target contributions, preserve linked households, benefit units and people, and rerun selection inside each holdout fold.
  • Record requested/realized sizes, selection provenance, dense/refit diagnostics and Logbook receipts.

Review boundary

These are candidate artifacts. Size requests refuse --release-candidate and remain unreleasable until matched comparisons and the promotion scorecard are adjudicated. No dataset registry, production pointer or downstream default changes. Full builds still require the dense build's peak memory and add selection/refit work; savings apply to the exported dataset and downstream use.

The prior mapping and inclusion-adjusted stretch reference need measured UK evaluation before release. No licensed full-data build or reform scorecard was run. The implementation plan and operating instructions are in docs/uk-dataset-size-plan-355.md.

Validation

  • Calibration, UK rowwise runtime/driver, doctrine, dense assembler and country-spec regression run: 433 passed.
  • Additional identity/seed, US exact-count compatibility and UK runtime run: 66 passed (overlapping the regression run).
  • Final size-specific and dry-run checks: 16 passed.
  • Repository-wide Ruff check, changed-file formatting, diff whitespace and CI test inventory verification passed.

@MaxGhenis MaxGhenis mentioned this pull request Sep 7, 2026
Open
10 tasks
@juaristi22
juaristi22 force-pushed the uk-rowwise-candidate-762-b branch 2 times, most recently from d5f04a8 to 3516100 Compare September 7, 2026 21:33
@juaristi22
juaristi22 changed the base branch from uk-rowwise-candidate-762-b to main September 8, 2026 10:36
juaristi22 and others added 7 commits September 8, 2026 15:01
@juaristi22
Add exact-household-count UK dataset candidates (#355)
acbc633
@juaristi22 @claude
Re-pin the spec-engine identities moved by the calibrate solve change (…
08ae0ca 
…#355)

The seed-protocol attestation hashes the source of microcosm.calibrate.solve
(spec_engine/seeds.py, _DIRECT_KERNEL_MODULES), which the informed-L0 and
exact-count selection change edits. Every identity downstream of it moves:
the am/be/uk country-bundle spec digests, the loader's semantic-hash golden
vector, the seed-protocol and compiled seed-map digests in the inventory
coverage table, and the committed US coverage report (regenerated with
tools/spec_engine_coverage.py, 42154/42154 fields, 41/41 checks). Main's
tree passes the same four test files unchanged, so the move is this
branch's, not an inherited one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Re-pin the H1 kernel parity fixture and the US spec digest moved by t…
366b826 
…he calibrate solve change (#355)

Two lanes the local shared-spec run did not cover still carried identities
that hash the calibrate source:

- `calibrate.adam@1`'s implementation hash (a source hash over the
  calibrate modules, microcosm/calibrate/kernels.py) moved to d24fc41d…;
  the H1 fixture was regenerated on its authoring platform
  (arm64/darwin/py3.14, numpy 2.4.6 / pandas 3.0.3 / scipy 1.17.1 /
  torch 2.12.0) with tools/graph_parity_fixtures.py. Only
  calibrate/pins.json changed: the direct bytes and the graph are
  identical, so the kernel's behaviour did not. The fit.qrf pins were
  left untouched — the generator resets that case's platform map to the
  authoring platform, which would have dropped the two linux pins CI
  asserts.
- The US spec_sha256 pinned in test_us_multispine_pool_tool.py moved
  with the country-spec digest already re-cut in the coverage report
  (9db2d6db…).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Size runs keep their dense reference and selection as evidence; --sel…
cfd57e0 
…ection-seed moves only the draw (#355)

A --dataset-households run replaces its calibration product with the
compact refit, but the dense joint solve it was cut from is byte-identical
to a standalone dense run on the same inputs, seed and epochs. Keeping it
lets a size candidate be compared with its own dense reference without a
second full-pool run:

- local_rowwise: the evidence labelling (alignment by name, exact target
  values, local/national diagnostics, past-cap censuses, initial loss)
  moves into _doctrine_solve_evidence and runs for both results; the
  solve product carries `dense_reference` (weights, initial weights,
  diagnostics, losses, censuses).
- driver: a size run writes dense_reference_diagnostics.csv (every
  target's dense estimate, local rows then national, with a `grain`
  column) and dataset_size_selection.csv (pool row index, household id,
  clone index, design weight, inclusion probability, certainty,
  Horvitz–Thompson baseline, refit weight), lists both under outputs with
  digests, and summarises the reference under solve.dataset_size
  (losses, fit by family, uk_weight_summary). Dense runs are unchanged;
  the two files are size-run-only in the publish order.
- --selection-seed (requires --dataset-households; defaults to --seed)
  seeds only the informed L0 search, the exact-count draw and the refit,
  threaded through the holdout as well, so two selections compare on one
  pool and one dense reference; recorded as parameters.selection_seed.

Tests: the doctrine solve keeps a reference equal to the standalone dense
solve and a selection seed leaves it untouched; the CLI size run writes
both sidecars, lists them, and records both seeds; --selection-seed
without a size is refused.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Carry the exact-count feasibility measurement with the size refusal a…
12b9a7c 
…nd receipt (#355)

The first licensed size rehearsal (55,000 of 792,690 at 100 epochs) was
refused inside select_exact_k: "degenerate boundary mass … adjust pi_hi or
k". With pi_hi=1.0 every gate below one stays in the boundary and its open
probability is scaled to the remaining draw size m, so the design is
feasible only when m * max(pi_boundary) <= sum(pi_boundary); the L0 budget
search stops on the count of not-fully-closed gates, which sits above the
open-probability mass while gates are only partly polarised. The refusal
carried no numbers.

refit_uk_dataset_size now measures selection_feasibility before the draw —
certainties and boundary draw at pi_hi=1, boundary mass and largest boundary
gate, feasibility, the largest household count feasible at pi_hi=1, the
smallest feasible pi_hi on a fixed grid, gate quantiles and counts, the
budget search's n_nonzero and lambda — and attaches it to the ValueError on
refusal and to the size receipt on success. Nothing is clamped or promoted;
the ruling on pi_hi stays a reviewed decision, now made from measured mass.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Evaluate a size candidate with one command: size_evaluation library, …
6ec596c 
…evaluate_uk_dataset_size steps 00-90, uk_fit_by_family lifted (#355)

Every evaluation piece for a --dataset-households candidate now has one
name and one place:

- uk_runtime/size_evaluation.py — pure measurement over a run directory:
  load_run (tolerant of pre-#877 dense runs), run_acceptance (the size-run
  checklist, not_applicable on dense runs), fit_tables (by grain and
  family), weight_tables (Kish ESS, distinct sources, stretch vs the
  Horvitz–Thompson baseline and vs pool design from the selection sidecar
  or a spine, self-checked against the manifest), area_support_tables
  (per-grain floors and breaches), gate_table (six ids with criticality),
  paired_targets (join on name; wins/ties/losses; the reference's red rows
  tracked through family/area/metric), dense_reference_deltas (size-only
  effect from the run's own dense reference), frozen_vs_recomputed
  (national rows against the incumbent-surface evaluator), footprint, and
  summarize with PRE_REGISTERED_OUTCOMES_V1 flags. No subprocess, no
  engine, no licensed data in tests.
- tools/evaluate_uk_dataset_size.py — the one command: steps
  00-run-acceptance, 10-dense-reference, 20-vs-reference/<label>,
  30-incumbent-score (score_uk_local_candidate), 40-incumbent-surface
  (evaluate_uk_incumbent_surface --engine-blocks 1 + frozen vs
  recomputed), 50-downstream (uk-candidate-eval T3/T4/T5 and the #731
  scorecard under this venv, each timed with /usr/bin/time -l), 90-summary
  (EVALUATION.md + JSON with the pre-registered table and a file index);
  every step writes receipt.json (inputs and outputs with digests, argv,
  wall, exit) and is skipped with a reason when its inputs are absent;
  re-running a step rewrites only its folder.
- _fit_by_family lifted from the driver into
  uk_runtime/diagnostics.py::uk_fit_by_family unchanged; the driver
  imports it.

Tests: test_uk_size_evaluation.py (synthetic run dirs in both manifest
shapes, table-format H5s, time -l parsing, red-row mapping, stretch
self-check, breaches, flags, refusals) and
test_uk_evaluate_dataset_size_tool.py (tree, receipts, manifest, summary;
skipped steps; a failing downstream stub recorded without aborting;
single-step re-run). Implemented by Codex from .codex-work/PLAN.md;
verified here: ruff, format, ci_test_groups --verify, 88 tests across the
four UK rowwise/evaluation files.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Re-cut the UK spec digest for the tree rebased onto #879 (#355)
573c4b4 
Both sides moved the UK country-bundle digest: main's #879 changed
uk/spec/sources.yaml and country_package.json, this branch's solve.py edit
moved the seed-protocol attestation inside the same bundle. The rebase
conflict was that one line; the combined tree's digest is d57d248f…,
read from the pin test on the rebased tree. The am/be digests, the loader
golden vector, the seed-protocol and seed-map digests and the committed US
coverage report were unchanged by #879 (49 of the 50 pin tests passed
before this cut).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
juaristi22 and others added 5 commits September 8, 2026 15:17
@juaristi22 @claude
Receipts for the #355 size experiment: Step 0 CI fix, inputs, spine-o…
df47f89 
…/spine-p, S0, S1/S1b refusals with the feasibility numbers (#355)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Skip the H5-reading size and evaluation tests where pytables is absen…
fb85c8f 
…t, as the wheels venv is (#355)

The wheels lane runs the suite against installed wheels in a venv without
pytables; the size CLI test read the exported candidate through
pd.HDFStore and failed there (the only failure in 7,963 tests). The
rowwise candidate test file's convention is importorskip("tables") and
importorskip("h5py") at the top of every CLI test; the size test now
follows it, and the two evaluation test files, which write PyTables-format
H5 fixtures, carry the module-level importorskip the same lane relies on.
With pytables present nothing is skipped (41 tests run); with the import
blocked the modules skip instead of erroring.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Make the exact-count certainty threshold a recorded candidate-run kno…
a6b1e0c 
…b: --selection-pi-hi (#355)

refit_uk_dataset_size takes pi_hi (default 1.0, the protected-carriers-only
design); the driver exposes it as --selection-pi-hi, candidate-only and
bounded to (0, 1], threaded through the holdout, recorded as
parameters.selection_pi_hi and in the size receipt (selection_pi_hi, the
draw receipt's pi_hi, and the feasibility measurement's
requested_pi_hi / feasible_at_requested_pi_hi). The refusal message names
the requested threshold. A threshold below one promotes learned
near-certain gates; the plan doc records the measured reason and the
2026-09-08 ruling (smoke at the measured feasible count; the 55,000
candidate at pi_hi 0.95, the US exact-k ladder's setting, and 2,000
epochs). Nothing changes for dense runs or for the default.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Refuse unsupported nonzero targets by name before the size selection (#…
b1d206d 
…355)

The 45,800 smoke on spine-p (the tree rebased onto #879) was refused inside
contribution_initialization: "nonzero target at row 20648 has no support",
a national row whose constraint row is all zeros on the cloned pool. The
dense solve tolerates such a row as a miss; a size selection cannot carry
it, and the row index alone is not actionable. refit_uk_dataset_size now
lists every nonzero target with no supporting household by name before
initialisation and refuses with the list, so the binding defect is fixed
upstream or the row is excluded with a signed reason, never selected
around. unsupported_nonzero_targets() is the reusable check.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@juaristi22 @claude
Time downstream steps portably: an in-process rusage shim instead of …
e8f7c6f 
…macOS's /usr/bin/time -l (#355)

The evaluation command timed each downstream script with `/usr/bin/time
-l`, a BSD flag GNU time rejects with exit 125, so on the linux CI runners
a failing stub could not surface its own exit code
(test_downstream_continues_after_a_failing_stub: 125 != 7). The timed
form now runs the script in a child interpreter through a shim that
executes it in-process and reports the child's own peak resident set
(bytes on every platform) on stderr at exit; wall time is measured by the
caller and the exit code is the script's own. The marker line is stripped
from the recorded stderr tail. Same receipt shape on macOS and linux.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

UK small national default: L0-selected ~50k-household file from the 535k pool

1 participant

@juaristi22