Use canonical SPM measurements and immutable population default

[MaxGhenis]

The country model now uses the canonical calculator artifact for SPM measurement, with observed county assignment by default and an explicit national or fixed-area selection. It preserves country tax/benefit formulas, applies the housing cap to the canonical amount before final storage conversion, and binds cloned/reform/baseline holders to their own policy variables.

Microsimulation() defaults to:

hf://datasets/policyengine/populace-us/populace_us_2024.h5@populace-us-2024-spm-20260909

That immutable tag is published. A real download with no offline override resolves it to a file whose SHA256 is 6496cc4393d4d3c6574f76eca231de5898c803b9067645591fd5c4d3e65aee84, matching the published H5 hash. Readback confirms 57,240 households, household.county_fips stored as five-digit strings, and person.is_spm_independent_minor_role present. The no-argument Microsimulation() tests therefore run for real; none of them is guarded or skipped.

County measurement requires observed county_fips; state-only requests can select national measurement explicitly with spm={"geography_kind": "national"}. A request with no usable county raises a country-level SPM_GEOGRAPHY_REQUIRED naming the two fixes a caller has — send county_fips as a five-digit string, or select geography_kind="national" — and that now covers integers, pandas missing values and codes that lost a leading zero, which the calculator alone would have reported as an unavailable county assignment. An unrecognised five-digit code still raises SPM_GEOGRAPHY_UNAVAILABLE. Unsupported measurement years raise SPM_YEAR_UNAVAILABLE; the artifact covers 2022–2035. Tax-only requests remain lazy. There is no consumer extrapolation or implicit geographic fallback.

An unpublished or renamed dataset build id now surfaces as a country-level FileNotFoundError naming the URI, with the Hugging Face error as its cause, instead of a transport error that never says which dataset the model was asked for.

Population data must supply the source-backed is_spm_independent_minor_role primitive and observed county inputs. Formula-owned stored SPM outputs are rejected; source observations belong in report-only columns. The legacy policyengine-us-data CPS files predate that contract and cannot produce resource or poverty outputs:

• cps_2023.h5 stores the formula-owned spm_unit_spm_threshold column, so the loader rejects it at construction. It also stores county FIPS as the CPS within-state integer code.
• enhanced_cps_2024.h5 loads and computes tax variables, and its county column does hold five-digit FIPS codes, but 18 of its 43,134 SPM units are a lone 15-to-17-year-old with no source-backed independence role, so those units classify no SPM measurement adult.

Both failures are now asserted as contract tests. The society-wide coverage those files used to carry — net income across the extended years, in-range income deciles, nonzero earnings — moved onto the shipped default build. The future-dated reform regression (#9075) runs two simulations over one 500-household slice of the shipped file, with a non-vacuity guard proving the reform reaches that population in its first year.

A simulation no longer clones policy state when its policy is the shipped policy. Core's TaxBenefitSystem.clone() rebuilds the whole parameter tree node by node and empties its at-instant caches, and this country deep-copies every one of its 6,157 variable objects on top. Three paths were paying that per simulation, and none of them has any policy of its own to isolate: no reform and no supplied system; a supplied system with no reform, which is how a caller that builds one system and runs many households through it works, including the household API and much of this suite; and the baseline branch core builds for a policy reform. Each now shares the shipped parameter tree and variable objects and keeps only its receipts and its own variable registry private, which is safe because every core operation a reform performs on a variable rebinds variables[name] to a new object rather than mutating the registered one. A supplied system with a reform still gets the full clone, because core applies the reform set to whatever system it is handed and the caller's own system must not be the one it reforms. test_ordinary_simulation_shares_default_policy_state asserts the shared tree is reused and that nothing reaches the shared instance.

Measured on one machine, one process, warm caches. 20 sequential single-household Simulation constructions plus household_net_income (scripts/benchmark_household_simulation.py):

revision	total	of which construction	first iteration	mean of the rest
main d8fb269	6.14s	0.50s	1.64s	0.237s
reviewed head f121882	109.81s	70.37s	3.42s	5.600s
this head	4.74s	0.48s	1.24s	0.184s

Net income is bit-identical across all three revisions for that household, so the comparison is like for like.

The same clone also sat behind the reviewed head's Rest job, which hit its 60-minute timeout at 52% of the Python suite where main's Rest job takes 21m22s end to end. Two same-machine measurements of that suite:

suite	main d8fb269	reviewed head f121882	this head
174 tests in four tests/core files, itemization branching on	26.12s	545.58s	26.56s
make test-other-python, itemization branching off	4m37s (437 tests)	not run locally	5m21s (510 tests)

Receipts for this head:

• pyproject.toml version is 1.825.1 and the committed uv.lock root matches it; python .github/release_lock.py --committed exits 0.
• spm-calculator resolves to 1.0.0 from https://pypi.org/simple, sdist SHA256 a99aac8c2c0bf81a9455105bbf872366f1ea9066a99cd793714881cd3db846dc and wheel SHA256 e354937a5e1a4045d4966ed594a528d8b02866fabaac9bb5672017004b627305. The exact spm-calculator==1.0.0 pin remains.
• policyengine-core resolves to 3.30.2 in both the lock and this environment.
• The bundled forecast is spm-rolling-2026-09-09, content SHA256 3d86d5c4c0423480e6b69b75d222ffa4a7a2639e4094df5ba2504af01be17173, default scenario ce_trend, 14 years covering 2022–2035.
• The breaking changelog fragment drives the major bump to 2.0.0 from 1.825.1 through the existing version helper; no manual PR bump.

Test coverage on this head:

• The three canonical SPM modules hold 36 test functions and collect 79 cases: tests/core/test_spm_system.py (16 functions, 30 cases), tests/unit/test_spm_integration_contract.py (9, 24), and tests/policy/baseline/household/income/spm_unit/test_spm_unit_spm_threshold.py (11, 25). Eleven of those cases are new on this head, covering shared policy state, the URI-naming download error, integer county columns in a dataset, and non-string county inputs in a situation.
• 16 YAML files change, carrying 122 cases. Eleven were repaired earlier in this PR; five more are repaired here — CBO after-tax income, marginal tax rate including health benefits, the Trump dividend benefit aggregates, and CA CPUC countable income and its integration cases — each with the same zero housing subsidy input the earlier repairs use. Every repaired file passes with itemization branching both on and off.

Prepared from authoritative upstream main b8ca61a23e1c7ca9f66181dee36d5d8bee89c916, preserving its Alabama 2025 deduction correction and automated version metadata. A full-index three-way patch transferred the reviewed canonical donor on 42103db1b7d5be74bd28ebfc8e93adb2c7859079, including its fixture repairs.

Related scientific integration history: #9081 (inspected; older draft 0.4.0 proposal). Also inspected #9421, the earlier 0.5.0 Git-pin draft. This is a fresh canonical integration PR; neither prior PR is overwritten or closed.

Automatic release safeguards check the committed PyPI lock before the version bump and before publication. After the existing bump and Towncrier run, a guarded refresh permits only the root version to change; resolver failure or dependency drift restores the reviewed lock bytes. Every dependency artifact must have a canonical PyPI file URL and SHA256 hash. The guard tests, the committed-lock check and the changelog check pass in CI on this branch.

Open, needing a decision before this can go green:

• tests/policy/baseline/partners/analytics_coverage/edge_cases/state/ca/care.yaml and .../fera.yaml fail with SPM_GEOGRAPHY_REQUIRED. CA CPUC countable income includes spm_unit_capped_housing_subsidy (PG&E Form 01-9077 counts housing subsidies), so CARE and FERA now need an SPM area. Both fixtures do name their county, as county_str: LOS_ANGELES_COUNTY_CA, but the model derives geography one-directionally (county_fips -> county -> county_str), so an explicitly named county never becomes a FIPS code. Three candidate fixes, none applied: add a county_fips input to the two fixtures (a partner contract edit, gated); backfill county_fips from an explicitly supplied county the way _backfill_state_code_from_str handles state (correct, but 328 YAML test files supply county_str, 134 of them partner fixtures, so it would change county from the first-county-in-state fallback to the named county across the partner suite); or drop spm_unit_capped_housing_subsidy from the CPUC income list in favour of housing_assistance (changes partner-visible CARE/FERA amounts for assisted households). This is the same coordinated-rollout item as state-only callers losing resource and poverty outputs, and it belongs with the API and app PRs.

No merge, release, publication, heavy population rerun or partner snapshot rewrite is included in this PR.

Root additions after the fix lane (b4453be → 21d0b0d)

• Congress CI shard (b4453be): batch 6 (contrib/congress/tlaib) hung without output until CI killed it. Per-subdir batching gave the whole proposal folder one subprocess regardless of the batcher's own memory model; tlaib carries reform-combo weight 8.25 and peaked at 15.0 GB on the 16 GB runner at main (run 34637166889), and locally the batch peaks at 18.2 GB on main and 18.1 GB on this branch, so the runner had no headroom and swapped instead of failing. test_batched.py now packs any per-subdir folder whose distinct combos exceed MAX_BATCH_COMBO_WEIGHT by combo weight, pinned by tests/code_health/test_batched_per_subdir_budget.py. On this branch the shard now passes in 17 minutes with tlaib as two subprocesses peaking at 8,705 MB and 10,615 MB (run 34648331803); no other congress folder changes.
• Housing cap consults the measurement only where there is assistance to cap (21d0b0d): the cap asked the calculator for the canonical housing portion for every unit, so net income, benefits, marginal tax rates and state programs whose countable income sums the cap (CA CPUC CARE/FERA) demanded SPM geography and composition from households that never use the measurement, and two partner contract fixtures for LA households failed on that alone. No partner uses SPM outputs. spm_unit_capped_housing_subsidy now evaluates the housing portion for assisted units only through policyengine_us.spm.masked_policyengine_amount (same provider path, so composition and county requirements, typed errors and receipts attach to exactly the units whose result depends on the measurement); unassisted units are 0 with no lookup. Assisted units without a county still fail closed with SPM_GEOGRAPHY_REQUIRED; an explicit national selection computes them. The contract tests pin the new behavior, the two partner fixtures pass unmodified, and the thirteen fixture files that had been given a spm_unit_capped_housing_subsidy: 0 input return to their main-branch content (93 cases pass).

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (f9b5549) to head (3e81597).
⚠️ Report is 67 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##              main     #9428   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            2         1    -1     
  Lines           20        18    -2     
  Branches         0         1    +1     
=========================================
- Hits            20        18    -2     

Flag	Coverage Δ
unittests	100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.