Track the source series when building the lagged CPI and earnings

[vahid-ahmadi]

lag_cpi.py and lag_average_earnings.py both built their series with a hardcoded range(..., 2030), so values stopped at 2029. A parameter carries its final value forward rather than raising, so nothing failed past that point — the series just held a stale growth rate, and create_economic_assumption_indices compounded that stale rate into the index.

Spotted by @edward-mcpherson.

CPI: harmless today

The OBR CPI series is flat at 2% from 2027, so the carried-forward value is exactly what the loop would have produced. Patching the old range to 2045 and diffing yoy_growth.obr.lagged_cpi, indices.obr.lagged_cpi and gov.benefit_uprating_cpi across 2024–2050 gives zero differing years. Bumping the number would have been a no-op.

Average earnings: already wrong

Earnings growth is not flat — it ramps from 0.021 in 2028 to 0.0383 by 2036 — so the lagged series stayed at 0.021 and the index diverged from 2030:

indices.obr.lagged_average_earnings	before	after
2029	1.34884	1.34884
2030	1.37717	1.37851
2035	1.52798	1.62829
2039	1.66043	1.89190

About 14% low by 2039.

No modelled outcome moves. The only consumer is housing_service_charges, and nothing consumes that in turn. Running three households at 2039 and computing every year-period variable, exactly one changes — housing_service_charges itself — and household_net_income is bit-identical. This is a correctness fix to an input series, not a change in results.

The fix

The end year is derived from the source series instead of being written down a second time. Start years are unchanged (2010 for CPI, 2022 for earnings), so historic values are untouched — deriving the start too would have pulled earnings back to 2010 and moved the index base, which is a bigger change than the bug warrants.

Two smaller things in the same area:

• The lagged earnings Parameter was constructed as gov.economic_assumptions.yoy_growth.lagged_average_earnings, missing the .obr, while being added as a child of obr. The mismatch propagated into the mirrored index node. Harmless for tree access, but .name is what gets exported to metadata and what a name-keyed reform would target.
• docs/book/validation/hbai.md still said service charges were uprated by CPI.

Known limitation, not addressed here

create_economic_assumption_indices loops range(start_year + 1, 2040), so every index flatlines at 2039 regardless of how far its source runs. Worth a separate look.

Tests

test_lagged_series.py asserts each lagged series reaches its source's last year, equals the source lagged one year at 2030/2035/2039, that the earnings index keeps growing, and that the parameter names match their position in the tree.

Full suite: 1,140 YAML policy tests, 211 pytest, ruff clean.

[juaristi22]

Program review

Base repository: PolicyEngine/policyengine-uk
PR number: 1851
Reviewed head SHA: 93b1a0e
Merge base SHA: 5e57717
Mode: full
Scope: changed behavior and affected dependencies
Source manifest: /private/tmp/policyengine-command-runs/f465a97d9d1a/pr-1851-review-sources.json
Review status: PARTIAL

Source Documents

• pr-1851-source-obr-efo-march-2026.html — cached original.
• pr-1851-source-obr-efo-economy.xlsx — cached original.
• pr-1851-source-obr-long-term-determinants.xlsx — cached original.
• pr-1851-source-obr-potential-output.html — cached original.
• pr-1851-source-boe-inflation.html — cached original.

Critical

None.

Should Address

A1 — Assert the terminal lag year and value exactly (OPEN)

Severity: SHOULD ADDRESS. Location: policyengine_uk/tests/test_lagged_series.py:20-48, especially line 30. Trigger: the current sources end at 2073-01-01; a one-year lag must therefore end at 2074-01-01 with the source's 2073 value. The implementation does this, and the focused diagnostic observed CPI/earnings terminals of 2074 with values 0.0200/0.0383. However, the committed test only requires the lagged last year to be >= 2073 and checks values through 2039, so an off-by-one implementation ending in 2073 would still pass because PolicyEngine carries its terminal value forward. Assert last(lagged) == last(source) + 1 and the terminal lagged/source equality. This is boundary-coverage risk in the new helper/test contract, not an observed runtime mismatch; it could allow a future source-horizon regression to restore a stale final lag unnoticed.

Suggestions

S1 — State or enforce the annual-January source contract (OPEN)

Severity: SUGGESTION. Location: policyengine_uk/parameters/gov/economic_assumptions/lagged_series.py:4-35. Trigger: a source containing a non-January instant, such as a July update. An exact one-year lag would preserve that transition at July of the following year, but lag_source_years retains only the maximum four-digit year and add_lagged_parameter emits January instants queried by whole year. That would silently discard the subannual transition. Both current sources contain only January 1 instants (verified across all entries), so this does not produce a current modeled error. Narrowing the helper/docstring to annual January series or validating that invariant would make its new generic-looking API match its actual behavior.

Evidence Gaps

• The newly activated 2031–2038 calendar-year average-earnings values could not be reconciled to the current OBR financial-year workbook and long-run assumption. Document and verify the conversion/version used, or update the values; see gap-policy-1 in pr-1851-review-policy.md.
• Role checks incomplete: pr-1851-review-policy.md; see recovered report.

Notes

• No PDF was needed: the policy reviewer checked official OBR HTML/XLSX evidence.
• The unchanged economic-assumption index builder still ends at 2039; this known limitation is outside the changed behavior.
• The policy source-acquisition and initial workbook-inspection stage lasted 6m23s, exceeding the five-minute source-budget target by 83 seconds; individual network calls remained under one second.

Validation Summary

Head 93b1a0e: remote CI passed Test, Lint, docs, and smoke imports on Python 3.11–3.14. Local combined bounded pytest run passed 10 tests in 5.94s (wrapper 6.98s); focused diagnostic passed and imported from the detached snapshot. Policy review validated 5 official originals (3 HTML, 2 XLSX), with 0 PDFs/renders and 1 unresolved material reconciliation gap. Code role 9m51s; policy role 11m38s.

Timing

Not measured.

Review Severity

COMMENT. Open findings: 0 critical, 1 should address, 1 suggestions.

[vahid-ahmadi]

Thanks — A1 is a fair catch and it's now fixed in the latest commit.

A1. You're right that the old assertion was too weak: _last_year(lagged) >= _last_year(source) plus value checks through 2039 would have passed an implementation that stopped a year early, precisely because the terminal value carries forward. It now pins both the year and the value:

assert _last_year(lagged) == last_source_year + 1
assert lagged(str(last_source_year + 1)) == source(str(last_source_year))

Confirmed it catches the case: changing the helper's + 2 to + 1 fails the test, and the old assertion passed that same mutation.

On the unresolved reconciliation. The 2031–2038 average earnings values aren't introduced by this PR — they're the existing gov.economic_assumptions.yoy_growth.obr.average_earnings series, which the diff doesn't touch. What changed is that they now reach the lagged series instead of being masked by it freezing at 0.021, so they matter where previously they didn't.

That makes the reconciliation question real but separate: if those values don't match the current OBR workbook, that's a defect in the source parameter and it affects everything uprated off average earnings, not just the lag. Happy to open an issue for it — I'd rather not fold a source-data revision into a fix for the freezing, since they'd want different reviewers and different evidence.

Suites: 1,140 YAML policy tests, 215 pytest, ruff clean.