Add results from Claude eval of variants

[adelavega]

Don't necessarily need to commit all this.

These are Claude's results of evaluating the various variations of extraction pipelines — stages 2
and 3 against neuroimaging-study-extraction.yaml, on 20 PMC OA papers + 40 neurometabench papers
with NiMADS gold. ~1,400 calls, $8.86, all gpt-5.6-luna.

Details: RESULTS.md · EXPRESSIVITY-GAPS.md ·
EVAL-PLAN.md (accuracy eval, not yet run).

Recommendation

linked mode, evidence excluded from the schema pass, low reasoning on both passes
(provisional — see caveat). $0.0084/paper for stages 2–3, ~$0.015 for all four.

Why that shape — six variants, 60 papers each

shape	unparseable	analysis recall	analyses/paper (gold 3.4)	$/paper
single call, schema field order	1	5%	2.7	$0.0077
single call, analyses first	4	8%	3.6	$0.0082
two calls, pass 2 discovers analyses	1	10%	8.6	$0.0097
two calls, pass 2 annotates stage 1's list	6	94%	2.8	$0.0110
same, evidence excluded	0	98%	3.6	$0.0084

• Single-pass drops analyses. 11 of 59 papers (19%) returned no analyses at all, with
  finish_reason: stop — not truncation. It's 12th of 14 in Study, behind 31-field Group.
  Reordering helps (19%→5%) but creates 97 dangling local_id refs.
• Unguided discovery over-generates — 8.6 analyses against a gold of 3.5, 10% matched. Stage 1
  is load-bearing; treat a stage-1 regression as a stage-2 outage.
• Evidence inline degrades the extraction and costs more: removing it gave unparseable 6→0,
  recall 94→98%, invented fields 21→0, fill 45→48%, cost $0.0110→$0.0084. Evidence is 57% of
  output tokens; only 24% of emitted tokens are content.

Reasoning effort changed nothing measurable, but that isn't the same as no effect. 2×2 on
effort, 60 papers/cell: priority-0 fill was 48% in all four; recall 98→100%, dangling refs
4/60→2/60, at 1.7–2.3× cost. Fill rate is coverage, not correctness — better values at the same
fill rate would be invisible here. Deciding this needs the human field ratings, and the
adjudication app already shows all four cells per field, so that comparison comes free with the
review. Low effort is a cost-based default for now, not a result.

Constrained decoding is impossible as written. Strict mode caps $ref nesting at 5; this
schema hits 13, and ExtractedValue → Evidence → EvidenceSet → EvidenceSpan is 6 on its own. So
json_object + post-hoc validation. If we want constrained decoding, the evidence wrapper has to
move.

Schema findings

83% of papers (50/60) hit something the schema couldn't hold — mined free from the schema's own
model_representation_notes. But 10 of 15 gaps are guidance problems, where the field exists
and wasn't used (all verified present in the YAML):

• Task/covariate/nuisance regressors — 28 papers → Analysis.terms with role: nuisance.
  Biggest gap, pure prompt fix.
• HRF details — 12 → ModelEstimation.hrf_model · filtering/autocorrelation — 10 →
  ModelEstimation.model_settings

Real gaps: structured factorial design and factor scope — 9 papers (no
field encodes a factor's within/between scope, levels, crossing); then masking (4), connectivity
lag settings (2), bootstrap metadata (1), DCM architecture (1).

14% of filled values are value_source: generated — Term.type, Term.role,
Term.variation_level, Condition.condition_kind, Assessment.assessment_type, Group.species.
These cannot be verbatim; no paper says "variation_level is within_subject." Worth saying so in
the field descriptions, since evidence spans can never validate them. Separately,
Analysis.mediation_path is the only one of 128 fields never filled — scope is good, the problem
is distortion.

Not established

• Field-level accuracy is unmeasured. All of the above is shape, enumeration and conformance.
  build_field_review.py builds the adjudication app: 3,650 priority-0 fields, 1,871 with
  cross-cell disagreement, ~6 h of review.
• Whether reasoning affects value quality is open, per above.
• No gold exists in the extraction schema's format; linked recall is partly true by construction
  (stage 1's names are the input). The valid comparison is between linked arms.
• Stage 4 (evidence pass) never run; the 95–97% span figure is from evidence emitted inline.
• Stage 1's 33/33 was on gpt-5-mini, not luna.

[adelavega]

Update — end-to-end pipeline, and an adjudication tool that reviews the end of it

Stage 3 — LLM evidence, replacing string matching

The old app picked quotes by first occurrence in the document: Term.name = "ghrelin" matched
the title line, and 88% of excerpt windows started mid-sentence. It was built to answer "does this
string exist" and was being used for "which sentence justifies this".

Stage 4 — normalization, and the answer to "does verbatim drift matter?"

normalize.py + vocab.py. Deterministic layer first
(the map's curated value_mappings, case-folding, canonical-as-key, gloss stripping), model second
with each value carrying its stage-3 evidence sentence rather than the whole paper.

	share
deterministic	66.6%
model	29.1%
unmapped	4.3% (8 on hard enums)

The headline result: running all four effort cells through normalization and re-applying the
app's difference rule —

	disagree across 4 cells
verbatim	1,887 / 3,330 (57%)
normalized	858 / 3,330 (26%)

Normalization removes 55% of cross-cell disagreements (65% on the 3,047 instances where every
cell mapped). Most of what a reviewer was being asked to adjudicate was wording, not substance.

Per-field collapse: ModelEstimation.model_type 113 distinct → 5, ConditionTerm.role
51 → 4, GroupTerm.role 25 → 4, Condition.condition_kind 48 → 10.

The deterministic layer was most of the win, and it belongs in the map

Seed coverage went 27% → 67% on two observations, neither of which needed a model:

• The map lists year -> years but not its own output years. 78 of 83 real values are literally
  "years", so that field went 4.8% → 98.8%.
• ConditionTerm.role and GroupTerm.role aren't in the map at all, yet the extractor already
  emits a near-canonical set (greater side ×139, lesser side ×135).

These wins currently live in bench/vocab.py, not in extraction-to-storage.map.yaml where
they belong. Six normalization targets are likewise my proposals rather than the map's:
spatial_scope, spatial_unit, roi_definition, Statistic.family, GroupTerm.role,
ConditionTerm.role — all have storage enums and abundant data, and the map carries a PENDING
comment admitting they have no derivation. They work; nobody has agreed them.

The review app now judges the end of the pipeline

build_field_review.py, rebuilt around the paper rather than a flat
field queue: title + abstract + links, then the whole study schema at once, then each analysis
one at a time beside the table rows it came from (match_rows resolves those for 88% of stage-1
analyses).

Normalized value is the judged value where a vocabulary applies, with verbatim and the evidence
sentence above as context; free-text fields still judge verbatim and say so. When a normalized
field is marked wrong, x/m attribute the fault to extraction or normalization —
different fixes, different stages. Variants are compared on normalized values.

Supporting: field_walk.py, paper_context.py
(title + abstract for 60/60 papers), score_fields.py,
score_normalization.py.

[adelavega]

Next steps

1. Adjudicate 12 papers (~1.4 h). field-verdicts.json does not exist — zero human verdicts
have been collected. Every accuracy claim in this PR is about shape, coverage, conformance and
self-consistency, never about whether the values are right. This single step turns most of the
"unmeasured" items below into numbers.

2. Spot-read 10 evidence sentences (10 min). 81.9% present says a sentence was returned, not
that it was the right one. My automatic proxy sits at 55% and would pass a wrong-but-plausible
sentence. This validates the stage everything downstream now depends on.

3. Fix the nuisance-regressor prompt. The biggest expressivity gap — 28 of 60 papers — is
task/covariate/nuisance regressors, and Analysis.terms with role: nuisance already exists. One
prompt edit. See EXPRESSIVITY-GAPS.md: 10 of 15 clustered gaps are
guidance problems, not schema gaps.

4. Move the deterministic mappings into the map and decide on the six proposed targets.

5. Then the vocabulary additions the unmapped list is asking for — ModelEstimation.level
cannot express "first-level and group-level" (a single-valued slot against a two-valued fact, ~30%
of that field's misses); Condition.condition_kind has active_comparator but no bare
comparator.

Gaps worth naming

• Recall is entirely unmeasured. Adjudicating output finds wrong values, never omissions. Needs
  the blind tier, still deferred.
• Two papers silently produced nothing — nmb_11050024 has 90,000 chars and 0 usable
  sentences; nmb_29428771 has 1 scalar priority-0 field. Both are the ACE-HTML layout (5 of 60
  papers). Neither stage warned. They should fail loudly on an empty pool.
• service_tier="flex" is still not sent by the extraction runs, so every cost figure is a
  projection from measured token counts rather than a billed number.
• Dangling local_ids in 4/60 papers — fix identified (emit null when the digest offers
  nothing), not applied.
• Paper-first prompt ordering would make the paper cacheable across all four stages; measured
  worth ~$31/20k papers, and free to do.
• Nothing in bench/ runs on another machine. sources.py and build_field_review.py
  hard-code /home/zorro/repos/{analysis-schema,autonima-results} and a session scratchpad path.
• README.md still documents ten files that don't exist, including extraction-readme.md —
  cited as the home of the rules the extraction prompt was built without.