fix(exaforce): suppress LLM decode runaways with frequency_penalty=0.1

[will-exaforce]

Summary

• Apply frequency_penalty=0.1 to OpenAI-compatible chat models, eliminating the intermittent decode runaway that was costing whole scans their LLM findings
• Add a drift-guarded wrap_module_callable primitive to _patchlib so fork behavior can wrap upstream callables without forking them
• Override with SKILLSPECTOR_FREQUENCY_PENALTY (0 disables); no upstream-tracked file changes

Details

The bug. Under strict json_schema structured output, nemotron-super-3-120b intermittently enters a degenerate decode state and emits whitespace until it hits max_completion_tokens. JSON permits unlimited whitespace between tokens, so the grammar never forces a stop. All 7 captured samples were 98.5–100% whitespace; the worst emitted { followed by 646,475 spaces and 3,601 newlines, burning ~98s (a healthy call is 0.4s).

The call raises LengthFinishReasonError, arun_batches drops the batch with no retry, and agentguard's _llm_degraded discards the entire scan's LLM findings on the first such warning and re-runs --no-llm. Measured at ~0.8%/call across four independent runs — which, at ~3 × files calls per unit, loses roughly a third of units.

Why this value. A 5,400-call paired sweep (6 penalties × 900, prod-shaped corpus):

penalty	runaway rate	malformed rule_id
0.00	0.78%	0.00%
0.05	0.22%	0.00%
0.10	0.00%	0.00%
0.20	0.11%	0.00%
0.30	0.22%	1.05%
0.50	0.00%	1.49%

Pooled any-penalty vs none: 0.78% → 0.11%, Fisher p = 0.0013 (independently reproduced by a 3,000-call two-arm run, p = 0.0034). No single arm survives Bonferroni across 5 comparisons, so the data supports "turn it on" but does not finely resolve 0.05 vs 0.1 vs 0.2.

0.1 is the smallest value reaching zero runaways with no observed structured-output damage. Above ~0.3 the penalty mangles rule_id ('', 'SQ', 'SQP-') because it discounts already-emitted tokens and a findings response repeats rule_id/severity/start_line once per finding — so corruption scales with findings per response (0% at 1–2 findings, ~3% at 4+), i.e. it is worst on the files with the most to report.

Rejected alternatives, all measured: any system prompt (detailed thinking off is flat at 1/500 vs 1/500; instructing the model not to pad is flat-to-worse); function_calling instead of json_schema (8.9% schema-validation failures, 58× output tokens, same yield); stop sequences on whitespace runs (aborted 0 of 500).

Placement. Implemented as a guarded runtime patch under src/skillspector/exaforce/ so upstream-tracked files stay at parity. It wraps get_chat_model in both llm_utils and llm_analyzer_base — the latter binds the symbol by value at import, so patching llm_utils alone would miss every analyzer. Models that don't declare frequency_penalty (ChatAnthropic, ChatBedrockConverse, the agent-CLI adapter) are skipped rather than sent an unsupported parameter, an explicit caller value always wins, and upstream renaming or resignaturing get_chat_model raises PatchDriftError at import.

Verification. pytest: 1380 passed, 4 failed — exactly the pre-existing exaforce-by-design failures (test_to_finding, test_model_dump, test_intent_validation, test_malicious_skill_findings_preserve_metadata), no new breakage. 24 full-graph scans against bedrock-mantle: zero LengthFinishReason, vs 5-in-12 and 9-in-15 on baseline; degraded-scan rate 67% → 21%.

Not yet benchmark-validated. The 900-unit run in benchmark.db (c8c9b0d384f6) started ~5 hours before this patch existed and is described as a recheck of the baseline config, so it measures 2.3.11 → 2.4.2, not this change. A clean one-change-per-run comparison should use c8c9b0d384f6 as the base. Worth watching there: findings/call rose 13–37% in the probes, and whether that is recall or false positives can only be settled against labelled ground truth.

No structured plan — this came out of a measurement-driven investigation; probes and full writeup live in tmp/mantle-llm-repro/ (gitignored).

Benchmark

	base · Ver 2.3.11	head · recheck of a1d8b9cd9a4b
run id	a1d8b9cd9a4b	c8c9b0d384f6
description	Ver 2.3.11	recheck of a1d8b9cd9a4b
model	nvidia.nemotron-super-3-120b	nvidia.nemotron-super-3-120b
LLM	on	on
units (total)	900	900
started	2026-07-13 04:00	2026-08-11 02:53
workers	8	8

Outcomes

Every scanned unit is exactly one of these, so the rows conserve to the total —
reducing one bucket (e.g. eliminating timeouts) just moves those units into the
others. Read the two bold group rows for the verdict; the indented rows break
them down (a single sub-row moving isn't independently good or bad).

	base	head	Δ
✅ correct (TP + TN)	807	822	▲ +15
true positives · malware caught	523	537	+14
true negatives · benign passed	284	285	+1
🔴 incorrect (FP + FN + errors)	93	78	▲ -15
false positives · benign flagged	16	15	-1
false negatives · malware missed	64	63	-1
errors / timeouts	13	0	-13
total scanned	900	900	0

Timing

Per-scan run_time stats. Total scan time is the sum (total work across all
scans) — resume-safe, unlike finished_at − started_at. ~est. wall divides
that by the run's worker count to approximate elapsed time; it's an estimate
(ignores ramp-up, the under-saturated tail, and auth pauses, so it runs low).

	scans	avg	median	p95	max	total scan time	~est. wall
base	900	51.34s	10.70s	245.85s	300.01s	12h50m	1h36m
head	900	12.40s	9.27s	45.06s	144.43s	3h05m	23m14s