Derive the report fields the model kept missing - #32
Merged
Merged
Conversation
A response was refused when its improvement plan was out of order or its phase tags disagreed with that plan, and both of those follow from the plan itself, so an assessment the model had otherwise earned reached the candidate as no evaluation. The server derives them now and the model answers only for the mapping it alone knows. The call around it was measured rather than guessed: attempts were cancelled under the model's own median latency, thinking was spending the output budget the report had to fit in, and the deadline funded fewer calls than the budget handed out. Separately, the browser bounded a summary the agent validates at 1200 characters to 300, so every candidate read one that stopped mid-sentence.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A report was refused whenever the model returned an improvement plan in the wrong order, or phase tags that disagreed with that plan. Both of those follow from the plan itself, so the server derives them now and the model answers only for the mapping it alone knows, which feedback improvement each plan item addresses. What used to reach the candidate as "no evaluation" is an assessment they had earned.
The call around it was measured rather than guessed. Generation lands in six seconds at the median and past twelve at the tail, so the eight-second attempt timeout was cancelling healthy calls; thinking tokens come out of maxOutputTokens, so a thinking response arrives as truncated JSON; and the call budget was a product of two guesses, which let a report die with four of its six calls unspent after two consecutive 503s. Attempts get twenty seconds, the thinking budget is pinned to zero, and the pool is one number the deadline can pay for, spent by whichever loop needs it.
Separately, sanitizeReport bounded the report summary at the generic 300-character field limit while the agent validates it at 1200, and real summaries run 400 to 450 characters, so every candidate has been reading a committee summary that stops mid-sentence. Both halves are named constants now, held together by a test in the style this repository already uses for the integrity bound.
Verified with scripts/test.sh green and 74 live report generations against gemini-3.1-flash-lite across three transcript shapes: a near-silent candidate, an ordinary session, and a 92KB transcript. No failures. Two runs in that set hit consecutive transport failures and recovered on a later call, which is the case the old fixed retry allotment lost.
Not addressed here: the ten-minute wait in #31 is far outside these bounds and nothing in this branch explains it.
Refs #31
Summary by cubic
Reports that were otherwise valid no longer become “no evaluation” when Gemini returns improvement-plan items out of order or phase tags inconsistent with that plan. The server now normalizes those derived fields and preserves the assessment the candidate earned.
Bug Fixes
Written for commit 51f4a12. Summary will update on new commits.