Surface Pydantic ValidationError to LLM in tool arg validation#1862
Open
idryzhov wants to merge 1 commit into
Open
Surface Pydantic ValidationError to LLM in tool arg validation#1862idryzhov wants to merge 1 commit into
idryzhov wants to merge 1 commit into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Python SDK’s define_tool wrapper to surface Pydantic ValidationError details to the model in a structured, actionable form (derived from each error’s loc + msg) rather than the generic redacted failure text, and adds unit tests to lock in the behavior.
Changes:
- Add a dedicated
ValidationErrorcatch indefine_tool’swrapped_handlerto build an “Invalid tool arguments:” message from Pydantic error entries. - Add unit tests ensuring validator failures and
extra="forbid"errors include the relevant field/key in the model-facing text.
Show a summary per file
| File | Description |
|---|---|
| python/copilot/tools.py | Catch pydantic.ValidationError in tool invocation wrapper and format a model-facing “Invalid tool arguments” message. |
| python/test_tools.py | Add tests verifying validation failures are surfaced to the model and include field names (including extra-field loc). |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Low
idryzhov
force-pushed
the
idryzhov-surface-pydantic-validation-error
branch
from
ecb8e1e to
bce6b79
Compare
Tool-argument validation failures raised as pydantic ValidationError are now returned to the model as a clean, actionable message built from each error's loc and msg, instead of the generic redacted text. All other exceptions stay fully redacted. The ValidationError handler is scoped to only the ptype.model_validate(args) call that deserializes the tool arguments, so a ValidationError raised from within a handler body is not surfaced and stays redacted by the broad fallback like any other exception. Safe to surface: the invalid values are arguments the LLM itself supplied, and validator messages are authored by tool developers. The user-facing text is assembled from only loc + msg; the full str(exc) (including raw input) is kept in the debug-only error field. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
idryzhov
force-pushed
the
idryzhov-surface-pydantic-validation-error
branch
from
bce6b79 to
044263a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tool-argument validation failures raised as pydantic ValidationError are now returned to the model as a clean, actionable message built from each error's loc and msg, instead of the generic redacted text. All other exceptions stay fully redacted.
The except branch lives here because tool arguments are deserialized via ptype.model_validate(args) just above in the same try block, so this is where the ValidationError originates and is the right place to catch it, ahead of the generic redaction fallback.
Safe to surface: the invalid values are arguments the LLM itself supplied, and validator messages are authored by tool developers. The user-facing text is assembled from only loc + msg; the full str(exc) (including raw input) is kept in the debug-only error field.