Return tool-call images instead of dropping them - #1963
Open
sujeito-operator wants to merge 1 commit into
Open
Conversation
decodeToolCallResponsePayload kept only inputText items and fell back to "OK", so an image-only plugin tool result reached the model as the word "OK" with the image discarded. Decode inputImage data URLs into images and render them as tool result blocks in the claude-code, pi and acp bridges.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
decodeToolCallResponsePayload(packages/provider-bridge-protocol/src/bridge-kit/bridge-tool-calls.ts) parsed the runtime'sitem/tool/callresult, then kept onlyinputTextitems and joined them.inputImageitems were filtered out and never looked at again, so a tool result carrying only an image produced an empty string, which thetext || "OK"fallback turned into the literal"OK".The server side is correct:
normalizeAgentToolResult(apps/server/src/services/plugins/plugin-service.ts) already converts a plugin's{ type: "image", data, mimeType }into{ type: "inputImage", imageUrl: "data:<mime>;base64,<data>" }. The image reaches the bridge and is discarded there.That decoder is shared, so the drop is in the claude-code, pi and acp bridges.
codexdoes not use it and is unaffected.This is the root cause of #1762:
browser_screenshotreturns the stringOKand no image. It matches the reproduction report — the capture works and nothing is written to disk because nothing is meant to be; the PNG is inline in the tool result and is dropped on the way back.What changed
decodeToolCallResponsePayloadnow returns{ content, images, isError }.inputImageitems whose URL isdata:<mime>;base64,<data>are split into{ data, mimeType }; any other URL is kept as text, because both tool-result contracts carry inline base64 and have nowhere to put a remote reference."OK"is now the fallback only when there is neither text nor image — an image-only result no longer gets relabelled as a text result.buildBridgeToolCallContentrenders a decoded payload as result blocks. MCP'sCallToolResult.content(claude-code and acp) and pi'sAgentToolResult.contentdeclare the same two members with the same field names, so one builder serves all three. Empty text is dropped so an image-only result carries the image alone.plugins/provider-claude-code/src/bridge/tool-proxy-mcp.ts,packages/agent-runtime/src/pi/bridge/tool-proxy.ts, and the ACP path (plugins/provider-acp/src/bridge/bridge.ts→tool-proxy-mcp.ts).imagesas a new field. Its schema defaults rather than requiring it: the MCP half is re-executed from the packaged artifact, so a missing key degrades to a text result instead of throwing. The existing long-tool-call test still sends a response with noimageskey, which pins that path.BridgeToolCallResult.imagesis optional because the tracker's two failure paths (sender throws, scope-wide resolution) have no image to report and never call the decoder.No
HOST_DAEMON_PROTOCOL_VERSIONbump: nothing on the server ↔ host-daemon wire changes.inputImagewas already inproviderToolCallResponseSchema,bridge-requests.tsandprovider-types.ts, and the server already emits it. Only the bridge-side decoding of an existing field changes.How you verified
packages/provider-bridge-protocol/src/bridge-kit/bridge-tool-calls.test.ts(13 cases). Fails before, passes after: reverting only the decoder body to the old text-only filter fails 4 of them — image-only, mixed text+image, non-data URL, and empty-payload data URL — while the remaining 84 tests in the package stay green.plugins/provider-claude-code/src/bridge/__tests__/tool-proxy-mcp.test.tsdrivesbuildBridgeMcpServeroverInMemoryTransport, andplugins/provider-acp/src/bridge/tool-proxy-mcp.test.tsdrives the packaged--mcp-stdioserver over stdio with the SDK client. Both assert the client receives{ type: "image", data, mimeType }.pending-tool-call-tracker.test.ts(success paths gainimages: []; the JSON-RPC-error path does not, since it never reaches the decoder) andbridge.test.tsin provider-acp.pnpm exec turbo run typecheckandtestacross@bb/provider-bridge-protocol,@get-bb/plugin-sdk,@bb/agent-runtime,bb-plugin-provider-claude-code,bb-plugin-provider-acp: typecheck 9/9 tasks green, tests 840 passed, 0 failed.prettier --checkclean on every touched file.What I could not verify here, stated because the reader should not assume otherwise: this box has no browser plugin and no macOS, so I did not run
browser_screenshotend to end against a live page. The evidence above is unit and MCP-transport level. The one behavioural claim I am not able to close is that the image, once returned, renders as expected in each provider's UI.Not related to #1503, which is a draft POC on the inbound
promptpath (packages/agent-runtime/src/claude-code/bridge/*) — images going to the provider. This is the outbound tool-result path and touches none of those files.Fixes #1762