Skip to content

Return tool-call images instead of dropping them - #1963

Open
sujeito-operator wants to merge 1 commit into
get-bb:mainfrom
sujeito-operator:fix/tool-call-image-results
Open

Return tool-call images instead of dropping them#1963
sujeito-operator wants to merge 1 commit into
get-bb:mainfrom
sujeito-operator:fix/tool-call-image-results

Conversation

@sujeito-operator

Copy link
Copy Markdown

What was wrong

decodeToolCallResponsePayload (packages/provider-bridge-protocol/src/bridge-kit/bridge-tool-calls.ts) parsed the runtime's item/tool/call result, then kept only inputText items and joined them. inputImage items were filtered out and never looked at again, so a tool result carrying only an image produced an empty string, which the text || "OK" fallback turned into the literal "OK".

The server side is correct: normalizeAgentToolResult (apps/server/src/services/plugins/plugin-service.ts) already converts a plugin's { type: "image", data, mimeType } into { type: "inputImage", imageUrl: "data:<mime>;base64,<data>" }. The image reaches the bridge and is discarded there.

That decoder is shared, so the drop is in the claude-code, pi and acp bridges. codex does not use it and is unaffected.

This is the root cause of #1762: browser_screenshot returns the string OK and no image. It matches the reproduction report — the capture works and nothing is written to disk because nothing is meant to be; the PNG is inline in the tool result and is dropped on the way back.

What changed

  • decodeToolCallResponsePayload now returns { content, images, isError }. inputImage items whose URL is data:<mime>;base64,<data> are split into { data, mimeType }; any other URL is kept as text, because both tool-result contracts carry inline base64 and have nowhere to put a remote reference. "OK" is now the fallback only when there is neither text nor image — an image-only result no longer gets relabelled as a text result.
  • New buildBridgeToolCallContent renders a decoded payload as result blocks. MCP's CallToolResult.content (claude-code and acp) and pi's AgentToolResult.content declare the same two members with the same field names, so one builder serves all three. Empty text is dropped so an image-only result carries the image alone.
  • Three call sites use it: plugins/provider-claude-code/src/bridge/tool-proxy-mcp.ts, packages/agent-runtime/src/pi/bridge/tool-proxy.ts, and the ACP path (plugins/provider-acp/src/bridge/bridge.tstool-proxy-mcp.ts).
  • The ACP bridge's dynamic-tool socket carries images as a new field. Its schema defaults rather than requiring it: the MCP half is re-executed from the packaged artifact, so a missing key degrades to a text result instead of throwing. The existing long-tool-call test still sends a response with no images key, which pins that path.
  • BridgeToolCallResult.images is optional because the tracker's two failure paths (sender throws, scope-wide resolution) have no image to report and never call the decoder.

No HOST_DAEMON_PROTOCOL_VERSION bump: nothing on the server ↔ host-daemon wire changes. inputImage was already in providerToolCallResponseSchema, bridge-requests.ts and provider-types.ts, and the server already emits it. Only the bridge-side decoding of an existing field changes.

How you verified

  • New packages/provider-bridge-protocol/src/bridge-kit/bridge-tool-calls.test.ts (13 cases). Fails before, passes after: reverting only the decoder body to the old text-only filter fails 4 of them — image-only, mixed text+image, non-data URL, and empty-payload data URL — while the remaining 84 tests in the package stay green.
  • End-to-end through a real MCP client at two bridges: plugins/provider-claude-code/src/bridge/__tests__/tool-proxy-mcp.test.ts drives buildBridgeMcpServer over InMemoryTransport, and plugins/provider-acp/src/bridge/tool-proxy-mcp.test.ts drives the packaged --mcp-stdio server over stdio with the SDK client. Both assert the client receives { type: "image", data, mimeType }.
  • Existing assertions that named the decoder's return shape were updated: pending-tool-call-tracker.test.ts (success paths gain images: []; the JSON-RPC-error path does not, since it never reaches the decoder) and bridge.test.ts in provider-acp.
  • pnpm exec turbo run typecheck and test across @bb/provider-bridge-protocol, @get-bb/plugin-sdk, @bb/agent-runtime, bb-plugin-provider-claude-code, bb-plugin-provider-acp: typecheck 9/9 tasks green, tests 840 passed, 0 failed. prettier --check clean on every touched file.

What I could not verify here, stated because the reader should not assume otherwise: this box has no browser plugin and no macOS, so I did not run browser_screenshot end to end against a live page. The evidence above is unit and MCP-transport level. The one behavioural claim I am not able to close is that the image, once returned, renders as expected in each provider's UI.

Not related to #1503, which is a draft POC on the inbound prompt path (packages/agent-runtime/src/claude-code/bridge/*) — images going to the provider. This is the outbound tool-result path and touches none of those files.

Fixes #1762

AGENT GENERATED: by Claude Opus 5

decodeToolCallResponsePayload kept only inputText items and fell back to
"OK", so an image-only plugin tool result reached the model as the word
"OK" with the image discarded. Decode inputImage data URLs into images
and render them as tool result blocks in the claude-code, pi and acp
bridges.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

browser_screenshot returns the string "OK" and no image; the PNG is captured then dropped

1 participant