Skip to content

fix(openai): move tool images into user messages - #2923

Merged
wgqqqqq merged 1 commit into
GCWing:mainfrom
wgqqqqq:wgq/fix-chat-completions-tool-images
Sep 9, 2026
Merged

fix(openai): move tool images into user messages#2923
wgqqqqq merged 1 commit into
GCWing:mainfrom
wgqqqqq:wgq/fix-chat-completions-tool-images

Conversation

@wgqqqqq

@wgqqqqq wgqqqqq commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Keep Chat Completions tool results as strings and move their image attachments into a following user message. Emit that message after the complete contiguous batch of tool results, preserving tool-call associations and image order.

Fixes #2888

Type and Areas

Type: Bug fix

Areas: AI adapters (OpenAI Chat Completions)

Motivation / Impact

A view_image result previously serialized image_url parts inside role: "tool". Strict compatible endpoints reject this shape, which can also break subsequent requests that reuse the history. The updated conversion keeps the tool text and IDs intact while providing the images through supported user content. Tool text that resembles a JSON content-parts array remains a string.

Verification

  • pnpm run fmt:rs and git diff --check: passed.
  • cargo test -p openbitfun-ai-adapters --lib providers::openai::message_converter::tests: 22 passed, including image ordering across parallel tool results, legacy DTO round trips, error results, and unchanged text-only batches.
  • cargo test -p openbitfun-ai-adapters --lib providers::anthropic::message_converter::tests: 3 passed.
  • cargo build -p openbitfun-desktop: passed.
  • Actual macOS desktop E2E through the OpenBitFun forwarding endpoint: reproduced a pre-fix GLM 422 rejection of tool content. With the fix, GLM and GPT-4o mini each completed view_image and a follow-up using the same image history, correctly identifying the red/blue fixture. Recorded requests confirmed string tool content and images only in user messages.
  • Official API regression using payloads exported from the fixed Rust converters: Zhipu GLM-5.3-Flash Chat Completions and DeepSeek deepseek-v4-flash-vision-exp Chat Completions, Anthropic Messages, and Responses each passed both the image request and its history continuation (HTTP 200, correct image answers).

Reviewer Notes

  • Anthropic and Responses conversion are unchanged. No persisted schema changes or migration are required.
  • Zhipu's Anthropic endpoint could not be validated because the account returned an expired Coding Plan error (429/1309).
  • Windows, the reporter's exact forwarding endpoint, and remote workspace/control, Peer Device Mode, and Detached Dispatch were not exercised. Image restoration after a cold restart is outside this verification; visual history checks used the running session.
  • AI-assisted implementation; tested with focused unit tests, live API regression, and local desktop E2E as detailed above. Platform and remote coverage remains limited.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable (no UI/localization changes required).

@wgqqqqq
wgqqqqq merged commit f89b2d7 into GCWing:main Sep 9, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: view_image 工具结果的图片被序列化进 tool 消息,OpenAI 兼容端点(z-ai/GLM)400 且会话持续失败

1 participant