.NET: Add support for file content messages in AG-UI #6951
Conversation
|
@microsoft-github-policy-service agree |
There was a problem hiding this comment.
Pull request overview
Adds first-class support in the .NET AG-UI implementation for messages whose content can be either a legacy string or a structured array of content blocks (e.g., mixed text + file/media blocks), enabling richer round-tripping between AGUIMessage and ChatMessage.
Changes:
- Introduces
AGUIMessageContent+ block/source models and a JSON converter to supportcontentas either a string or an array of blocks. - Updates AG-UI ↔ chat conversions to map block-based content to/from
AIContenttypes (e.g.,TextContent,UriContent,DataContent,HostedFileContent) and preserves tool-call batching behavior. - Extends source-generated JSON serializer registrations and adds unit tests + a sample HTTP request demonstrating block content.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| dotnet/src/Microsoft.Agents.AI.AGUI/Shared/AGUIMessageContent.cs | Adds the structured content + block/source model and string/block behavior. |
| dotnet/src/Microsoft.Agents.AI.AGUI/Shared/AGUIMessageContentJsonConverter.cs | Implements content polymorphic JSON (string vs block array). |
| dotnet/src/Microsoft.Agents.AI.AGUI/Shared/AGUIMessage.cs | Switches AGUIMessage.Content from string to AGUIMessageContent. |
| dotnet/src/Microsoft.Agents.AI.AGUI/Shared/AGUIJsonSerializerContext.cs | Registers new content types for source-generated serialization. |
| dotnet/src/Microsoft.Agents.AI.AGUI/Shared/AGUIChatMessageExtensions.cs | Adds mixed-content conversion between AG-UI messages and ChatMessage contents. |
| dotnet/tests/Microsoft.Agents.AI.AGUI.UnitTests/AGUIJsonSerializerContextTests.cs | Adds serializer tests covering block-array content and adjusts round-trip assertions. |
| dotnet/tests/Microsoft.Agents.AI.AGUI.UnitTests/AGUIChatMessageExtensionsTests.cs | Adds conversion tests for block-based content. |
| dotnet/samples/05-end-to-end/AGUIClientServer/AGUIServer/AGUIServer.http | Adds an example request demonstrating file/data block + text in one message. |
| if (string.Equals(sourceType, "url", StringComparison.OrdinalIgnoreCase) && | ||
| !string.IsNullOrWhiteSpace(sourceValue)) | ||
| { | ||
| aiContent = new UriContent(sourceValue!, mediaType); | ||
| return true; | ||
| } | ||
|
|
||
| if (string.Equals(sourceType, "data", StringComparison.OrdinalIgnoreCase) && | ||
| !string.IsNullOrWhiteSpace(sourceData)) | ||
| { | ||
| try | ||
| { | ||
| aiContent = new DataContent(Convert.FromBase64String(sourceData), mediaType); | ||
| return true; | ||
| } | ||
| catch (FormatException) | ||
| { | ||
| } | ||
| } |
| public override string ToString() => this.IsText | ||
| ? this.Text | ||
| : this.Blocks is null | ||
| ? string.Empty | ||
| : string.Concat(this.Blocks.Where(block => string.Equals(block.Type, "text", StringComparison.OrdinalIgnoreCase)).Select(block => block.Text ?? string.Empty)); |
| [Fact] | ||
| public void AsChatMessages_WithContentBlocks_ConvertsToChatContents() | ||
| { | ||
| // Arrange | ||
| const string ImageUrl = "https://example.com/image.png"; | ||
| AGUIMessageContent userContent = new( | ||
| new[] |
|
AG-UI hosting structure is under big change, see:#6653 |
rogerbarreto
left a comment
rogerbarreto
left a comment
There was a problem hiding this comment.
@ElderJames Thanks for the contribution! The direction is solid: AG-UI multimodal content maps cleanly onto the MEAI content types, and the bidirectional approach in AGUIChatMessageExtensions is the right place for it. A few things to align with the AG-UI spec and the existing Python adapter before this is ready.
Spec reference
Using the canonical ag-ui-protocol schema (sdks/python/ag_ui/core/types.py):
- Only
UserMessage.contentisUnion[str, List[InputContent]].developer,system,assistant,tool,reasoningcontent is a plain string. - Canonical media parts (
image/audio/video/document) carry onlytype,source, and optionalmetadata. The flaturl/data/id/filename/mimeTypefields exist only on the deprecatedbinarypart. sourceis{type:"data", value:<base64>, mimeType}(mimeType required) or{type:"url", value, mimeType?}.
Blocking
-
DataContentemits base64 into the blockurlfield.BuildMediaContentBlockalways setsUrl = value, and for theDataContentbranchvalueis the base64 string. The result is a block with"url": "<base64...>", which is malformed. The correct payload is already insource.value, sourlshould not be set for data sources. -
HostedFileContentemitstype:"document"withsource: null. Per specDocumentInputContent.sourceis required, so this is non-conformant. An id-only reference should use the deprecatedbinaryshape (which allowsid) or be resolved to a real source. The Python adapter maps a binary id toag-ui://binary/{id}; matching that keeps the two languages consistent.
Please consider
-
Content widened to all roles.
AGUIMessage.ContentbecomesAGUIMessageContentfor every role, but only user messages may carry block arrays. Text-only messages collapse back to a string so the common path is fine, but the outbound path can still emit arrays for non-user roles. Suggest constraining block-array emission to the user role. -
Canonical vs deprecated shape mixing. Emitting
type:"image"/"document"with extra top-levelurl/mimeType/filenameblends the canonical and deprecated-binaryshapes. The Python SDK tolerates it (extra="allow"), but stricter SDKs (TypeScript zod, Rust) may reject it. Prefer source-only media parts. -
Align with the Python reference adapter.
python/packages/ag-ui/.../_message_adapters.pyalready handles this end to end (source-type aliasesuri/base64,mime_typevsmimeType,data:URI passthrough, binary-id handling). Reusing its semantics avoids cross-language drift.
Docs
ADR 0010-ag-ui-support.md states the original scope was core text streaming with no custom content types. This PR expands the content model, so a short ADR update noting the multimodal content-block addition would be good.
Round-trip tests for DataContent and HostedFileContent (not just the url/image case) would catch issues 1 and 2.
Nice to have
A small runnable end-to-end .NET sample showing a multimodal message (for example an image or document upload flowing through to the agent) would help users adopt this. Not a blocker, and note the Python package does not have one either, so this is a suggestion rather than a parity requirement.
So, does this Pull Request still need to be pushed forward? @Kumima @rogerbarreto |
|
@ElderJames, I don't think so, as well pointed by @Kumima, I think the official package now is the source of support for AGUI, feel free to close this PR. Thanks! We will move forward bringing the change proposed in the PR below into MAF. |
|
Ok, thanks. |
Motivation & Context
This commit is required to correctly support AG-UI message payloads that contain mixed structured content, not just plain-string content.
It solves the limitation where AG-UI messages were treated as single text payloads, which prevented reliable round-trip handling for multi-part content such as text + media/data blocks.
The change enables richer message flows and serialization compatibility for scenarios involving document/image/audio/video blocks, URI/data payloads, hosted file references, and mixed content in both directions (AGUIMessage ↔ ChatMessage).
Description & Review Guide
This change introduces first-class AG-UI content-block modeling and dual-direction serialization support for mixed message contents.
What are the major changes?
AGUIMessageContent,AGUIMessageContentBlock, andAGUIMessageContentSourceinAGUIMessageContent.cs.AGUIMessageContentJsonConverterto serialize/deserializecontentas either:AGUIMessage.Contenttype fromstringtoAGUIMessageContent(AGUIMessage.cs).AGUIJsonSerializerContextregistrations to include new content-block types for source-generation serializers.AGUIChatMessageExtensionswith:AGUIMessagetoChatMessagefor block-based content,ChatMessagetoAGUIMessageproducing content blocks,TextContent,UriContent,DataContent,HostedFileContent, tool results/calls, and assistant message edge cases,url,data, hosted file id).AGUIChatMessageExtensionsTestsAGUIJsonSerializerContextTestsWhat is the impact of these changes?
What do you want reviewers to focus on?
AGUIMessage.Contenttype migration impact: verify all call paths in AGUI conversion flow expectAGUIMessageContentand not raw strings.AGUIMessageContentJsonConverter(stringvs block array parsing) for backward compatibility.AGUIChatMessageExtensionsfor:TextContent+UriContent/DataContent),image/audio/video/document/file/binary).Related Issue
Fixes #6931
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.