fix: extract user message from AG-UI multimodal user content#231
Open
IanVS wants to merge 2 commits into
Open
Conversation
…fixture bug Demonstrates how aimock's AG-UI recorder loses user message text when CopilotKit sends multimodal content (text + file attachment). The user message's content is a structured array of parts instead of a string, so extractLastUserMessage returns "" and the recorder falls back to writing match.message: "__NO_USER_MESSAGE__", making the fixture unmatchable on replay. Includes: - Next.js 15 app with CopilotChat and attachments enabled - CopilotRuntime + HttpAgent route at /api/copilotkit pointing at aimock - AGUIMock recording proxy and a tiny SSE upstream stub - A headless curl path with a synthetic AG-UI-spec-conformant payload - recorded-sample/ with a before-fix fixture exhibit Also extends .prettierignore and the eslint ignore list so examples/ build output and node_modules do not pollute repo-wide checks.
When a CopilotKit (or any @ag-ui/core-conforming) client sent a multimodal
user message — e.g. content as `[{type:"text", text:"..."}, {type:"document",
source:...}]` — extractLastUserMessage returned "" because it only handled
typeof content === "string". The recorder then keyed the fixture off a
predicate and serialized match.message: "__NO_USER_MESSAGE__", producing
fixtures that couldn't replay.
extractLastUserMessage now walks structured content arrays and joins their
text parts. Non-text parts (documents, images, audio, etc.) are skipped.
The "" return contract on missing user text is preserved, so the sentinel
path still fires for genuinely empty user input.
Also:
- Widen AGUIMessage.content to `string | AGUIMessageContentPart[]`; the new
exported AGUIMessageContentPart type describes the per-part shape. The
array arm is intentionally permissive so future part types don't force
parallel updates here.
- Export NO_USER_MESSAGE_SENTINEL from the package entry (and the stub) so
downstream consumers can detect the sentinel without string-matching the
magic literal. The on-disk string value is unchanged.
Tests: 13 new cases covering string content, single/multi text parts, mixed
text+file content, file-only content (still sentinels), multi-turn last-user
wins, role filtering, and the wire-format guard on the sentinel constant.
End-to-end verified via examples/no-user-message-repro: same structured
payload that produced "__NO_USER_MESSAGE__" now produces "summarize this".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
extractLastUserMessageonly handledtypeof content === "string", so any AG-UI client that sent multimodal user content (text + file attachment as a structuredcontentarray — what CopilotKit's chat does whenever the user attaches a file) caused the recorder to emitmatch.message: "__NO_USER_MESSAGE__"and produce an unmatchable fixture.This PR teaches the extractor to walk
AGUIMessageContentPart[]and jointype: "text"parts. Non-text parts are skipped.Changes
AGUIMessage.contentwidened tostring | AGUIMessageContentPart[](new exported type). Permissive on the array arm so future part types don't force parallel updates.extractLastUserMessagenow handles both string and structured content; preserves the""-when-empty contract thatmatchesFixture/proxyAndRecordAGUIdepend on.Reproduction
A self-contained Next.js + CopilotKit example is included at
examples/no-user-message-repro/. The README documents two paths: full chat UI with file attachment, and a one-line headlesscurlagainst a synthetic AG-UI multimodal payload.recorded-sample/before-fix.jsonandafter-fix.jsonare pre-captured exhibits of the same request against 1.26.1 and against this branch, respectively.