Conversation
… are emitted A tool result carrying an image is followed by a synthetic role:"user" image message. That message was flushed inside the per-content-block loop, so with two image-returning tools in one assistant tool_calls batch the request serialized as tool, user, tool, user. Strict OpenAI-compatible endpoints reject that with "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'", and because the shape is persisted in the transcript every later turn fails the same way, so the session stays broken. Defer the image messages to the end of the message, matching the pending_image_messages handling already in the Databricks formatter. Fixes aaif-goose#11893
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11893
Summary
Issue #11893 is Ready on the Goose Issues board (moved out of Inbox by @jamadeo on 2026-09-09).
format_messages_with_optionsdeclares and flushesimage_messagesinside the per-content-block loop, so each image-bearing tool result is immediately followed by its syntheticrole: "user"image message. All tool results of one batch live in a singleMessage::user(), so two image-returning tools in one assistant response serialize as:Strict OpenAI-compatible endpoints reject that with
Bad request (400): An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The shape is persisted in the transcript, so every later turn re-sends it and fails identically.This implements the two-phase emit @jamadeo specified on the issue: emit every
role: "tool"message first in tool-call order, then the syntheticrole: "user"image messages. It ports thepending_image_messagesdeferral thatformats/databricks.rsalready has (merged in #9241) into the sibling formatter, which never got it. The March attempts #7393 and #7450 touched both files and were closed; #9241 landed the Databricks half alone.The production change is four lines in one file: declare
pending_image_messagesbesideoutput(message-scoped instead of content-block-scoped), push into it, andoutput.append(&mut pending_image_messages)once the content loop finishes.output.insert(0, converted)still puts the message itself ahead of everything, so ordering downstream is unchanged.Testing
The verification plan on the issue is that the formatter must put all tool results of a batch before the synthetic image messages, with tool-call order preserved.
Two regression tests were added to
formats::openai::tests, mirroringtest_parallel_tool_responses_with_images_are_consecutiveandtest_mixed_tool_responses_image_and_text_orderingthat already exist informats/databricks.rs. The openai formatter had no equivalent coverage.Tests were written first and confirmed failing on pristine
mainat d57c9a4, before the source change:The left side is exactly the wire order reported in the issue.
With the fix applied:
@GreenTea321 flagged on the issue that
test_merge_split_tool_calls_with_image_gapmight have to be inverted because it asserts the interleaved order. It does not: that test callsmerge_split_tool_call_messagesdirectly on a hand-builtVec<Value>, so it is independent of the emit site. It passes unchanged, as shown above.Deliberately not covered: this is the emit-site reordering only.
merge_split_tool_call_messagesandis_image_only_user_messageare untouched, and a session whose transcript already contains the bad interleaving is not repaired retroactively, the shape just stops being produced. Non-vision models and single-image tool results serialize byte-identically, since no synthetic image message is created in those cases. This is a formatter change rather than an agent-loop change, so theagent.rs/state_machineparity requirement does not apply.Also not verified here: the live 400 against a real OpenAI-compatible vision endpoint. The reporter reproduced that on macOS 1.49.0 and @GreenTea321 independently on Windows 1.50.0; this PR verifies the wire shape those reports blame.
Related Issues
Discussion: #11893. Prior art: #9241 (merged, Databricks half), #7393 and #7450 (closed), #7400.