feat(agent): pass filesystem images to subagents via task images param - #9741
Open
youssefbm2008 wants to merge 1 commit into
Open
feat(agent): pass filesystem images to subagents via task images param#9741youssefbm2008 wants to merge 1 commit into
youssefbm2008 wants to merge 1 commit into
Conversation
A vision-capable child model could not receive images produced mid-turn by tools (ffmpeg frames, rendered SVG/PNG, screenshots): image candidates were populated only from the user's typed turn input, so a path created during the task never reached the image pipeline (esengine#6530). Add an explicit images parameter to task and read_only_task. Paths are resolved at dispatch through the same security matrix as @-references (workspace confinement via os.OpenRoot, symlink rejection, 1 B-64 MB cap, TOCTOU same-file check, MIME sniffing, vision downscaling), then merged with the parent's turn candidates (param first, deduped, order preserved) into the child's user-images context. Text-only children keep metadata only; the child provider still owns the vision decision. The file-to-data-URL converter moves from internal/control to a new internal/fileref package so agent can share it without a layering cycle; control re-exports for its @-reference path. Resolution is injected into TaskTool via WithImageResolver and wired to the boot workspace root. Closes esengine#6530
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an explicit
images: string[]parameter totaskandread_only_taskso an agent can hand filesystem images — produced mid-turn by tools (ffmpeg frames, rendered charts, screenshots) — to a vision-capable subagent as real image content blocks. Closes #6530.Problem
Image candidates were populated only from the user's typed turn input (
Controller.resolveInputImageCandidates). A path created during the task could never enter the image pipeline, so vision-capable children couldstatthe file but never see the pixels. The@-ref workaround does not help because the model never spontaneously adds@, and it is a harness input-stage feature.What this does
imagesparameter ontask/read_only_task(max 8, deduplicated, order preserved). Resolved at dispatch time against the workspace root.@-references use:os.OpenRootworkspace confinement, symlink rejection, 1 B–64 MB window, TOCTOUos.SameFilecheck, MIME sniffing, vision-aware downscaling (CompressForVision). Unlike the parent turn path (best-effort skip), a failing path fails the call — the model explicitly asked for these pixels and silent dropping would produce confident wrong answers.SubagentImageCandidatesbehavior is unchanged when noimagesis passed. Text-only children keep metadata only (child provider owns the vision decision).internal/controlto the newinternal/filerefpackage (FileImageDataURL,CompressForVision, MIME sniffing) soagentcan use it without a layering cycle (controlimportsagent).controldelegates; zero behavior change (error strings, thresholds, and MIME handling are byte-identical up to the rename).TaskTool.WithImageResolver(DI seam followingWithWorkspaceLease/WithProfileLookup), production wiresfileref.FileImageDataURLwith the boot workspace root.fleet/parallel_taskssharebuildTaskSpec, so per-call images rideContextRequest— the delegation-boundary member that decides "what the child starts from" — instead of wideningbuildTaskSpec's positional signature.Test plan
internal/fileref/image_test.go— real-image conversion, outside-workspace, symlink, missing/empty, non-image,..escape, no-root rejection (moved + new).internal/agent/task_images_test.go— param reaches vision child,read_only_taskvariant, merge with candidates (param-first, deduped, order), param-only turn, missing/empty/over-cap fail clearly, dedupe of identical paths, legacy constructions without a resolver ignore the param, schema documents the param.TestTaskToolPropagatesSubagentImageCandidatesand the fullinternal/controlimage suite pass unchanged.go test ./internal/agent/ ./internal/control/ ./internal/fileref/ ./internal/boot/all green;go vet ./...clean;repolintclean.Cache-impact: low - provider-visible prompt prefix is unchanged; the only new provider-visible surface is the tool-schema
imagesproperty on task/read_only_task, which is byte-stable across turns. Tool-result storage, compaction, and the system-prompt prefix are untouched.Cache-guard: go test ./internal/agent/ -run 'TestTaskTool' ./internal/boot/effect_test.go; schema-string tests (TestTaskToolImageParamSchemaDocumentsMax, TestTaskToolSchemaExposesOnlyContinueFromForPersistence) pin the tool surface.
System-prompt-review: SivanCola (tool-schema only; base prompt, memory, and prefix assembly untouched)
Documentation-impact: updated - docs/SPEC.md (en+zh) task parameter list and docs/SUBAGENT_PROFILES(.zh-CN).md document the images parameter, its security checks, and text-only-child behavior.