Skip to content

feat(agent): pass filesystem images to subagents via task images param - #9741

Open
youssefbm2008 wants to merge 1 commit into
esengine:main-v2from
youssefbm2008:feat/task-images-param
Open

feat(agent): pass filesystem images to subagents via task images param#9741
youssefbm2008 wants to merge 1 commit into
esengine:main-v2from
youssefbm2008:feat/task-images-param

Conversation

@youssefbm2008

Copy link
Copy Markdown

Summary

Adds an explicit images: string[] parameter to task and read_only_task so an agent can hand filesystem images — produced mid-turn by tools (ffmpeg frames, rendered charts, screenshots) — to a vision-capable subagent as real image content blocks. Closes #6530.

Problem

Image candidates were populated only from the user's typed turn input (Controller.resolveInputImageCandidates). A path created during the task could never enter the image pipeline, so vision-capable children could stat the file but never see the pixels. The @-ref workaround does not help because the model never spontaneously adds @, and it is a harness input-stage feature.

What this does

  • images parameter on task / read_only_task (max 8, deduplicated, order preserved). Resolved at dispatch time against the workspace root.
  • Security matrix preserved — every param-passed path goes through the exact pipeline @-references use: os.OpenRoot workspace confinement, symlink rejection, 1 B–64 MB window, TOCTOU os.SameFile check, MIME sniffing, vision-aware downscaling (CompressForVision). Unlike the parent turn path (best-effort skip), a failing path fails the call — the model explicitly asked for these pixels and silent dropping would produce confident wrong answers.
  • Merge semantics — call param first, then parent turn candidates, deduplicated; the existing SubagentImageCandidates behavior is unchanged when no images is passed. Text-only children keep metadata only (child provider owns the vision decision).
  • T1 extraction: file→data-URL conversion moved from internal/control to the new internal/fileref package (FileImageDataURL, CompressForVision, MIME sniffing) so agent can use it without a layering cycle (control imports agent). control delegates; zero behavior change (error strings, thresholds, and MIME handling are byte-identical up to the rename).
  • Wiring: TaskTool.WithImageResolver (DI seam following WithWorkspaceLease/WithProfileLookup), production wires fileref.FileImageDataURL with the boot workspace root. fleet/parallel_tasks share buildTaskSpec, so per-call images ride ContextRequest — the delegation-boundary member that decides "what the child starts from" — instead of widening buildTaskSpec's positional signature.

Test plan

  • internal/fileref/image_test.go — real-image conversion, outside-workspace, symlink, missing/empty, non-image, .. escape, no-root rejection (moved + new).
  • internal/agent/task_images_test.go — param reaches vision child, read_only_task variant, merge with candidates (param-first, deduped, order), param-only turn, missing/empty/over-cap fail clearly, dedupe of identical paths, legacy constructions without a resolver ignore the param, schema documents the param.
  • Existing TestTaskToolPropagatesSubagentImageCandidates and the full internal/control image suite pass unchanged.
  • go test ./internal/agent/ ./internal/control/ ./internal/fileref/ ./internal/boot/ all green; go vet ./... clean; repolint clean.

Cache-impact: low - provider-visible prompt prefix is unchanged; the only new provider-visible surface is the tool-schema images property on task/read_only_task, which is byte-stable across turns. Tool-result storage, compaction, and the system-prompt prefix are untouched.
Cache-guard: go test ./internal/agent/ -run 'TestTaskTool' ./internal/boot/effect_test.go; schema-string tests (TestTaskToolImageParamSchemaDocumentsMax, TestTaskToolSchemaExposesOnlyContinueFromForPersistence) pin the tool surface.

System-prompt-review: SivanCola (tool-schema only; base prompt, memory, and prefix assembly untouched)

Documentation-impact: updated - docs/SPEC.md (en+zh) task parameter list and docs/SUBAGENT_PROFILES(.zh-CN).md document the images parameter, its security checks, and text-only-child behavior.

A vision-capable child model could not receive images produced mid-turn
by tools (ffmpeg frames, rendered SVG/PNG, screenshots): image candidates
were populated only from the user's typed turn input, so a path created
during the task never reached the image pipeline (esengine#6530).

Add an explicit images parameter to task and read_only_task. Paths are
resolved at dispatch through the same security matrix as @-references
(workspace confinement via os.OpenRoot, symlink rejection, 1 B-64 MB cap,
TOCTOU same-file check, MIME sniffing, vision downscaling), then merged
with the parent's turn candidates (param first, deduped, order preserved)
into the child's user-images context. Text-only children keep metadata
only; the child provider still owns the vision decision.

The file-to-data-URL converter moves from internal/control to a new
internal/fileref package so agent can share it without a layering cycle;
control re-exports for its @-reference path. Resolution is injected into
TaskTool via WithImageResolver and wired to the boot workspace root.

Closes esengine#6530
@github-actions github-actions Bot added v2 Go rewrite (1.x) — main-v2 branch, active development agent Core agent loop (internal/agent, internal/control) config Configuration & setup (internal/config) labels Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent Core agent loop (internal/agent, internal/control) config Configuration & setup (internal/config) v2 Go rewrite (1.x) — main-v2 branch, active development

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: 允许智能体将文件系统中的文件发送到多模态模型进行处理

1 participant