feat: align PlayCanvas MCP tools with agent-tool-design / ACI best practices + runtime observability & input by wangcan26 · Pull Request #88 · playcanvas/editor-mcp-server

wangcan26 · 2026-06-29T07:41:59Z

Background / Goal

Align the MCP tools with agent-tool-design / ACI best practices and close the full runtime feedback loop (edit → run → observe → interact → verify), making the tools more ergonomic and verifiable for LLM agents.
#85

Key Changes

Unified response envelope (11, foundational)

Every tool returns one shape, wrapped centrally in wss.call / callImage:

Errors live in meta._status / meta._message (underscore-prefixed metadata); the protocol-level isError is still set. Scalar ops return small objects ({ deleted: 1 }) instead of bare true.
callImage returns the image block plus a parallel text block carrying the same meta, and forwards the real mimeType (image/webp).
Added a 30s timeout to _send() so requests never hang silently.

Agent ergonomics

3 Empty results are success: list_* / resolve_entities return data: [] with meta.total = 0 instead of an error.
8 Actionable errors: messages tell the agent how to recover (e.g. "call list_entities / resolve_entities to obtain a valid id").
1 Pagination: list_entities / list_assets accept limit (default 50) / offset and return total / count / _has_more / _next_cursor.
2 Reduce reliance on raw UUIDs: summaries include a human-readable hierarchy path (e.g. Root/Player/Camera); new resolve_entities locates entities by name.
12 Return the new state snapshot with the action result: create / modify / add_components / remove_components / reparent / duplicate / instantiate / scene:settings:modify and asset mutations return the resulting summary inline, collapsing act→observe.
4 Close tool asymmetries: new set_material_properties and attach_script; add_components brought to parity with ComponentNameSchema.
5 / 6 Stronger tool descriptions (incl. "when NOT to use", units/coordinate conventions) and MCP annotations (readOnlyHint / destructiveHint / openWorldHint).

Runtime observability (10 MVP)

Dual-peer transport: WSS manages two sockets — editor and runtime — distinguished by a { register } handshake. runtime:* routes to the launched page, everything else to the editor. Added hasRuntime() / waitForRuntime().
Runtime content script (extension/launch.js, injected into launch.playcanvas.com at document_start, MAIN world): hooks console / error / unhandledrejection before the app boots, screenshots the live frame buffer (readPixels on frameend, 800px WebP), and reads mcp_port from the URL to connect back as the runtime peer.
New tools: launch_start (returns { url, sceneId, ready }), launch_stop, capture_runtime (readOnlyHint), read_runtime_logs (level/keyword filter + pagination, newest-first, defaults to warn+error).

Runtime input injection (10 cont.)

New tool inject_input: dispatches an ordered list of input events to the running instance and returns { dispatched }.
- key: press (down+up, optional holdMs to hold) / down / up, with repeat and friendly names (Space, ArrowUp, Enter, letters/digits).
- mouse: move / down / up / click at (x, y) with button.
- touch: start / move / end / tap at (x, y).
- Coordinates are CSS pixels from the canvas top-left (matching capture_runtime framing); betweenMs spaces consecutive events.
Runtime handler (runtime:input): dispatches synthetic DOM events — keyboard events fire on both window and document (the two common pc.Keyboard targets) and back keyCode / which via getters (these can't be set through the KeyboardEvent constructor); mouse/touch fire on the app canvas.
This lets the agent act like a real player (e.g. hold Space for a charge-jump) and then verify the effect via capture_runtime / read_runtime_logs.

Deliberate follow-ups (not in this MR)

7 Consistent public tool namespacing — a breaking rename, deferred to one coordinated change with docs + downstream config.
9 Evaluation harness — separate effort; can now rely on 10's runtime ground-truth signal.
10 advanced — query_runtime_state and step_runtime (deterministic frame stepping) are not yet implemented.

Test plan

tsc --noEmit / npm run lint pass
node --check extension/main.js + launch.js pass
Server boots and registers all tools
Manual (edit-time): pagination, a create→summary round-trip, an error path, set_material_properties
Manual (runtime): launch_start → ready:true, read_runtime_logs, capture_runtime, launch_stop
Manual (runtime): inject_input — hold Space (holdMs) for a charge-jump and a mouse click at canvas coords, then confirm via capture_runtime

Note: the runtime parts can't be verified in-repo; they need a real editor + browser. Requires allowing pop-ups and reloading the unpacked extension.

Add unified {data,meta} response envelope with inline _status/_message, pagination for list_*, semantic hierarchy paths, inline state snapshots on mutations, actionable errors, richer descriptions, and MCP annotations. New tools: resolve_entities, attach_script, set_material_properties; bring add_components to parity with ComponentNameSchema. Refs enhancement-issue.md

…s#10 MVP) Add the missing edit->observe feedback loop so an agent can run a scene and verify it actually works, not just inspect edit-time state. New tools (src/tools/runtime.ts): - launch_start: start a real Launch instance (editor's Launch button), wait for it to connect, return { url, sceneId, ready }. - launch_stop: close the running instance. - capture_runtime: screenshot the RUNNING app (live frame buffer, WebP). - read_runtime_logs: console/exception output, newest-first, filtered by level/keyword + paginated (defaults to warn+error). Transport (src/wss.ts): - Dual-peer model: editor + runtime sockets distinguished by a register handshake; runtime:* methods route to the launched page, the rest to the editor. Add hasRuntime()/waitForRuntime() and raw()/ok()/fail() helpers. Extension: - launch.js: new content script injected into launch.playcanvas.com at document_start; hooks console/error before boot, screenshots via readPixels on frameend, reads mcp_port from the URL to connect back as the runtime peer. - main.js: register as editor; add launch:start/launch:stop. - manifest.json: launch-page content script + host_permissions. Robustness fixes uncovered while wiring this up: - wss: default a connecting peer to the editor role so clients that don't send the register handshake (older extension builds) still work; fix a listener leak on reconnect; surface EADDRINUSE clearly. - server: dedupe PIDs in findPids() and bound the port-free wait so startup can no longer deadlock (a multi-line lsof result previously broke the kill command and the unbounded poll hung forever). - main.js: WSC.call returns a structured error for unknown methods instead of undefined, so an outdated extension yields a clear message. - runtime: launch_start guards an undefined editor response with actionable guidance to reload the extension. Refs: enhancement-issue.md (playcanvas#10)

…as#10) Complete the runtime feedback loop: an agent can now drive the running Launch instance with real keyboard/mouse/touch input, then verify the result via capture_runtime / read_runtime_logs (previously it could only observe, not interact -- e.g. it couldn't perform a charge-jump like a real player). New tool (src/tools/runtime.ts): - inject_input: dispatch an ordered list of input events to the running app. Supports key press (down+up, optional holdMs hold) / down / up with repeat, mouse move/down/up/click, and touch start/move/end/tap. Coordinates are CSS pixels from the canvas top-left (matching capture_runtime framing); betweenMs spaces consecutive events. Extension (extension/launch.js): - runtime:input handler dispatches synthetic DOM events. Keyboard events fire on both window and document (the two common pc.Keyboard targets) and back keyCode/which via getters since they can't be set through the KeyboardEvent constructor; mouse/touch fire on the app canvas. Includes a key-name map (Space, arrows, Enter, letters/digits, etc.). Docs: - README: document inject_input under Runtime tools with usage notes. Refs: enhancement-issue.md (playcanvas#10 inject_input)

kpal81xd

Overall looks v good - just a few design comments to resolve.

Resolve the four review comments from playcanvas#88: - wss: drop the duplicate public-tool-name argument from call/callImage/ok/fail so call sites keep the single `assets:script:parse`-style websocket method name; meta.tool is now derived from that method name. - wss: rename underscore-prefixed envelope metadata (_status/_message/_has_more/ _next_cursor/_hint) to status/message/hasMore/nextCursor/hint, since they already live under `meta` and underscores are reserved for private props. - wss: switch JSDoc to TSDoc (drop the `{type}` annotations now that TypeScript carries the types); turn off jsdoc/require-param-type and jsdoc/require-returns-type locally so the TSDoc style lints clean. - launch.js: remove the banner separator comments. Update README/PR docs and tool descriptions to match the new meta field names.

kpal81xd

LGTM great PR thank you so much 😄

@returns

Drop redundant {type} annotations from @param/@returns tags across the extension scripts (launch.js, main.js, popup.js) to match the project's TSDoc convention. Class field @type {} annotations are kept, since plain JS files have no inline-type alternative.

Resolve conflicts by keeping feat's runtime/envelope functionality while conforming code style to main's eslint-config v3 migration (playcanvas#82): - eslint.config.mjs: adopt main's simplified `export default typescriptConfig` - wss.ts / server.ts / runtime.ts: keep feat features, apply main's type conventions (unknown[], typed Map, CallToolResult, `err instanceof Error` guards, top-level `import type`, no unused catch bindings) - material.ts: use main's `import type { WSS }` style, keep MaterialSchema import

…lity Address the observability gaps surfaced by the CHIEF session attribution (runtime read-back was blind, forcing relaunch/recapture guessing loops). - runtime: add query_runtime_state tool + runtime:state handler returning live entity world/local transform, enabled, components, rigidbody (type/mass/linear+angular velocity) and element text (HUD), with name/ids filtering and limit/offset pagination — the non-visual ground truth for "did the ball move / is physics simulating / what does the score say" without guessing screenshot timing - entity: validate modify_entities paths on write, rejecting unknown top-level keys and component paths for absent components with an actionable message listing the entity's valid paths/components, so a no-op edit is no longer mistaken for success (issue playcanvas#8) - schema/docs: document that rigidbody/collision only simulate when Ammo is enabled (IMPORT AMMO), and that collision halfExtents/radius are in local space and scaled by entity world scale

@returns

Address review feedback on playcanvas#88: revert the extension scripts (launch.js, main.js, popup.js) back to JSDoc with {type} annotations — plain JS has no inline-type alternative, so the doc comments are the only place the types live. Instead of dropping types everywhere, scope the TSDoc relaxation to TypeScript: add a `files: ['**/*.ts']` override in eslint.config.mjs that turns off jsdoc/require-param-type and jsdoc/require-returns-type, so .ts keeps TSDoc (types carried by the compiler) while .js keeps its {type} fields. - eslint.config.mjs: add .ts-only override for the two jsdoc type rules - extension/{launch,main,popup}.js: restore {type} on @param/@returns npm run lint (eslint src) passes.

@returns

Address review feedback on playcanvas#88: revert the extension scripts (launch.js, main.js, popup.js) back to JSDoc with {type} annotations — plain JS has no inline-type alternative, so the doc comments are the only place the types live. Instead of dropping types everywhere, scope the TSDoc relaxation to TypeScript: add a `files: ['**/*.ts']` override in eslint.config.mjs that turns off jsdoc/require-param-type and jsdoc/require-returns-type, so .ts keeps TSDoc (types carried by the compiler) while .js keeps its {type} fields. - eslint.config.mjs: add .ts-only override for the two jsdoc type rules - extension/{launch,main,popup}.js: restore {type} on @param/@returns npm run lint (eslint src) passes.

@returns

Address review feedback on playcanvas#88: revert the extension scripts (launch.js, main.js, popup.js) back to JSDoc with {type} annotations — plain JS has no inline-type alternative, so the doc comments are the only place the types live. Instead of dropping types everywhere, scope the TSDoc relaxation to TypeScript: add a `files: ['**/*.ts']` override in eslint.config.mjs that turns off jsdoc/require-param-type and jsdoc/require-returns-type, so .ts keeps TSDoc (types carried by the compiler) while .js keeps its {type} fields. - eslint.config.mjs: add .ts-only override for the two jsdoc type rules - extension/{launch,main,popup}.js: restore {type} on @param/@returns npm run lint (eslint src) passes.

@returns

* feat(tools): align MCP tools with agent-tool-design best practices Add unified {data,meta} response envelope with inline _status/_message, pagination for list_*, semantic hierarchy paths, inline state snapshots on mutations, actionable errors, richer descriptions, and MCP annotations. New tools: resolve_entities, attach_script, set_material_properties; bring add_components to parity with ComponentNameSchema. Refs enhancement-issue.md * feat(runtime): add live Launch runtime observability tools (#10 MVP) Add the missing edit->observe feedback loop so an agent can run a scene and verify it actually works, not just inspect edit-time state. New tools (src/tools/runtime.ts): - launch_start: start a real Launch instance (editor's Launch button), wait for it to connect, return { url, sceneId, ready }. - launch_stop: close the running instance. - capture_runtime: screenshot the RUNNING app (live frame buffer, WebP). - read_runtime_logs: console/exception output, newest-first, filtered by level/keyword + paginated (defaults to warn+error). Transport (src/wss.ts): - Dual-peer model: editor + runtime sockets distinguished by a register handshake; runtime:* methods route to the launched page, the rest to the editor. Add hasRuntime()/waitForRuntime() and raw()/ok()/fail() helpers. Extension: - launch.js: new content script injected into launch.playcanvas.com at document_start; hooks console/error before boot, screenshots via readPixels on frameend, reads mcp_port from the URL to connect back as the runtime peer. - main.js: register as editor; add launch:start/launch:stop. - manifest.json: launch-page content script + host_permissions. Robustness fixes uncovered while wiring this up: - wss: default a connecting peer to the editor role so clients that don't send the register handshake (older extension builds) still work; fix a listener leak on reconnect; surface EADDRINUSE clearly. - server: dedupe PIDs in findPids() and bound the port-free wait so startup can no longer deadlock (a multi-line lsof result previously broke the kill command and the unbounded poll hung forever). - main.js: WSC.call returns a structured error for unknown methods instead of undefined, so an outdated extension yields a clear message. - runtime: launch_start guards an undefined editor response with actionable guidance to reload the extension. Refs: enhancement-issue.md (#10) * feat(runtime): add inject_input tool for live runtime input (#10) Complete the runtime feedback loop: an agent can now drive the running Launch instance with real keyboard/mouse/touch input, then verify the result via capture_runtime / read_runtime_logs (previously it could only observe, not interact -- e.g. it couldn't perform a charge-jump like a real player). New tool (src/tools/runtime.ts): - inject_input: dispatch an ordered list of input events to the running app. Supports key press (down+up, optional holdMs hold) / down / up with repeat, mouse move/down/up/click, and touch start/move/end/tap. Coordinates are CSS pixels from the canvas top-left (matching capture_runtime framing); betweenMs spaces consecutive events. Extension (extension/launch.js): - runtime:input handler dispatches synthetic DOM events. Keyboard events fire on both window and document (the two common pc.Keyboard targets) and back keyCode/which via getters since they can't be set through the KeyboardEvent constructor; mouse/touch fire on the app canvas. Includes a key-name map (Space, arrows, Enter, letters/digits, etc.). Docs: - README: document inject_input under Runtime tools with usage notes. Refs: enhancement-issue.md (#10 inject_input) * fix: address review feedback on agent-tool-design PR Resolve the four review comments from #88: - wss: drop the duplicate public-tool-name argument from call/callImage/ok/fail so call sites keep the single `assets:script:parse`-style websocket method name; meta.tool is now derived from that method name. - wss: rename underscore-prefixed envelope metadata (_status/_message/_has_more/ _next_cursor/_hint) to status/message/hasMore/nextCursor/hint, since they already live under `meta` and underscores are reserved for private props. - wss: switch JSDoc to TSDoc (drop the `{type}` annotations now that TypeScript carries the types); turn off jsdoc/require-param-type and jsdoc/require-returns-type locally so the TSDoc style lints clean. - launch.js: remove the banner separator comments. Update README/PR docs and tool descriptions to match the new meta field names. * docs: convert JSDoc type annotations to TSDoc style Drop redundant {type} annotations from @param/@returns tags across the extension scripts (launch.js, main.js, popup.js) to match the project's TSDoc convention. Class field @type {} annotations are kept, since plain JS files have no inline-type alternative. * feat(runtime): add query_runtime_state and harden play-mode observability Address the observability gaps surfaced by the CHIEF session attribution (runtime read-back was blind, forcing relaunch/recapture guessing loops). - runtime: add query_runtime_state tool + runtime:state handler returning live entity world/local transform, enabled, components, rigidbody (type/mass/linear+angular velocity) and element text (HUD), with name/ids filtering and limit/offset pagination — the non-visual ground truth for "did the ball move / is physics simulating / what does the score say" without guessing screenshot timing - entity: validate modify_entities paths on write, rejecting unknown top-level keys and component paths for absent components with an actionable message listing the entity's valid paths/components, so a no-op edit is no longer mistaken for success (issue #8) - schema/docs: document that rigidbody/collision only simulate when Ammo is enabled (IMPORT AMMO), and that collision halfExtents/radius are in local space and scaled by entity world scale * docs: keep JSDoc types in .js, relax TSDoc rule for .ts only Address review feedback on #88: revert the extension scripts (launch.js, main.js, popup.js) back to JSDoc with {type} annotations — plain JS has no inline-type alternative, so the doc comments are the only place the types live. Instead of dropping types everywhere, scope the TSDoc relaxation to TypeScript: add a `files: ['**/*.ts']` override in eslint.config.mjs that turns off jsdoc/require-param-type and jsdoc/require-returns-type, so .ts keeps TSDoc (types carried by the compiler) while .js keeps its {type} fields. - eslint.config.mjs: add .ts-only override for the two jsdoc type rules - extension/{launch,main,popup}.js: restore {type} on @param/@returns npm run lint (eslint src) passes. * fix(mcp): stabilize connection for single- and multi-client use The connection became unstable (especially with multiple agent MCP clients). Fixes: - Editor extension now auto-reconnects after an unexpected drop instead of getting stuck DISCONNECTED until a manual CONNECT (extension/main.js). - Multiple MCP server instances no longer kill each other's process on startup (the kill/restart storm). Instead the port owner keeps serving and standby instances retry-bind and take over automatically when it exits. Force-reclaim is available via MCP_TAKEOVER=1 (server.ts, wss.ts). - Add a per-socket 'error' handler so abrupt socket errors don't surface as process-wide uncaught exceptions (wss.ts). - Make the ping loop self-heal: stop pinging when there is no live editor socket instead of spinning into the void (wss.ts). - Wrap socket.send() so a throw during send rejects cleanly without leaking a pending callback/timer (wss.ts). - Raise the request timeout from 30s to 60s so slow but legitimate ops (batch asset creation, script upload) are not cut off (wss.ts). * fix(mcp): graceful port handoff so the newest client controls the editor The previous coexist model was "first instance wins": a stale-but-alive server that still held port 52000 would keep the editor, leaving the active client stuck on standby ("another instance owns port 52000"). Switch to "latest instance wins, gracefully": on startup a new server asks the current owner to hand off ({ yield: true }). The owner releases the port and drops to standby WITHOUT exiting (so its MCP client does not restart it — no kill/restart storm), and takes the port back automatically if the new owner exits. Reclaim-by-kill remains only as a fallback for an unresponsive/old/orphaned owner; MCP_TAKEOVER=1 forces a hard reclaim. * fix(mcp): stop orphaned server processes from piling up and hogging the port Stale editor-mcp-server processes accumulated: when a client/launcher died, the leaf server (reparented to init, ppid 1) kept running forever, holding port 52000 and forcing every other session onto standby. The deep launch chain (cli.mjs -> npx -> tsx -> node) made this worse. - cli.mjs: launch the server as a DIRECT child (node --import tsx server.ts) instead of `npx tsx`, and tie the child's lifetime to the launcher (forward signals, kill child on exit). Shorter chain => reliable reaping. - server.ts: exit when stdin ends, and exit when reparented to init (ppid === 1) — so a server whose client/launcher is gone releases the port instead of lingering as an orphan. --------- Co-authored-by: 王璨 <wangcan.1991@bytedance.com>

王璨 added 3 commits June 23, 2026 14:26

wangcan26 mentioned this pull request Jun 29, 2026

Enhancement: align MCP tools with agent-tool-design / ACI best practices #85

Closed

willeastcott requested review from kpal81xd and willeastcott June 29, 2026 09:41

willeastcott added the enhancement New feature or request label Jun 29, 2026

kpal81xd requested changes Jun 29, 2026

View reviewed changes

Comment thread extension/launch.js Outdated

Comment thread src/tools/assets/script.ts Outdated

Comment thread src/wss.ts

Comment thread src/wss.ts

kpal81xd assigned wangcan26 Jun 29, 2026

kpal81xd approved these changes Jun 29, 2026

View reviewed changes

Comment thread extension/main.js

willeastcott approved these changes Jun 29, 2026

View reviewed changes

kpal81xd requested changes Jun 30, 2026

View reviewed changes

Comment thread extension/launch.js

王璨 added 3 commits July 1, 2026 10:16

wangcan26 force-pushed the feat/enhancement_tool branch 2 times, most recently from aacb6c2 to 4f891f6 Compare July 1, 2026 07:58

kpal81xd approved these changes Jul 1, 2026

View reviewed changes

kpal81xd merged commit d3abf45 into playcanvas:main Jul 1, 2026
1 check passed

Navigation Menu

Uh oh!

feat: align PlayCanvas MCP tools with agent-tool-design / ACI best practices + runtime observability & input - #88

feat: align PlayCanvas MCP tools with agent-tool-design / ACI best practices + runtime observability & input#88
kpal81xd merged 8 commits into
playcanvas:mainfrom
wangcan26:feat/enhancement_tool

wangcan26 commented Jun 29, 2026

Uh oh!

kpal81xd left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kpal81xd left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

wangcan26 commented Jun 29, 2026

Background / Goal

Key Changes

Unified response envelope (11, foundational)

Agent ergonomics

Runtime observability (10 MVP)

Runtime input injection (10 cont.)

Deliberate follow-ups (not in this MR)

Test plan

Uh oh!

kpal81xd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kpal81xd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants