frontend tools

[pmeier]

Summary

Enable users to mount custom JavaScript tools into the frontend for both production (Helm-deployed Docker image) and local development (npm run dev). The frontend dynamically loads these tools, presents a multi-select toggle near the chat prompt for enabling/disabling them per-thread, and executes tool calls client-side when the backend/LLM requests them. Tools are discovered via a tools.json configuration file and individual JS modules served as static assets.

Motivation

The chat pack is frequently used in live demos and workshops with audiences that include non-experts. Explaining concepts like agents, models, and tools in the abstract is difficult — toggling individual tools on and off during a live demo makes these concepts tangible and natural. An audience can see what happens when a tool is available vs. disabled, observe the tool call in action, and understand the difference between the model's reasoning and its tool usage. The per-thread toggle UI is designed specifically to support this interactive, step-by-step demonstration format.

Goals

• Allow users to add custom JS/TS tools to the frontend container without rebuilding the image.
• Support the same tool workflow during local development (npm run dev) without Docker or Helm.
• Provide a UI for selecting which tools are active, persisted per-thread in localStorage.
• Execute frontend tool calls client-side: intercept tool call events from the backend, run the corresponding JS function, and restart the run with the result.
• Ship tools via Helm chart values, supporting both inline definitions and external ConfigMap references.

Non-Goals

• Building or bundling tool code at Helm install time. Tool JS files must be served as-is by nginx.
• Tool execution on the backend. Frontend tools are purely client-side.
• Cross-device persistence of tool state. State is per-browser via localStorage.
• Editing or creating tools through the UI. Tools are defined declaratively via Helm or, in development, via a local tools/ directory.

Background / Motivation

AG-UI supports frontend tools by accepting a tools array (JSON schemas) in the run input sent to /api/threads/{id}/run. When the LLM decides to call a tool, it emits ToolCallStart, ToolCallArgs, and ToolCallEnd SSE events, then ends the run. The frontend must then execute the tool and restart the run with the tool result.

Currently, the frontend passes tools: [] in every run (see hooks.ts TODO comment). Tool calls are purely backend-executed and rendered for display. This design adds full client-side tool support.

Design

1. Tool Configuration File (tools.json)

A JSON file served at /tools/tools.json. In production, this is a static asset mounted into the nginx container via Helm. In development, it is served by the Vite dev server from frontend/public/tools/tools.json.

[
  {
    "tool": {
      "name": "get_weather",
      "description": "Get current weather for a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": { "type": "string", "description": "City name" }
        },
        "required": ["location"]
      }
    },
    "importPath": "/tools/weather.js",
    "entrypoint": "fetchWeather"
  }
]

• tool: The full JSON schema object for the tool, as expected by AG-UI (name, description, parameters). The name field also serves as the unique identifier for the tool throughout the system (used as the key in localStorage, the lookup key when matching ToolCallStart events, and the display label). No separate id field is needed.
• importPath: The URL path to load the JS module via dynamic import(). In production, served by nginx. In development, served by the Vite dev server.
• entrypoint: The name of the named export to call in the loaded module.

Authoring language: Tools should be written in TypeScript for type safety and developer experience. However, the browser cannot load .ts files directly — tool modules must be transpiled to JavaScript (ES modules) before being placed in the tools/ directory. The importPath must reference the transpiled .js file.

Each JS tool module exports one or more named async functions:

// /tools/weather.js
export async function fetchWeather(args) {
  const { location } = args;
  // ... execute logic ...
  return { temperature: 72, conditions: "sunny" };
}

The function receives parsed JSON arguments and may return any JSON-serializable value. The frontend handles JSON.stringify() on the result before embedding it in the content field of the tool role message.

2. Frontend Tool Discovery

Add a new API utility module frontend/src/api/tools.ts:

export interface ToolConfig {
  tool: {
    name: string;
    description: string;
    parameters: Record<string, unknown>;
  };
  importPath: string;
  entrypoint: string;
}

export async function getToolConfig(): Promise<ToolConfig[]> {
  const resp = await fetch('/tools/tools.json');
  if (!resp.ok) return [];
  return resp.json();
}

The app config is extended to include the loaded tools. A new context or integration with the existing AppConfig makes tools available throughout the app.

3. Tool Execution Flow

The tool execution loop in useOnSubmit / the run handler:

1. Send run: Include all active frontend tool schemas in the tools field of createRun.Options for every run, regardless of whether it's a user prompt or a tool result continuation.
2. Stream events: Process events normally. ToolCallStart/ToolCallArgs/ToolCallEnd events are accumulated for tool calls whose name matches a loaded frontend tool.
3. Detect pending tool calls: After RUN_FINISHED, check for any frontend tool calls that were started but not resolved (no corresponding ToolCallResult event came from the backend — because backend tools produce results, frontend tools don't).
4. Execute tools: For each pending frontend tool call:
   • Dynamically import() the tool module from importPath.
   • Call the named entrypoint function with the parsed arguments.
   • Collect the result string.
5. Restart run: Start a new run with a single tool role message (or multiple if there were multiple pending tool calls):

{
  "messages": [
    {
      "role": "tool",
      "id": "<uuid>",
      "toolCallId": "<tool_call_id>",
      "content": "<JSON string result>"
    }
  ],
  "tools": [ ...frontend tool schemas... ]
}

6. Repeat: Continue until the run finishes with no pending frontend tool calls.

4. Per-Thread Tool State

Tool enabled/disabled state is stored in localStorage with key format chat:tools:{threadId}:

{ "get_weather": true, "search_docs": false }

• When no thread exists (new chat), use a default key chat:tools:default.
• On thread switch, load the state from the corresponding key.
• Default: all tools are disabled.
• Provide utility functions in a new frontend/src/lib/tools.ts module:

function loadToolState(threadId: string | null): Record<string, boolean>;
function saveToolState(threadId: string | null, state: Record<string, boolean>): void;
function getActiveToolNames(threadId: string | null, allTools: ToolConfig[]): ToolConfig[];

5. Tool Selector UI

A button placed inside ChatInput next to the file upload (paperclip) button. When clicked, it opens an overlayed panel docked above the button.

Closed state

A button containing:

• A sliders-horizontal icon (Lucide sliders-horizontal).
• The text "Tools".
• A badge showing the number of currently active frontend tools (hidden when count is 0).
• A down-pointing caret indicating the button opens an overlayed panel.

Opened panel

The panel is docked directly above the button (toward the top of the viewport). It lists all available tools in two sections: Frontend Tools and Backend Tools.

Each tool entry has the same left-side layout:

• Tool name in bold.
• Tool description below the name in muted/smaller text.

Frontend tools (right side):

• A toggle switch to enable/disable the tool.
• Toggles default to off.
• State is persisted to localStorage per-thread.

Backend tools (right side):

• A lock icon (Lucide lock-keyhole) instead of a toggle.
• Backend tools are always active — they are baked into the agent configuration and cannot be changed from the frontend.

Backend tool data source

Backend tool names and descriptions are parsed from the agent's capabilities.tools field, which is part of the AG-UI AgentCapabilitiesSchema from @ag-ui/core. This data is returned by /api/config and is already loaded into useAppConfig().agents.

Implementation:

• New component: frontend/src/chat/toolselector.tsx
• Uses @/components/ui/tooltip (for the button), @/components/ui/badge (for the active count), and custom panel overlay (positioned relative to the button).
• Integrated into ChatInput, positioned to the left of the file upload button.
• Backend tools sourced from useAppConfig().agents → capabilities.tools.
• Frontend tools sourced from the tools context (loaded via getToolConfig()).
• The active frontend tool list flows down to the useOnSubmit hook, which includes them in every run.

6. Local Development Support

Tools must work during npm run dev without Docker or Helm.

Tool directory layout:

frontend/
  public/
    tools/
      tools.json        # Tool config (same format as production)
      weather.js        # Tool JS module

Files in frontend/public/tools/ are served as-is by the Vite dev server at /tools/, matching the production URL structure. This means the same tools.json and importPath values work in both environments.

Development workflow:

1. Place tools.json and tool JS files in frontend/public/tools/.
2. Run npm run dev — tools are automatically available.
3. For TypeScript tool development, users should write .ts files (e.g., in frontend/tools/) and transpile them to .js output in frontend/public/tools/. A simple tsc script or Vite config can handle this. The browser can only load the transpiled .js files.

7. Helm Chart Changes

values.yaml additions

frontend:
  # ... existing values ...
  tools:
    # List of tool definitions. Can be inline or reference a ConfigMap.
    inline: []
    # - name: get_weather
    #   tool:
    #     name: get_weather
    #     description: Get current weather
    #     parameters: { ... }
    #   importPath: /tools/weather.js
    #   entrypoint: fetchWeather
    #   source: inline

    # External ConfigMap references containing tool files.
    configMaps: []
    # - name: my-tools-config
    #   mountPath: /tools
    #   items:
    #     - key: weather.js
    #       path: weather.js

New template: templates/frontend-tools-configmap.yaml

Generates a ConfigMap containing:

• tools.json — assembled from inline tool definitions.
• Any JS files from inline tool source fields.

Modified template: templates/frontend-deployment.yaml

• Add a volume mount for the tools ConfigMap at /usr/share/nginx/html/tools.
• If user-provided ConfigMaps are referenced (frontend.tools.configMaps), mount those as additional volumes.

Modified: values.schema.json

Add schema definitions for frontend.tools.inline and frontend.tools.configMaps.

8. File Changes Summary

File	Change
frontend/src/api/tools.ts	New — ToolConfig type, getToolConfig() fetcher
frontend/src/lib/tools.ts	New — localStorage persistence utilities
frontend/src/chat/toolselector.tsx	New — tool selector button + overlayed panel with frontend and backend tool listings
frontend/src/chat/chatinput.tsx	Modified — integrate tool selector button next to the file upload button
frontend/src/chat/hooks.ts	Modified — pass active tool schemas to createRun, detect pending frontend tool calls, execute tools, restart run
frontend/src/routes/_authenticated/chat.tsx	Modified — preload tool config in route loader
frontend/src/context/index.ts	Modified — export new tool context
frontend/src/context/tools.ts	New — tool config context and hook
helm/nebari-chat/values.yaml	Modified — add frontend.tools section
helm/nebari-chat/values.schema.json	Modified — schema for tools
helm/nebari-chat/templates/frontend-tools-configmap.yaml	New — ConfigMap for inline tools
helm/nebari-chat/templates/frontend-deployment.yaml	Modified — mount tools volume

Tradeoffs & Risks

1. Dynamic import() in production: Tool authoring should be done in TypeScript for type safety, but the browser cannot load .ts files. Tools must be transpiled to JavaScript ES modules before being placed in the tools/ directory. This is an inherent constraint of the browser runtime and should be clearly documented in the developer workflow.

2. Tool code security: User-provided JS runs in the browser with full access to the DOM and any credentials in scope. Tools can make network requests using the browser's cookies/tokens. This is an inherent risk of client-side tools. Mitigation: document the security implications and recommend tools be audited before deployment.

3. localStorage limits: localStorage has a ~5MB limit per origin, but we only store boolean flags keyed by thread ID, so this is negligible. However, localStorage is per-browser — switching devices loses tool preferences. Acceptable for the MVP.

4. Multiple tool calls per run: If the LLM calls multiple frontend tools in a single turn, the frontend executes all of them sequentially (or in parallel) before restarting the run. Parallel execution is faster but could cause ordering issues if tools have side effects. We execute them in parallel (Promise.all) for speed, accepting the risk.

5. No secrets in frontend tools: Frontend tools run in the browser, so there is no mechanism for securely storing service-level secrets (API keys, database credentials, internal tokens). Anything in the tool JS, config, or container is visible via DevTools. Frontend tools can only safely use the user's own auth token (already available in the browser) or credentials explicitly provided by the user through a separate UI. If a tool requires a service secret, it belongs on the backend. This limits the class of tools suitable for frontend execution to: calling public APIs, calling services where the user's token suffices, or calling user-provided keys.

6. Tools ConfigMap size limits: Kubernetes ConfigMaps are limited to 1MB. For large tool codebases, users should use external storage or bake tools into a custom image.

Open Questions

1. Component visibility — With backend tools now displayed in the panel, should the tool selector button be hidden only when there are no frontend tools (as before), or should it also appear when backend tools exist so users can at least see what tools the agent has? Showing it for backend-only scenarios would give users visibility into agent capabilities but adds UI chrome that isn't actionable (no toggling possible). Options:
   • Hide when no frontend tools exist (current spec) — clean UI, but hides agent tool info.
   • Always show if any tools exist (frontend or backend) — full visibility, but shows a non-interactive button when only backend tools are present.
   • Always show — consistent placement, badge/count reflects frontend tools, panel shows backend tools as reference.

[pmeier]

Here are screenshots of how we did the UI selector in previous custom demos

[jbouder]

I think this plan makes sense. A few thoughts…

1. Definitely suggest using typescript from beginning
2. Need to decide where in the project we should put the original tool TS files. Frontend directory probably makes the most sense
3. On design, we should also try to show the backend tools in the UI, probably disabled though with some indicator.

[jbouder]

One other thought. Are there any existing node-based tool library examples out there that we could pull in?

[pmeier]

1. Definitely suggest using typescript from beginning

I don't think TS would work in general (need to update the design). Since the proposal is to mount the tools at runtime, no TS is allowed as the browser can't handle it.

2. Need to decide where in the project we should put the original tool TS files. Frontend directory probably makes the most sense

What are "original tool TS files". Like built-in ones?

3. On design, we should also try to show the backend tools in the UI, probably disabled though with some indicator.

Makes sense. That is already supported as part of the agent capabilities. I'm working on setting them automatically in nebari-dev/ravnar#38. They can already be set manually so the development on this is not blocked.

[jbouder]

1. Definitely suggest using typescript from beginning

I don't think TS would work in general (need to update the design). Since the proposal is to mount the tools at runtime, no TS is allowed as the browser can't handle it.

2. Need to decide where in the project we should put the original tool TS files. Frontend directory probably makes the most sense

What are "original tool TS files". Like built-in ones?

3. On design, we should also try to show the backend tools in the UI, probably disabled though with some indicator.

Makes sense. That is already supported as part of the agent capabilities. I'm working on setting them automatically in nebari-dev/ravnar#38. They can already be set manually so the development on this is not blocked.

In reference to TS files, my suggestion is adding another project which contains the TS files and tsconfis as needed. Then as part of the dev/build process, those would get transpiled into JS and placed in the public directory.

Not required, but using TypeScript where possible is usually recommended.

[pmeier]

In reference to TS files, my suggestion is adding another project which contains the TS files and tsconfis as needed. Then as part of the dev/build process, those would get transpiled into JS and placed in the public directory.

So you are saying we should recommend tool authors to use TS and mounting the transpiled JS? I'm ok with that recommendation. But that doesn't change that the dynamic import can only support JS.

[smeragoel]

Thanks for the detailed writeup @pmeier! Weighing in one the design side of things:

1. Placement: The current placement works 👍🏼
2. Icon suggestion: I would recommend swapping the gear icon for the SliderHorizontal icon (we're using Lucide iirc). It's a small distinction, but the gear reads more as "settings" while the slider reads as "Configuration" and the latter seems more appropriate imo.
3. Showing backend tools: It makes sense for the demo use case. Instead of it being disabled, we can also show them as locked/read-only. We can have the same layout for the backend tools as the frontend tool, except, instead of a checkbox, we can use a lock icon, and a tooltip that says something like "Managed by the Agent". I remember this came up as feedback in a previous demo that the disabled toggles can read as broken state rather than locked.

[pmeier]

I've updated the design to address the comments made so far.

[pmeier]

As mentioned in #110 (comment) nebari-dev/ravnar#38 will automatically hydrate the capabilities and has an example of what the output can look like. One important thing for this work is the capabilities.tools.client_provided field, which indicates whether the agent can support frontend tools. pydantic_ai can, but agno currently can't (see agno-agi/agno#7801).