|
| 1 | +# Agent Plugins |
| 2 | + |
| 3 | +The agent plugin system extends OPM with composable NLP components for conversational AI, tool use, and text understanding. Plugins are discovered via Python entry points exactly like all other OPM plugin types. |
| 4 | + |
| 5 | +Base classes: `ovos_plugin_manager/templates/agents.py`, `ovos_plugin_manager/templates/agent_tools.py` |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## Entry Point Groups |
| 10 | + |
| 11 | +| Group | Base Class | Purpose | |
| 12 | +|---|---|---| |
| 13 | +| `opm.agents.chat` | `ChatEngine` | Multi-turn chat / agentic loops — `continue_chat(messages)` → `AgentMessage` | |
| 14 | +| `opm.agents.chat.multimodal` | `MultimodalChatEngine` | Chat with image/audio/file inputs | |
| 15 | +| `opm.agents.toolbox` | `ToolBox` | Groups of callable `AgentTool` functions exposed to agents via bus or direct call | |
| 16 | +| `opm.agents.summarizer` | `SummarizerEngine` / `ChatSummarizerEngine` | Document or chat-history summarisation | |
| 17 | +| `opm.agents.retrieval` | `RetrievalEngine` | Knowledge-base / vector-index query (`query(q, lang, k)` → `List[Tuple[str, float]]`) | |
| 18 | +| `opm.plugin.persona` | `dict` | Static persona config dict; consumed by `ovos-persona` to wire a ChatEngine with a system prompt | |
| 19 | + |
| 20 | +`AgentContextManager` (`agents.py:35`) — optional companion base class for plugins that augment conversation context (RAG, memory, history trimming). Not a standalone entry point group; used inside `ChatEngine` implementations. |
| 21 | + |
| 22 | +--- |
| 23 | + |
| 24 | +## Available ToolBoxes (`opm.agents.toolbox`) |
| 25 | + |
| 26 | +Each `ToolBox` implements `discover_tools() → List[AgentTool]` (`agent_tools.py:314`). Tools are callable directly via `ToolBox.call_tool(name, kwargs)` or over the OVOS bus via the `ovos.persona.tools.{toolbox_id}.call` message topic (`agent_tools.py:102`). |
| 27 | + |
| 28 | +| Plugin ID | Class | Tools | Package | API Key | |
| 29 | +|---|---|---|---|---| |
| 30 | +| `ovos-wikipedia-tools` | `WikipediaToolBox` | `search_wikipedia`, `get_wikipedia_sections`, `get_wikipedia_page` | `ovos-wikipedia-solver` | None — public Wikipedia REST API | |
| 31 | +| `ovos-ddg-tools` | `DuckDuckGoToolBox` | `search_duckduckgo`, `get_duckduckgo_infobox` | `ovos-ddg-solver-plugin` | None — DuckDuckGo Instant Answer API | |
| 32 | +| `ovos-wolfram-alpha-tools` | `WolframAlphaToolBox` | `compute`, `compute_full` | `ovos-wolfram-alpha-solver` | Optional — free key at developer.wolframalpha.com; demo key bundled | |
| 33 | +| `ovos-weather-tools` | `WeatherToolBox` | `get_current_weather`, `get_daily_forecast`, `get_hourly_forecast` | `ovos-skill-weather` | None — Open-Meteo public API | |
| 34 | +| `ovos-datetime-tools` | `DateTimeToolBox` | `get_current_datetime`, `convert_timezone`, `get_timezone_for_location` | `ovos-skill-date-time` | None — stdlib + pytz | |
| 35 | +| `ovos-ip-tools` | `IPAddressToolBox` | `get_local_ip_addresses`, `get_public_ip` | `ovos-skill-ip` | None | |
| 36 | +| `ovos-iss-tools` | `ISSLocationToolBox` | `get_iss_position`, `get_iss_crew` | `ovos-skill-iss-location` | Optional — geonames.org user for reverse geocoding | |
| 37 | +| `ovos-speedtest-tools` | `SpeedTestToolBox` | `run_speedtest` | `ovos-skill-speedtest` | None — Speedtest.net | |
| 38 | +| `ovos-wallpapers-tools` | `WallpapersToolBox` | `search_wallpapers` | `ovos-skill-wallpapers` | None — wallhaven.cc public API | |
| 39 | +| `ovos-wikihow-tools` | `WikiHowToolBox` | `search_wikihow`, `get_wikihow_steps` | `ovos-skill-wikihow` | None — pywikihow scraper | |
| 40 | +| `ovos-wordnet-tools` | `WordNetToolBox` | `lookup_word`, `define_word` | `ovos-skill-wordnet` | None — local NLTK corpus | |
| 41 | +| `ovos-skill-md-toolbox` | `SkillMDToolBox` | dynamic — one tool per installed `SKILL.md` | `ovos-agentic-loop` | Requires a configured `ChatEngine` (brain) | |
| 42 | +| `ovos-filesystem-tools` | `FileSystemToolBox` | `read_file`, `write_file`, `list_directory`, `search_in_files`, `find_files` | `ovos-agentic-loop` | None | |
| 43 | +| `ovos-shell-tools` | `ShellToolBox` | `run_command` | `ovos-agentic-loop` | None | |
| 44 | +| `ovos-web-search-tools` | `WebSearchToolBox` | `web_search` | `ovos-agentic-loop` | None | |
| 45 | +| `ovos-clock-tools` | `ClockToolBox` | `get_current_datetime` | `ovos-agentic-loop` | None | |
| 46 | + |
| 47 | +### Tool schema |
| 48 | + |
| 49 | +Each `AgentTool` (`agent_tools.py:40`) carries: |
| 50 | +- `name` — snake_case identifier used by the LLM |
| 51 | +- `description` — natural-language purpose shown to the LLM |
| 52 | +- `argument_schema` — Pydantic `ToolArguments` subclass; JSON Schema auto-generated for LLM tool-calling APIs |
| 53 | +- `output_schema` — Pydantic `ToolOutput` subclass; validated on every call |
| 54 | +- `tool_call` — the Python callable; receives an instantiated `ToolArguments`, returns `ToolOutput` |
| 55 | + |
| 56 | +`ToolBox.tool_json_list` (`agent_tools.py:290`) converts all tools to the JSON Schema list format expected by OpenAI / Anthropic / Gemini tool-calling endpoints. |
| 57 | + |
| 58 | +--- |
| 59 | + |
| 60 | +## Available Chat Engines (`opm.agents.chat`) |
| 61 | + |
| 62 | +| Plugin ID | Class | Backend | Package | |
| 63 | +|---|---|---|---| |
| 64 | +| `ovos-chat-openai-plugin` | `OpenAIChatEngine` | OpenAI API | `ovos-openai-plugin` | |
| 65 | +| `ovos-chat-gemini-plugin` | `GeminiChatEngine` | Google Gemini | `ovos-gemini-plugin` | |
| 66 | +| `ovos-chat-gemini-code-plugin` | `GeminiCodeChatEngine` | Gemini (code) | `ovos-gemini-plugin` | |
| 67 | +| `ovos-chat-gemini-session-plugin` | `GeminiSessionChatEngine` | Gemini (session) | `ovos-gemini-plugin` | |
| 68 | +| `ovos-chat-claude-plugin` | `ClaudeChatEngine` | Anthropic Claude | `ovos-claude-plugin` | |
| 69 | +| `ovos-chat-claude-code-plugin` | `ClaudeCodeChatEngine` | Claude (code) | `ovos-claude-plugin` | |
| 70 | +| `ovos-chat-claude-code-session-plugin` | `ClaudeCodeSessionChatEngine` | Claude (session) | `ovos-claude-plugin` | |
| 71 | +| `ovos-chat-kilo-plugin` | `KiloChatEngine` | Kilo (Anthropic) | `ovos-kilo-plugin` | |
| 72 | +| `ovos-chat-kilo-session-plugin` | `KiloSessionChatEngine` | Kilo (session) | `ovos-kilo-plugin` | |
| 73 | +| `ovos-chat-gguf-plugin` | `GGUFChatEngine` | Local GGUF (llama.cpp) | `ovos-gguf-plugin` | |
| 74 | +| `ovos-chat-qwen-code-plugin` | `QwenCodeChatEngine` | Qwen-Code | `ovos-qwen-code-plugin` | |
| 75 | +| `ovos-chat-opencode-plugin` | `OpenCodeChatEngine` | OpenCode | `ovos-opencode-plugin` | |
| 76 | +| `ovos-chat-opencode-session-plugin` | `OpenCodeSessionChatEngine` | OpenCode (session) | `ovos-opencode-plugin` | |
| 77 | +| `ovos-wikigpt` | `WikiGPTSolver` | Wikipedia RAG | `ovos-wikipedia-solver` | |
| 78 | +| `ovos-react-loop` | `ReActLoopEnginePlugin` | ReAct over any ChatEngine + ToolBoxes | `ovos-agentic-loop` | |
| 79 | +| `ovos-plan-execute-loop` | `PlanAndExecuteEnginePlugin` | Plan-and-Execute | `ovos-agentic-loop` | |
| 80 | +| `ovos-reflexion-loop` | `ReflexionEnginePlugin` | Reflexion | `ovos-agentic-loop` | |
| 81 | +| `ovos-self-ask-loop` | `SelfAskEnginePlugin` | Self-Ask | `ovos-agentic-loop` | |
| 82 | +| `ovos-chain-of-thought-loop` | `ChainOfThoughtEnginePlugin` | Chain-of-Thought | `ovos-agentic-loop` | |
| 83 | +| `ovos-mos-king-reranker` | `ReRankerKingMoSPlugin` | Mixture-of-Solvers (reranker) | `ovos-MoS` | |
| 84 | +| `ovos-mos-king-generative` | `GenerativeKingMoSPlugin` | MoS (generative king) | `ovos-MoS` | |
| 85 | +| `ovos-mos-democracy` | `DemocracyMoSPlugin` | MoS (majority vote) | `ovos-MoS` | |
| 86 | +| `ovos-mos-duopoly-reranker` | `ReRankerDuopolyMoSPlugin` | MoS (duopoly reranker) | `ovos-MoS` | |
| 87 | +| `ovos-mos-duopoly-generative` | `GenerativeDuopolyMoSPlugin` | MoS (duopoly generative) | `ovos-MoS` | |
| 88 | + |
| 89 | +### Multimodal Chat Engines (`opm.agents.chat.multimodal`) |
| 90 | + |
| 91 | +| Plugin ID | Class | Backend | Package | |
| 92 | +|---|---|---|---| |
| 93 | +| `ovos-chat-multimodal-gemini-plugin` | `GeminiMultimodalChatEngine` | Gemini | `ovos-gemini-plugin` | |
| 94 | +| `ovos-chat-multimodal-claude-plugin` | `ClaudeMultimodalChatEngine` | Claude | `ovos-claude-plugin` | |
| 95 | +| `ovos-chat-multimodal-kilo-plugin` | `KiloMultimodalChatEngine` | Kilo | `ovos-kilo-plugin` | |
| 96 | +| `ovos-chat-multimodal-qwen-code-plugin` | `QwenCodeMultimodalChatEngine` | Qwen-Code | `ovos-qwen-code-plugin` | |
| 97 | + |
| 98 | +`ChatEngine.continue_chat` signature — `agents.py:210`: |
| 99 | +```python |
| 100 | +def continue_chat(self, messages: List[AgentMessage], |
| 101 | + session_id: str = "default", |
| 102 | + lang: Optional[str] = None, |
| 103 | + units: Optional[str] = None) -> AgentMessage: |
| 104 | +``` |
| 105 | + |
| 106 | +`ChatEngine` also provides `stream_tokens`, `stream_sentences`, and `get_response` helpers (`agents.py:228–300`). Plugins only need to implement `continue_chat`. |
| 107 | + |
| 108 | +--- |
| 109 | + |
| 110 | +## Available Personas (`opm.plugin.persona`) |
| 111 | + |
| 112 | +Each persona entry is a dict defining `chat_engine`, `system_prompt`, and optionally `toolboxes`. Loaded and wired by the `ovos-persona` service. |
| 113 | + |
| 114 | +| Persona ID | Backend | Package | |
| 115 | +|---|---|---| |
| 116 | +| `OpenAI` | `ovos-chat-openai-plugin` | `ovos-openai-plugin` | |
| 117 | +| `Claude` | `ovos-chat-claude-plugin` | `ovos-claude-plugin` | |
| 118 | +| `Gemini` | `ovos-chat-gemini-plugin` | `ovos-gemini-plugin` | |
| 119 | +| `Kilo` | `ovos-chat-kilo-plugin` | `ovos-kilo-plugin` | |
| 120 | +| `QwenCode` | `ovos-chat-qwen-code-plugin` | `ovos-qwen-code-plugin` | |
| 121 | +| `OpenCode` | `ovos-chat-opencode-plugin` | `ovos-opencode-plugin` | |
| 122 | +| `Wikipedia` | Wikipedia solver | `ovos-wikipedia-solver` | |
| 123 | +| `WikiGPT` | `ovos-wikigpt` | `ovos-wikipedia-solver` | |
| 124 | +| `DuckDuckGo` | DDG solver | `ovos-ddg-solver-plugin` | |
| 125 | +| `Wolfram Alpha` | Wolfram solver | `ovos-wolfram-alpha-solver` | |
| 126 | +| `WikiHow` | WikiHow solver | `ovos-skill-wikihow` | |
| 127 | +| `Wordnet` | WordNet solver | `ovos-skill-wordnet` | |
| 128 | + |
| 129 | +--- |
| 130 | + |
| 131 | +## How to Implement a ToolBox |
| 132 | + |
| 133 | +Register under `opm.agents.toolbox` in `pyproject.toml`: |
| 134 | + |
| 135 | +```toml |
| 136 | +[project.entry-points."opm.agents.toolbox"] |
| 137 | +my-tools = "my_package.toolbox:MyToolBox" |
| 138 | +``` |
| 139 | + |
| 140 | +Minimal implementation (`agent_tools.py:56`): |
| 141 | + |
| 142 | +```python |
| 143 | +from ovos_plugin_manager.templates.agent_tools import AgentTool, ToolArguments, ToolBox, ToolOutput |
| 144 | +from pydantic import Field |
| 145 | + |
| 146 | +class MyArgs(ToolArguments): |
| 147 | + query: str = Field(..., description="Input text.") |
| 148 | + |
| 149 | +class MyOutput(ToolOutput): |
| 150 | + result: str = Field(..., description="Tool result.") |
| 151 | + |
| 152 | +class MyToolBox(ToolBox): |
| 153 | + toolbox_id = "my-tools" |
| 154 | + |
| 155 | + def __init__(self, config=None): |
| 156 | + self.config = config or {} |
| 157 | + super().__init__(toolbox_id=self.toolbox_id) |
| 158 | + |
| 159 | + def discover_tools(self): |
| 160 | + return [AgentTool( |
| 161 | + name="my_tool", |
| 162 | + description="Does something useful.", |
| 163 | + argument_schema=MyArgs, |
| 164 | + output_schema=MyOutput, |
| 165 | + tool_call=lambda args: MyOutput(result=args.query.upper()), |
| 166 | + )] |
| 167 | +``` |
| 168 | + |
| 169 | +`ToolBox.call_tool` validates input and output against the Pydantic schemas automatically (`agent_tools.py:195`). `discover_tools` is called once at init and again lazily if a tool is not found in the cache (`agent_tools.py:104`). |
| 170 | + |
| 171 | +--- |
| 172 | + |
| 173 | +## How to Implement a ChatEngine |
| 174 | + |
| 175 | +Register under `opm.agents.chat` in `pyproject.toml`: |
| 176 | + |
| 177 | +```toml |
| 178 | +[project.entry-points."opm.agents.chat"] |
| 179 | +my-chat-engine = "my_package.chat:MyChatEngine" |
| 180 | +``` |
| 181 | + |
| 182 | +Minimal implementation (`agents.py:195`): |
| 183 | + |
| 184 | +```python |
| 185 | +from ovos_plugin_manager.templates.agents import ChatEngine, AgentMessage, MessageRole |
| 186 | +from typing import List, Optional |
| 187 | + |
| 188 | +class MyChatEngine(ChatEngine): |
| 189 | + def continue_chat(self, messages: List[AgentMessage], |
| 190 | + session_id: str = "default", |
| 191 | + lang: Optional[str] = None, |
| 192 | + units: Optional[str] = None) -> AgentMessage: |
| 193 | + # messages[-1] is the latest user message |
| 194 | + reply = call_my_llm_api([m.__dict__ for m in messages]) |
| 195 | + return AgentMessage(role=MessageRole.ASSISTANT, content=reply) |
| 196 | +``` |
| 197 | + |
| 198 | +For streaming, override `stream_tokens` (token-level) or `stream_sentences` (sentence-level, TTS-ready) (`agents.py:228–278`). The default implementations fall back to `continue_chat`. |
| 199 | + |
| 200 | +--- |
| 201 | + |
| 202 | +## Configuration |
| 203 | + |
| 204 | +Config is passed as a plain `dict` to `__init__`. OPM reads plugin config from the OVOS `Configuration()` singleton under the plugin's entry point name. Standard keys used by most agent plugins: |
| 205 | + |
| 206 | +| Key | Type | Default | Description | |
| 207 | +|---|---|---|---| |
| 208 | +| `lang` | `str` | session lang | BCP-47 language code | |
| 209 | +| `system_prompt` | `str` | `""` | System prompt for `AgentContextManager` plugins (`agents.py:61`) | |
| 210 | +| `context_ttl` | `int` | `120` | Seconds before coreference context is pruned (`agents.py:598`) | |
| 211 | + |
| 212 | +ToolBox-specific keys are documented in each plugin's module docstring. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?v=4&size=40){kind=avatar}
0 commit comments