feat(knowledge): Tavily connector + web-research wrapper with caching (#1234)

### Why this matters

GAIA agents could only reach the web through keyless DuckDuckGo HTML
scraping — no
Tavily, no result caching, and no way to track or cap API spend. This PR
is the Phase 1
foundation for the Knowledge Agent: a `_TAVILY` connector that exposes
Tavily
`search`/`extract` as MCP tools to **all** agents from one
keyring-stored API key, plus a
`gaia.web.tavily` wrapper giving cached, credit-budgeted
search/extract/crawl with an
automatic DuckDuckGo fallback when Tavily isn't configured. Agents get
higher-quality web
research without re-paying for repeat queries or silently blowing past a
credit budget.

### Test plan

- [ ] `python -m pytest tests/unit/test_tavily_wrapper.py` — 16 tests
(mocked SDK): cache hit/TTL, credit ledger, budget warn/block,
DuckDuckGo fallback
- [ ] `python -m pytest tests/unit/connectors/test_catalog_docs_url.py`
— the new connector's `docs_url` resolves
- [ ] `python util/lint.py --black --isort` — clean
- [ ] With a key: `gaia connectors configure mcp-tavily --set
TAVILY_API_KEY=tvly-…` then `gaia knowledge search "…"` returns Tavily
results; without it, the same command falls back to DuckDuckGo
- [ ] `gaia knowledge usage` prints the credit ledger

### Open questions for reviewers

- **Catalog scope:** `_TAVILY` has no built-in agent consumer yet (the
Knowledge Agent lands in a later phase), so there's no
`REQUIRED_CONNECTORS` wiring. Acceptable for a foundation PR, or should
the entry wait? (@kovtcharov-amd)
- **Async `crawl`:** sync `TavilyClient` has `crawl`;
`AsyncTavilyClient` doesn't yet — deferred unless you want parity now.

Phase 1 of #1141.

---------

Co-authored-by: Tomasz Iniewicz <itomek@users.noreply.github.com>

Author: TravisHaa

docs/connectors/tavily.mdx

@@ -0,0 +1,107 @@
+---
+title: "Tavily"
+icon: "globe"
+description: "Give GAIA agents web search and content extraction via Tavily."
+---
+
+<Info>
+  **Connector ID:** `mcp-tavily` · **Type:** `mcp_server` · **Catalog entry:** [`src/gaia/connectors/catalog/mcp_servers.py`](https://github.com/amd/gaia/blob/main/src/gaia/connectors/catalog/mcp_servers.py)
+</Info>
+
+## What you'll need
+
+[Tavily](https://tavily.com) is a web-search API built for AI agents. The
+connector is an **MCP server** — GAIA spawns the
+[`tavily-mcp`](https://github.com/tavily-ai/tavily-mcp) process on demand via
+`npx` and routes tool calls (`tavily-search`, `tavily-extract`) through it, so
+the tools become available to **all** GAIA agents.
+
+It needs a single secret: a **Tavily API key**. You'll create one, paste it
+into GAIA once, and you're done. The key lives encrypted in your OS keyring;
+the MCP server reads it via a `$keyring` reference at launch.
+
+## Step 1 — Get an API key
+
+1. Sign in at <a href="https://app.tavily.com/" target="_blank">app.tavily.com</a>.
+2. Copy your API key from the dashboard. It starts with `tvly-` followed by a
+   string of characters (e.g. `tvly-AbCd…`). If your key doesn't start with
+   `tvly-`, you're looking at the wrong value.
+
+The free tier includes a monthly credit allowance; a basic search costs 1
+credit and an advanced search costs 2.
+
+## Step 2 — Configure GAIA
+
+**From the CLI:**
+
+```bash
+gaia connectors configure mcp-tavily --set TAVILY_API_KEY=tvly-...
+```
+
+**From the Agent UI:**
+
+1. Launch the Agent UI: `gaia chat --ui`.
+2. **Settings** (gear) → **Connections** → click the **Tavily** tile.
+3. Paste the key into the **Tavily API Key** field and click **Save**.
+
+Either path stores the key in your OS keyring (a single slot, distinct from
+other connectors) and writes a `$keyring` reference into
+`~/.gaia/mcp_servers.json` — the key never lives in plaintext on disk.
+
+## Step 3 — Use it
+
+Once configured, the `tavily-search` / `tavily-extract` MCP tools are available
+to any agent you grant them to:
+
+```bash
+gaia connectors grants grant mcp-tavily builtin:chat --scopes "*"
+```
+
+GAIA also ships a Python wrapper (`gaia.web.tavily`) used by web-research
+workflows, with response caching, a credit budget, and a CLI:
+
+```bash
+gaia knowledge search "AMD ROCm latest release" --max-results 5
+gaia knowledge usage          # show credits spent
+```
+
+<Note>
+  If the connector isn't configured, `gaia knowledge search` and the wrapper
+  fall back to a keyless DuckDuckGo search — so search works out of the box,
+  and Tavily simply upgrades its quality and adds `extract`/`crawl`.
+</Note>
+
+## Common issues
+
+### `Unauthorized` / `401` from the MCP server
+
+The key in your keyring is wrong or revoked. Click **Disconnect** on the tile
+(or `gaia connectors disconnect mcp-tavily`) and re-add a fresh key.
+
+### `npx: command not found`
+
+`tavily-mcp` is launched via `npx`. Install Node 18+ and ensure `npx` is on
+your `PATH`:
+
+```bash
+node --version   # must be >= 18
+which npx        # must resolve to a real path
+```
+
+### Budget exceeded
+
+`gaia knowledge` warns (if nearing the budget) then blocks once a session passes its `--budget` credit cap. Raise the cap, or pass `--no-block` to warn and proceed instead of blocking.
+
+## Revoking access
+
+- **From GAIA:** Settings → Connections → Tavily → **Disconnect** (or
+  `gaia connectors disconnect mcp-tavily`). The key is removed from the keyring
+  and the entry is dropped from `mcp_servers.json`.
+- **From Tavily:** rotate or delete the key in your
+  [Tavily dashboard](https://app.tavily.com/).
+
+## See also
+
+- [Connectors overview](/connectors)
+- [Tavily documentation](https://docs.tavily.com/)
+- [Connectors security model](/security/connections)

docs/docs.json

@@ -92,6 +92,7 @@
                   "connectors/google",
                   "connectors/microsoft",
                   "connectors/github",
+                  "connectors/tavily",
                   "security/connections",
                   "security/connectors"
                 ]

docs/reference/cli.mdx

@@ -1997,6 +1997,55 @@ gaia cache clear --all
 
 ---
 
+### Knowledge Command
+
+Web research via [Tavily](https://tavily.com), with SQLite result caching, a per-session
+credit budget, and an automatic keyless DuckDuckGo fallback when the `mcp-tavily`
+connector isn't configured. See the [Tavily connector](/connectors/tavily).
+
+```bash
+gaia knowledge {search,extract,usage} [OPTIONS]
+```
+
+**Actions:**
+
+| Action | Description |
+|--------|-------------|
+| `search` | Web search for a query. Falls back to DuckDuckGo when Tavily isn't configured. |
+| `extract` | Extract clean content from one or more URLs. Requires the Tavily connector. |
+| `usage` | Print cached credit-usage totals (per operation + session total). |
+
+**Options:**
+
+| Flag | Type | Applies to | Description |
+|------|------|------------|-------------|
+| `--max-results` | integer | `search` | Maximum results to return (default: 5) |
+| `--depth` | `basic` \| `advanced` | `search`, `extract` | Depth; `advanced` costs more credits (default: `basic`) |
+| `--budget` | integer | `search`, `extract` | Credit cap for the session; omit for unlimited |
+| `--no-block` | flag | `search`, `extract` | Warn instead of blocking when the budget cap is exceeded |
+
+**Examples:**
+
+<CodeGroup>
+```bash Search the web
+gaia knowledge search "AMD ROCm latest release" --max-results 5
+```
+
+```bash Extract page content
+gaia knowledge extract https://example.com/post
+```
+
+```bash Show credit usage
+gaia knowledge usage
+```
+</CodeGroup>
+
+`search` automatically degrades to DuckDuckGo when the `mcp-tavily` connector isn't
+configured; `extract` requires Tavily and raises an actionable error otherwise. Configure
+the connector with `gaia connectors configure mcp-tavily --set TAVILY_API_KEY=tvly-...`.
+
+---
+
 ### Kill Command
 
 Terminate processes running on specific ports.

setup.py

@@ -113,6 +113,7 @@
         "python-multipart>=0.0.9",
         # gaia connectors is a base CLI command; keyring is its OS credential store (OAuth tokens #915). #1621
         "keyring>=24.0.0,<26.0.0",
+        "tavily-python>=0.5.0",
     ],
     extras_require={
         "image": [

src/gaia/cli.py

@@ -1912,6 +1912,60 @@ def build_parser():
 
     telegram_parser.set_defaults(action="telegram")
 
+    # Knowledge command — web research via the Tavily wrapper
+    knowledge_parser = subparsers.add_parser(
+        "knowledge",
+        help="Web research via Tavily (search|extract|usage), with caching and a credit budget",
+    )
+    knowledge_subparsers = knowledge_parser.add_subparsers(
+        dest="knowledge_action", help="knowledge action to perform"
+    )
+
+    k_search = knowledge_subparsers.add_parser("search", help="Run a web search")
+    k_search.add_argument("query", help="Search query")
+    k_search.add_argument(
+        "--max-results", type=int, default=5, help="Max results (default: 5)"
+    )
+    k_search.add_argument(
+        "--depth",
+        choices=("basic", "advanced"),
+        default="basic",
+        help="Search depth — advanced costs more credits (default: basic)",
+    )
+    k_search.add_argument(
+        "--budget",
+        type=int,
+        default=None,
+        help="Credit cap for this session; omit for unlimited",
+    )
+    k_search.add_argument(
+        "--no-block",
+        action="store_true",
+        help="Warn instead of blocking when the budget cap is exceeded",
+    )
+
+    k_extract = knowledge_subparsers.add_parser(
+        "extract", help="Extract clean content from one or more URLs (requires Tavily)"
+    )
+    k_extract.add_argument("urls", nargs="+", help="One or more URLs to extract")
+    k_extract.add_argument(
+        "--depth",
+        choices=("basic", "advanced"),
+        default="basic",
+        help="Extract depth (default: basic)",
+    )
+    k_extract.add_argument(
+        "--budget", type=int, default=None, help="Credit cap for this session"
+    )
+    k_extract.add_argument(
+        "--no-block",
+        action="store_true",
+        help="Warn instead of blocking when the budget cap is exceeded",
+    )
+
+    knowledge_subparsers.add_parser("usage", help="Show cached credit-usage totals")
+    knowledge_parser.set_defaults(action="knowledge")
+
     # Add model download command
     download_parser = subparsers.add_parser(
         "download",
@@ -4167,6 +4221,11 @@ def main():
         handle_cache_command(args)
         return
 
+    # Handle Knowledge command (Tavily web research)
+    if args.action == "knowledge":
+        handle_knowledge_command(args)
+        return
+
     # Handle Memory command
     if args.action == "memory":
         handle_memory_command(args)
@@ -5094,6 +5153,69 @@ def handle_blender_command(args):
         sys.exit(1)
 
 
+def _print_knowledge_usage(client):
+    """Print a one-line credit-usage summary for a Tavily client."""
+    usage = client.usage()
+    cap = usage["cap"]
+    cap_str = "unlimited" if cap is None else str(cap)
+    print(f"\n💳 Credits used: {usage['total_credits']} (cap: {cap_str})")
+
+
+def handle_knowledge_command(args):
+    """Handle `gaia knowledge` — Tavily web research with caching + budget.
+
+    Args:
+        args: Parsed command-line arguments
+    """
+    action = getattr(args, "knowledge_action", None)
+    if action is None:
+        print("❌ Error: No knowledge action specified")
+        print("Available actions: search, extract, usage")
+        print("Run 'gaia knowledge --help' for more information")
+        return
+
+    from gaia.web.tavily import (
+        BudgetConfig,
+        TavilyBudgetExceeded,
+        TavilyClient,
+        TavilyConfigError,
+    )
+
+    budget = BudgetConfig(
+        cap=getattr(args, "budget", None),
+        block=not getattr(args, "no_block", False),
+    )
+    client = TavilyClient(budget=budget)
+    try:
+        if action == "search":
+            result = client.search(
+                args.query, search_depth=args.depth, max_results=args.max_results
+            )
+            source = result.get("source", "tavily")
+            print(f"\n=== Results for {args.query!r} (source: {source}) ===")
+            for i, r in enumerate(result.get("results", []), 1):
+                print(f"{i}. {r.get('title', '')}")
+                print(f"   {r.get('url', '')}")
+                content = r.get("content") or r.get("snippet") or ""
+                if content:
+                    print(f"   {content[:200]}")
+            _print_knowledge_usage(client)
+        elif action == "extract":
+            result = client.extract(args.urls, extract_depth=args.depth)
+            print(json.dumps(result, indent=2))
+            _print_knowledge_usage(client)
+        elif action == "usage":
+            _print_knowledge_usage(client)
+    except TavilyBudgetExceeded as e:
+        print(f"🛑 {e}")
+        sys.exit(1)
+    except TavilyConfigError as e:
+        print(f"❌ {e}")
+        sys.exit(1)
+    finally:
+        client.close()
+
+
 def handle_cache_command(args):
     """Handle the cache management command.
 

src/gaia/connectors/catalog/mcp_servers.py

@@ -49,6 +49,30 @@
     ),
 )
 
+_TAVILY = ConnectorSpec(
+    id="mcp-tavily",
+    display_name="Tavily",
+    icon="🛜",
+    category="dev-tools",
+    tier=1,
+    type="mcp_server",
+    description="Web search and content extraction for agents through the Tavily API.",
+    docs_url="https://amd-gaia.ai/docs/connectors/tavily",
+    mcp_command="npx",
+    mcp_args=("-y", "tavily-mcp@latest"),
+    mcp_env_keys=("TAVILY_API_KEY",),
+    config_schema=(
+        ConfigField(
+            key="TAVILY_API_KEY",
+            label="Tavily API Key",
+            kind="secret",
+            placeholder="tvly-…",
+            help_md="Get an API key from your [Tavily dashboard](https://app.tavily.com/).",
+            secret=True,
+        ),
+    ),
+)
+
 _MEMORY = ConnectorSpec(
     id="mcp-memory",
     display_name="Memory",
@@ -79,6 +103,7 @@
 
 _ALL_SPECS = (
     _GITHUB,
+    _TAVILY,
     _MEMORY,
     _GIT,
 )