Skip to content

feat(ai): production agent templates + golden-transcript eval harness - #61

Merged
yezz123 merged 1 commit into
yezz123:mainfrom
nguyenminhduc9988:feat/agent-templates
Jul 12, 2026
Merged

yezz123 merged 1 commit into
yezz123:mainfrom
nguyenminhduc9988:feat/agent-templates

Conversation

@nguyenminhduc9988

Copy link
Copy Markdown
Contributor

Hi Yasser — the PR we discussed over email. Three MIT drop-in agent templates for the LLM Gateway, one level up from dogeapi.ai.examples: typed outputs plus deterministic guardrails, and an eval harness so buyers can customize a template and verify nothing broke.

What's in dogeapi.ai.templates

support_triage — classifies inbound support messages into TriageResult (category / severity 1-4 / summary / suggested_reply / needs_human). Two guardrails beyond the prompt:

  • a deterministic output validator enforces a severity floor: severity-4 results are always needs_human=True, no matter what the model said;
  • a fetch_recent_tickets tool wired through SupportContext deps (plain sequence by default, so it runs with zero infrastructure — swap in your ticket-store query when you drop it in).

rag_answerer — answers over caller-retrieved chunks and refuses to emit claims without a citation: an output validator raises ModelRetry on empty citations or ids outside the retrieval set, and forces insufficient_context=true when retrieval came back empty instead of letting the model improvise. Retrieval is deliberately out of scope so the template stays storage-agnostic (pgvector, Qdrant, whatever).

evals — golden-transcript harness: GoldenCase (exact-match expected field assertions + contains substring checks), JSON loader, run_golden_cases() → EvalReport with a CI-friendly summary(). One crashed case fails that case only. Ships sample goldens for the triage agent.

Conventions followed

  • Built on build_agent(); pydantic_ai stays a lazy import everywhere, so the ai extra is only required once a build_* factory is called (same discipline as agents.py/examples.py). The two tool-defining modules skip from __future__ import annotations (tool signatures must hold real types at runtime for schema building) — noted in their docstrings.
  • Tests are offline respx mocks against the gateway, mirroring test_pydantic_agent.py: 9 new tests covering structured parsing, the severity floor, the citation-retry loop, empty-retrieval refusal, and the harness itself.
  • Full backend suite: 123 passed locally (postgres:17-alpine + redis, uv sync --all-extras). ruff check, ruff format --check, and mypy --strict clean on the new files.

Happy to adjust naming, module placement (templates vs anything you prefer), or docs format — I kept the README delta to one line pending your preference.

🤖 Generated with Claude Code

Adds dogeapi.ai.templates - drop-in, MIT agents built on build_agent(),
one level up from dogeapi.ai.examples:

- support_triage: typed TriageResult with a deterministic severity floor
  (severity-4 always needs_human=True via output validator) and a
  fetch_recent_tickets tool wired through SupportContext deps.
- rag_answerer: CitedAnswer over caller-retrieved chunks; an output
  validator rejects uncited answers or citations outside the retrieval
  set (ModelRetry), and forces insufficient_context=true on empty
  retrieval instead of improvised answers.
- evals: golden-transcript harness (GoldenCase/EvalReport, JSON loader,
  run_golden_cases) so customized templates can be regression-checked;
  ships sample goldens for the triage agent.

Follows the ai-module conventions: pydantic_ai stays a lazy import (the
'ai' extra is only needed when a build_* factory is called), respx-mocked
offline tests, mypy --strict clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@yezz123
yezz123 merged commit 86f6e33 into yezz123:main Jul 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants