Repository navigation
feat(ai): production agent templates + golden-transcript eval harness - #61
Merged
Merged
Conversation
Adds dogeapi.ai.templates - drop-in, MIT agents built on build_agent(), one level up from dogeapi.ai.examples: - support_triage: typed TriageResult with a deterministic severity floor (severity-4 always needs_human=True via output validator) and a fetch_recent_tickets tool wired through SupportContext deps. - rag_answerer: CitedAnswer over caller-retrieved chunks; an output validator rejects uncited answers or citations outside the retrieval set (ModelRetry), and forces insufficient_context=true on empty retrieval instead of improvised answers. - evals: golden-transcript harness (GoldenCase/EvalReport, JSON loader, run_golden_cases) so customized templates can be regression-checked; ships sample goldens for the triage agent. Follows the ai-module conventions: pydantic_ai stays a lazy import (the 'ai' extra is only needed when a build_* factory is called), respx-mocked offline tests, mypy --strict clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
yezz123
approved these changes
Jul 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Yasser — the PR we discussed over email. Three MIT drop-in agent templates for the LLM Gateway, one level up from
dogeapi.ai.examples: typed outputs plus deterministic guardrails, and an eval harness so buyers can customize a template and verify nothing broke.What's in
dogeapi.ai.templatessupport_triage— classifies inbound support messages intoTriageResult(category / severity 1-4 / summary / suggested_reply / needs_human). Two guardrails beyond the prompt:needs_human=True, no matter what the model said;fetch_recent_ticketstool wired throughSupportContextdeps (plain sequence by default, so it runs with zero infrastructure — swap in your ticket-store query when you drop it in).rag_answerer— answers over caller-retrieved chunks and refuses to emit claims without a citation: an output validator raisesModelRetryon empty citations or ids outside the retrieval set, and forcesinsufficient_context=truewhen retrieval came back empty instead of letting the model improvise. Retrieval is deliberately out of scope so the template stays storage-agnostic (pgvector, Qdrant, whatever).evals— golden-transcript harness:GoldenCase(exact-matchexpectedfield assertions +containssubstring checks), JSON loader,run_golden_cases()→EvalReportwith a CI-friendlysummary(). One crashed case fails that case only. Ships sample goldens for the triage agent.Conventions followed
build_agent();pydantic_aistays a lazy import everywhere, so theaiextra is only required once abuild_*factory is called (same discipline asagents.py/examples.py). The two tool-defining modules skipfrom __future__ import annotations(tool signatures must hold real types at runtime for schema building) — noted in their docstrings.test_pydantic_agent.py: 9 new tests covering structured parsing, the severity floor, the citation-retry loop, empty-retrieval refusal, and the harness itself.uv sync --all-extras).ruff check,ruff format --check, andmypy --strictclean on the new files.Happy to adjust naming, module placement (
templatesvs anything you prefer), or docs format — I kept the README delta to one line pending your preference.🤖 Generated with Claude Code