AIが捏造した医学引用を、公開前に捕まえる。
Site: https://kgraph57.github.io/evidentia/
The evidentia command verifies every citation in a piece of medical writing against CrossRef, PubMed, OpenAlex, arXiv, and ClinicalTrials.gov and grades each one in a 4-tier classification. The companion agent skill starts with a claim ledger, then runs four named loops (engine, semantic, adversarial, correction) before any A–F score. Built by a board-certified pediatrician.
📖 日本語で読む方は 日本語版 README → README.ja.md へ
Why now: A Lancet audit of 2.5 million biomedical papers (Topaz et al., Lancet 2026;407(10541):1779–1781; doi:10.1016/S0140-6736(26)00603-3) found fabricated references in 1 in 2,828 papers in 2023, 1 in 458 in 2025 (about sixfold), and 1 in 277 in the first 7 weeks of 2026 — about a 10-fold rise from 2023 that tracks the spread of AI writing tools. (The audit screened the PubMed Central open-access subset.) Coverage: STAT (7 May 2026) · Nature · Columbia Nursing · Retraction Watch. A Department of Error was later published (doi:10.1016/s0140-6736(26)01339-5, Jul 2026); rates quoted here follow the original correspondence and STAT reporting.
A fabricated DOI looks exactly like a real one. Evidentia is the open-source tool that resolves each one and tells you which is which.
⚕️ Scope: Evidentia is a pre-publication aid for writers, editors, and researchers — not clinical decision support. It does not diagnose, treat, or replace professional medical judgment.
Homepage: https://kgraph57.github.io/evidentia/
As a command-line tool (no install, no API key):
npx evidentia check your-article.mdAs an agent skill (claim ledger + four verification loops; works because SKILL.md is at skills/medical-fact-check/SKILL.md):
npx skills add kgraph57/evidentiaAs a Claude Code plugin:
/plugin marketplace add kgraph57/evidentia
/plugin install evidentia@evidentiaThen just say: "Fact-check this article" / 「この記事をファクトチェックして」.
As a Codex Desktop plugin:
Marketplace source: https://github.com/kgraph57/evidentia.git
Plugin: evidentia
Here is Evidentia run on a real AI-generated answer about vitamin D and childhood infections — four citations, formatted perfectly, all plausible:
$ npx evidentia check examples/inputs/ai-generated-answer.md
Evidentia: 4 citations — 1 verified, 1 mismatch, 2 hallucinated (75.0% fabrication rate)
[OK ] doi:10.1136/bmj.i6583 — Paper exists and the cited metadata matches the registry record.
[HAL] doi:10.1056/nejmoa2105512 — DOI does not resolve in CrossRef or OpenAlex, and no matching paper was found.
[HAL] pmid:18768876 — Identifier resolves to a different paper ("Trafficking of antigen-specific
CD8+ T lymphocytes…") than the one cited.
[MIS] doi:10.1002/14651858.cd012734 — Paper exists, but cited metadata disagrees with the record (year).
One citation was real. One DOI was invented. One PMID pointed to an unrelated paper. One had the wrong year. A human reviewer would have to check all four by hand. Evidentia did it in seconds. See the full engine report. The skill pipeline on this same file — claim ledger → semantic → adversarial KILL — is walked in examples/case-studies/vitamin-d-adversarial.md.
This is a deliberately tough example. Most carefully written articles score far lower — Evidentia's value is catching the handful that slip through, every time, without fatigue.
Most "citation checkers" stop at "could not verify." Evidentia keeps going — it resolves the identifier and tells you why a citation is suspect:
| Tier | Verdict | Meaning |
|---|---|---|
| ✅ 1 | Verified | The paper exists and the cited title/authors/year/journal match the registry record. |
| Bibliographic mismatch | A real paper exists, but the DOI/PMID is wrong, or the metadata disagrees (a real source cited carelessly — or a fabricated identifier bolted onto a real title). | |
| ❌ 4 | Hallucination | The identifier resolves to nothing, or resolves to a completely different paper. This is the signature of AI-generated text. |
| 🔍 2 | Content review needed | The paper is real, but whether it's used in the right context needs a human or an LLM. Handled by the Evidentia skill, below. |
Evidentia is deliberately split into a part a computer can do perfectly and a part that needs judgment:
1. The engine (CLI + MCP server) — pure, deterministic citation verification. No API key, no LLM, no hallucination of its own. It answers one question with certainty: does this cited paper actually exist, and does the identifier point to it? Use it in a terminal, in CI, or as an MCP tool inside any agent.
2. The skill (Claude Code) — wraps the engine in a claim ledger and four named loops: engine, semantic honesty, adversarial red-team, correction (cap 3). The 15-criteria rubric is one pass inside that pipeline, not the product. Verdicts are KILL / MAJOR / MINOR / PASS. KILL or MAJOR cannot ship as A. This is the "is it used honestly?" layer the engine can't do alone.
You can use either on its own. Together they cover citation existence (deterministic) and citation honesty (appraisal).
The engine answers existence. The skill answers honesty. It does not start by scoring 15 boxes.
Claim ledger first. Extract every testable claim and its attached citation before any lookup. Headlines count. See templates/claim-ledger.md.
Four named loops (each has a stop condition — full rules in skills/medical-fact-check/references/verification-workflow.md):
- Engine —
evidentia check. Retry once if unreachable; then markunresolved, never a guessed Hallucination. Engine output is ground truth for existence. Never override T4 to "probably real." - Semantic — one extra abstract lookup per Tier 1 cite. Does the claim match primary outcome, population, direction of effect? Do not use this loop to upgrade a T3 or T4.
- Adversarial — five lenses + a 10-line checklist. Verdict: KILL / MAJOR / MINOR / PASS. Max 3 passes. Re-enter from the engine, not from scoring.
- Correction — if the author revises, re-run engine → semantic → adversarial. Cap 3, then stop and report what is still open.
KILL or MAJOR cannot ship as A. A KILL (any T4 presented as real, or advice that could cause harm if followed) forces overall score ≤ D — F if fabrication. That is the system working, not a failure of the tool.
Worked example: vitamin D — claim ledger → engine → semantic → KILL on the committed four-citation demo.
flowchart TD
A[Acquire content] --> B[Extract claim ledger]
B --> C[Run evidentia engine]
C --> D[Semantic honesty check]
D --> E[15-criteria appraisal]
E --> F[Adversarial red-team]
F --> G[Score plus report]
G --> H{User revises?}
H -->|yes: max 3| C
H -->|no or cap| I[Stop]
The 15 criteria (one pass inside the pipeline, not the product)
- Evidence level & study design
- Citation & source accuracy (powered by the engine above)
- Statistical interpretation
- Causation vs. correlation
- Bias & conflicts of interest
- Exaggeration & overclaiming
- Target population fit
- Temporal validity
- Jargon–readability balance
- Ethical considerations
- Logical consistency
- Images & figures
- Alternative explanations
- Clinical relevance
- Information completeness
Each item is rated Excellent / Good / Fair / Poor, aggregated into an overall A–F score with a public-health risk level (LOW / MEDIUM / HIGH) — then gated by the adversarial verdict. See skills/medical-fact-check/SKILL.md.
Give any agent the ability to verify citations:
claude mcp add evidentia -- npx -y evidentia-mcpThe server exposes one tool, verify_citations(text), returning the tiered report as Markdown or JSON. JSON responses include both the public 4-tier verdict and a machine-readable lookupVerified / resolverOutcomes trace for agents and CI.
Block a pull request that introduces a fabricated citation. Drop .github/workflows/evidentia.yml into any medical-content repo:
- run: npx evidentia check content/**/*.md --fail-on-fabrication--fail-on-fabrication exits non-zero if any citation is a mismatch or hallucination.
The skill follows the open Agent Skills SKILL.md standard, so it loads in Claude Code today and in any agent that adopts the format. The engine is a plain npm package and an MCP server — usable from Claude Code, Codex CLI, Cursor, or your own scripts.
# one-off, no install
npx evidentia check article.md
# or install globally
npm install -g evidentia
evidentia check article.md --format md --out report.mdevidentia check <file|url|-> Verify citations in a file, web page, or stdin
--format <md|text|json> Output format (default: text)
--out <file> Write the report to a file
--mailto <email> Contact email for the CrossRef/OpenAlex polite pool
--cache <file> Reuse registry HTTP responses from a local JSON cache
--fail-on-fabrication Exit 1 if any citation is mismatch/hallucination (CI)
--offline Extraction only, no network
npx skills add kgraph57/evidentiaWorks because SKILL.md is at skills/medical-fact-check/SKILL.md.
/plugin marketplace add kgraph57/evidentia
/plugin install evidentia@evidentiaOr copy the skill manually:
git clone https://github.com/kgraph57/evidentia.git
cp -r evidentia/skills/medical-fact-check ~/.claude/skills/
For each citation, Evidentia extracts every identifier (DOI, PMID, arXiv, NCT trial ID, ISBN) and any nearby title/author/year, then:
- Resolves the DOI against CrossRef, falling back to OpenAlex.
- Resolves the PMID against PubMed E-utilities.
- Resolves arXiv IDs against the arXiv API.
- Resolves NCT trial IDs against ClinicalTrials.gov.
- If the identifier doesn't resolve, searches by title in OpenAlex — this is how it distinguishes "real paper, wrong DOI" (Tier 3) from "this paper does not exist" (Tier 4).
- Compares the cited title/authors/year against the registry record to catch an identifier that silently points to a different paper.
Each JSON citation also includes lookupVerified (true, false, or unresolvable) and resolverOutcomes (matched, unmatched, unreachable, or skipped, with whether the lookup was keyed by an identifier or a title). This keeps agent workflows auditable without changing the human-facing 4-tier verdict.
It is deliberately careful about what it does not flag: a book (ISBN), a clinical guideline, or any source that isn't indexed in these registries is marked "Content review needed" (Tier 2), never "hallucination" — only a failing DOI/PMID/arXiv/NCT identifier (which is supposed to resolve) earns a fabrication verdict. Identifier-less entries in a reference list are surfaced for review rather than silently skipped.
All registries are free and keyless. Pass --mailto to join the faster "polite pool."
| Input | Result |
|---|---|
| AI-generated answer (real + fabricated mix) | 75% fabrication rate |
| Vitamin D adversarial walkthrough (skill pipeline on the same file) | KILL, score F — two T4 presented as real |
| Clean reference list (all real) | 0% — all verified |
- The engine verifies citation existence and bibliographic accuracy — not whether a real paper is summarized correctly. That semantic check is the skill's job (Tier 2), and even then it relies on what's publicly indexed (abstracts, open-access full text, metadata).
- This is decision support for writers, editors, and researchers — not clinical decision support. It does not diagnose, treat, or replace professional medical judgment.
- Very new papers may not yet be indexed, which can surface as a false "unverified." Re-run later, or pass
--mailtofor the freshest index.
- Batch input —
evidentia check a.md b.md …with an aggregate report (benchmark: 22 cases) - arXiv ID verification, resolver traces, and optional local lookup cache
-
evidentia-bench— grow to 100+ cases and publish per-model fabrication rates - CrossRef/OpenAlex abstract retrieval to assist Tier-2 context checks
- More media presets in the skill
Track it on the pinned roadmap issue.
Issues and PRs welcome — see CONTRIBUTING.md. Security reports: SECURITY.md.
Built by Ken Okamoto, MD — a board-certified pediatrician and medical-AI entrepreneur. Evidentia comes out of the daily problem of separating real evidence from confident-sounding fabrication in AI-assisted medical writing.
Evidentia (the citation-verification tool documented here) is unrelated to "Evidentia" genealogy software.