- Lyuboslav Lyubenov — X: https://x.com/LuboslavLubeno1
- Radoslav Radev (Independent Auditor / Evaluator) — X: https://x.com/radev_eth
- Kann - X: https://x.com/KannAudits
- Lyuboslav Lyubenov: study conception and design, dataset selection, tool orchestration, data analysis, manuscript drafting.
- Radoslav Radev: independent blind scoring of all tool findings, per-finding technical notes, quality ratings used in the analysis.
We evaluated three web-based AI-powered smart contract audit tools against judge-adjudicated ground truth from three public Sherlock contests. Using one independent auditor and a strict matching policy, we measured precision, recall, and quality of findings. Results show no single tool achieved both high precision and high recall across contests: Several candidate tools were excluded for operational/access reasons. We provide measured per-tool results, representative case studies, and practical recommendations.
(These are listed up-front for transparency — all names are tools attempted or considered during the study. Tools released after Aug 2025 or private/invitation only were out of scope.)
-
AlmanaxAI — INCLUDED
- Reason: Web-based; accepted projects and produced outputs for all three contests.
- Summary: Few findings, shallow checks, one confirmed TP on Crestal.
-
AuditAgent (Nethermind) — INCLUDED
- Reason: Web-based; accepted projects and produced extensive reports.
- Summary: High volume of findings; best aggregate recall but high false-positive rate.
-
SavantChat (Savant / Savant.chat) — INCLUDED
- Reason: Web-based; ran successfully and produced detailed reports/PoCs.
- Summary: High recall in Crestal with deep PoCs; verbose/duplicative outputs increased FP counts.
-
LISA — EXCLUDED
- Reason: Produced no actionable vulnerabilities; outputs resembled generic assistant text; 3-file upload limit impractical.
-
Bughunter.live — EXCLUDED
- Reason: Platform repeatedly failed to run projects (errors); unstable across runs.
-
Finite Monkey — EXCLUDED
- Reason: Could not compile / did not start for projects attempted.
-
Hound — EXCLUDED
- Reason: Stalled, long-running runs (example: one run consumed >55M tokens, >28 hours, >$150 across attempts) with no usable results.
-
Solidity Scan — EXCLUDED
- Reason: Lower-tier behavior only surfaced issues detectable by known static analyzers (ChaseTheLight, Aderyn); lacked AI-specific added value.
-
QuillAI_Network — EXCLUDED
- Reason: Credit system prevented runs (insufficient credits; no purchase option available).
-
Quant Pulsar — EXCLUDED
- Reason: Outputs similar to static analyzers; no discernible AI-specific benefit.
- Introduction
- Methods
- Results
- Per-contest summary (yBOLD, Crestal, CAP)
- Aggregate summary
- Discussion
- Recommendations
- Conclusion & next steps
- Appendix: Per-tool per-finding mapping table (machine-friendly)
AI-assisted auditing tools promise faster, scalable identification of smart-contract vulnerabilities. However, claims about their effectiveness are often anecdotal or unstandardized. This pilot study provides a transparent snapshot of current AI audit tool performance in realistic contest-grade codebases. Primary research questions:
- How accurate are current AI audit tools in identifying judge-adjudicated vulnerabilities?
- What precision vs recall trade-offs do tools present?
- Which vulnerability categories are commonly missed or misidentified?
- What operational barriers impede tool use in practice?
- Selected 3 public Sherlock contests (budget-constrained pilot):
- Yearn yBOLD (nSLOC ≈ 475) — https://audits.sherlock.xyz/contests/977
- Crestal Network (nSLOC ≈ 508) — https://audits.sherlock.xyz/contests/755
- CAP protocol (nSLOC ≈ 2,405) — https://audits.sherlock.xyz/contests/990
- Ground truth: only judge-adjudicated (accepted) issues from each contest’s public judging threads (first/official accepted issues).
- Tools evaluated end-to-end: AlmanaxAI, AuditAgent (Nethermind), SavantChat (web-based, latest builds at evaluation start Aug 2025).
- Tools were run via their default web UI/project upload flows. No custom configs were used.
- An independent auditor (single evaluator) matched each tool finding to judge-adjudicated issues using semantic/issue-root matching.
- Strict binary matching for precision/recall: partially-correct or tangential findings were counted as false positives for the primary metrics.
- Primary metrics:
- Precision = TP / (TP + FP)
- Recall = TP / (Total judge-accepted issues for the contest)
- Quality scoring (auditor): 0–4 scale:
- 4: Valid vulnerability, accurate description, correct severity
- 3: Valid vulnerability, mostly accurate description, minor issues
- 2: Valid vulnerability with poor description OR reasonable false positive
- 1: Clear false positive but shows code understanding
- 0: Nonsense/irrelevant
- Only one auditor scored findings; inter-rater reliability was not measured.
- Small dataset (3 contests) — pilot scope only; not fully generalizable.
- Public contest data may have been used in tool training data, potentially inflating performance.
- Tools were run only once per contest; no multiple runs to sample variability.
- Total judge-adjudicated issues across contests: 20 (yBOLD: 3, Crestal: 7, CAP: 10).
- For each contest we report per-tool findings: number of findings, TP, FP, precision, recall, average quality (where available).
Ground truth examples:
- #154 First-depositor theft via share inflation
sherlock-audit/2025-05-yearn-ybold-judging#154 - #47 estimatedTotalAssets() omits on-hand COLL (pps arbitrage)
sherlock-audit/2025-05-yearn-ybold-judging#47 - #90 No maximum on dustThreshold allows management to lock user funds
sherlock-audit/2025-05-yearn-ybold-judging#90
| Tool | Findings | TP | FP | Precision | Recall | Avg Quality |
|---|---|---|---|---|---|---|
| AlmanaxAI | 4 | 0 | 4 | 0.00 | 0.00 | ~1.5 |
| AuditAgent | 14 | 1 (#90) | 13 | 0.07 | 0.33 | ~1.2 |
| SavantChat | 3 | 0 | 3 | 0.00 | 0.00 | ~1.0 |
Ground truth (examples; full links in Appendix):
- Issues include #10, #11, #12, #46, #205, #225, #235 (see judge thread links in Appendix).
| Tool | Findings | TP | FP | Precision | Recall | Avg Quality |
|---|---|---|---|---|---|---|
| AlmanaxAI | 2 | 1 (#10) | 1 | 50.0% | 14.3% | 4.0 |
| AuditAgent | 69 | 5 | 64 | 7.25% | 71.4% | ~3.6 |
| SavantChat | ~17 | 7 | ~10 | ~41% | 100% | 3–4 |
Notes: SavantChat achieved full recall for this contest with detailed PoCs but verbose/duplicative output increased FP counts under strict rules. AuditAgent covered most TPs but with heavy noise. AlmanaxAI had one solid TP.
Ground truth (examples; full links in Appendix): judge-accepted issues enumerated in contest thread (10 issues).
| Tool | Findings | TP | FP | Precision | Recall | Avg Quality |
|---|---|---|---|---|---|---|
| AlmanaxAI | 11 | 0 | 11 | 0.00 | 0.00 | ~1.0 |
| AuditAgent | 110 | 2 | 108 | ~0.018 | 0.20 | ~1.5 |
| SavantChat | 19 | 0 | 19 | 0.00 | 0.00 | ~1.0 |
| Tool | Findings | TP | FP | Precision | Recall |
|---|---|---|---|---|---|
| AlmanaxAI | 17 | 1 | 16 | ≈5.9% | 5.0% |
| AuditAgent | 193 | 8 | 185 | ≈4.1% | 40.0% |
| SavantChat | 39 | 7 | 32 | ≈17.9% | 35.0% |
Aggregate results excluding Crestal (two contests only):
| Tool | Findings | TP | FP | Precision | Recall |
|---|---|---|---|---|---|
| AlmanaxAI | 15 | 0 | 15 | 0.00 | 0.00 |
| AuditAgent | 124 | 3 | 121 | ~2.4% | ~23.1% |
| SavantChat | 22 | 0 | 22 | 0.00 | 0.00 |
High-level observation: Tools performed best on Crestal because its bugs were localized, syntactic/state-machine/auth issues that are easy for pattern-based detectors to spot, which significantly boosted aggregate scores. AuditAgent produced the highest recall but with massive noise; SavantChat had the best single-contest recall (Crestal) with deep PoCs but needs deduplication; AlmanaxAI had limited coverage.
- Tools present different trade-offs: higher recall typically incurs higher false-positive volume and triage cost.
- Teams must decide if they can absorb triage effort (favoring higher recall) or need leaner, more precise outputs.
- Economic/accounting reasoning (shares/pps, cross-contract invariants) is a repeated blind spot.
- Template over-reporting creates many false positives (initializers, tx.origin, generic reentrancy flags).
- Verbose duplication inflates FP counts (same root cause reported multiple ways).
- Operational limitations (upload limits, credits, invites, instability) significantly hinder real-world use.
- Tools performed best on Crestal because its bugs were localized, syntactic/state-machine/auth issues that are easy for pattern-based detectors to spot.
- Tools struggled with Yearn/CAP's cross-contract economic/accounting logic, which requires deeper reasoning.
Current web-based AI audit tools can surface real issues but still suffer from false positives, duplication, and blind spots around economic/design reasoning. The tools primarily find template issues (access control, reentrancy patterns, basic math) but cannot reliably discover business logic flaws, cross-contract integration issues, or more complex economic vulnerabilities. AuditAgent provided the best recall in this pilot, SavantChat provided high-quality PoCs for one contest, and AlmanaxAI provided limited coverage. These tools are far from actually discovering significant, novel bugs in production systems.
Notes about this table
- The mapping table below was constructed from the independent auditor’s summaries and verbatim notes you provided. It is not an exhaustive dump of all tool output (raw outputs were not available to the authors).
- Columns: Tool | Contest | Finding short text | Match (TP / FP / Excluded) | Auditor score (0–4 or N/A) | Link to judged issue (if TP).
- Where the auditor provided a numeric score for a finding, that value is used. Where the auditor provided a qualitative rating only, the Score column may be "N/A" or a small range (as noted).
- If you want this exported to CSV/JSON, tell me and I will produce downloadable machine-readable files.
| Tool | Contest | Finding short text | Match | Auditor score (0–4) | Judged issue link (if TP) |
|---|---|---|---|---|---|
| AlmanaxAI | yBOLD | Missing access control on newStrategy (factory) |
FP | 1 | N/A |
| AlmanaxAI | yBOLD | Unbounded loop in aprAfterDebtChange |
FP | 2 | N/A |
| AlmanaxAI | yBOLD | No input validation for asset address | FP | 1 | N/A |
| AlmanaxAI | yBOLD | MIN_INT negation overflow in oracle math | FP | 2 | N/A |
| AuditAgent | yBOLD | No upper limit for dustThreshold (dustThreshold cap) |
TP | 3 | sherlock-audit/2025-05-yearn-ybold-judging#90 |
| AuditAgent | yBOLD | Auction starting price mis-scaled (extra WAD division) | FP | 1 | N/A |
| AuditAgent | yBOLD | Unbounded loop in APR oracle (claim) | FP | 1 | N/A |
| AuditAgent | yBOLD | MIN_INT overflow (claim) | FP | 1 | N/A |
| SavantChat | yBOLD | _tendTrigger can freeze collateral if totalAssets == 0 (claim) |
FP | 1 | N/A |
| SavantChat | yBOLD | "Critical miscalculation of auction starting price" (claim) | FP | 1 | N/A |
| SavantChat | yBOLD | Missing ERC20 allowance for SP (claim) | FP | 1 | N/A |
| AlmanaxAI | Crestal | Missing authorization on payWithERC20 |
TP | 4 | sherlock-audit/2025-03-crestal-network-judging#10 |
| AlmanaxAI | Crestal | Missing access control on upgradeProject (not in judged set) |
FP | 1 | N/A |
| AuditAgent | Crestal | Unrestricted payWithERC20 (detailed) |
TP | 4 | sherlock-audit/2025-03-crestal-network-judging#10 |
| AuditAgent | Crestal | Meta-tx / EIP-712 replay & missing parameter binding across WithSig paths |
TP | 4 | sherlock-audit/2025-03-crestal-network-judging#11 |
| AuditAgent | Crestal | Public pickup / DoS variants (pickup stuck) | TP | 3 | sherlock-audit/2025-03-crestal-network-judging#12 |
| AuditAgent | Crestal | Signature binding across actions/types (various) | TP | 4 | sherlock-audit/2025-03-crestal-network-judging#225 |
| AuditAgent | Crestal | Worker registry/list problems (dupes/empty keys, unbounded list) | TP | 3 | sherlock-audit/2025-03-crestal-network-judging#235 |
| AuditAgent | Crestal | Storage layout / misc template issues (representative) | FP | 1 | N/A |
| SavantChat | Crestal | payWithERC20 arbitrary transfer (#10) | TP | 3–4 | sherlock-audit/2025-03-crestal-network-judging#10 |
| SavantChat | Crestal | Replay / param-binding holes (#11) | TP | 3–4 | sherlock-audit/2025-03-crestal-network-judging#11 |
| SavantChat | Crestal | Public pickup DoS family (#12) | TP | 3–4 | sherlock-audit/2025-03-crestal-network-judging#12 |
| SavantChat | Crestal | Signature binding across actions/types (#225) | TP | 3–4 | sherlock-audit/2025-03-crestal-network-judging#225 |
| SavantChat | Crestal | Worker list/dupe keys/unbounded growth (#235) | TP | 3–4 | sherlock-audit/2025-03-crestal-network-judging#235 |
| SavantChat | Crestal | Additional duplicates/overlapping stylistic state-machine fragility findings | FP | 2 | N/A |
| AlmanaxAI | CAP | Claims across multiple files (example: unauthenticated mint() in Token.sol) |
FP | 1 | N/A |
| AlmanaxAI | CAP | Role-ID collision via bit-OR packing (claim) | FP | 1 | N/A |
| AlmanaxAI | CAP | Uninitialized ERC1967 proxy (claim) | FP | 1 | N/A |
| AlmanaxAI | CAP | tx.origin identity / reentrancy via decimals() (claims) |
FP | 1 | N/A |
| AlmanaxAI | CAP | Bit-shift overflow in reserve index (claim) | FP | 1 | N/A |
| AuditAgent | CAP | VaultAdapter: zero rate when utilization > kink (multiplier stuck at 0) | TP | 4 | sherlock-audit/2025-07-cap-judging#66 |
| AuditAgent | CAP | Liquidation payout bounds — liquidator loss / payout risk | TP | 3 | sherlock-audit/2025-07-cap-judging#316 |
| AuditAgent | CAP | Large set of template findings (proxy init races, tx.origin, rescueERC20 missing access) | FP | 1 | N/A |
| SavantChat | CAP | Multiple ERC-4626 / FractionalReserve cluster claims (accounting edge cases) | FP | 1–2 | N/A |
| SavantChat | CAP | Reentrancy claims (Vault.burn, BorrowLogic.borrow) relying on ERC-777 / malicious token assumptions | FP | 1 | N/A |
| SavantChat | CAP | Oracle/health math div-by-zero / bit-map shift claims (not matching adjudicated root causes) | FP | 1 | N/A |
| Tool | Contest | Finding short text | Match | Auditor score | Judged issue link |
|---|---|---|---|---|---|
| LISA | N/A | Platform produced non-actionable assistant-like outputs; 3-file upload limit | Excluded | N/A | N/A |
| Bughunter.live | N/A | Platform instability/errors prevented running projects | Excluded | N/A | N/A |
| Finite Monkey | N/A | Could not compile / start | Excluded | N/A | N/A |
| Hound | N/A | Stalled long-running runs; impractical cost and no results | Excluded | N/A | N/A |
| Solidity Scan | N/A | Only flags issues visible to known static analyzers (no AI added value) | Excluded | N/A | N/A |
| QuillAI_Network | N/A | Credit system prevented runs; no practical purchase option | Excluded | N/A | N/A |
| Octane Security | N/A | Invitation-only; no invite obtained | Excluded | N/A | N/A |
| Quant Pulsar | N/A | Outputs similar to static analyzers; no detectable AI benefit | Excluded | N/A | N/A |