[ast-daemon]

Summary

Implements 7 new mutations targeting OWASP LLM Top 10 2025 categories for comprehensive LLM security testing.

OWASP LLM Top 10 Coverage

LLM01 - Prompt Injection (3 mutations)

• BadLikertJudgeMutation: Likert scale framing attack with 60%+ reported bypass rate
• CrescendoMutation: Multi-turn gradual escalation attack (configurable 3-7 turns)
• DeceptiveDelightMutation: Positive framing bypass technique

LLM05 - Insecure Output Handling (1 mutation)

• OutputInjectionMutation: Tests XSS, SQLi, and command injection payloads

LLM06 - Excessive Agency (1 mutation)

• ExcessiveAgencyMutation: Tests unauthorized action claims (file ops, API calls, code exec)

LLM07 - System Prompt Leakage (1 mutation)

• SystemExtractionMutation: System prompt extraction attempts via multiple techniques

LLM09 - Misinformation (1 mutation)

• HallucinationTrapMutation: Triggers confident fabrication via non-existent references

CLI Enhancements

# Test specific OWASP categories
artemis redteam scenario.yaml --owasp LLM01,LLM05

# Full OWASP compliance scan
artemis redteam scenario.yaml --owasp-full

# Filter by minimum severity
artemis redteam scenario.yaml --min-severity high

Features

• CVSS-like scoring: Each mutation includes detailed severity metrics
• OWASP mapping: Mutations are mapped to their corresponding OWASP categories
• Configurable options: All mutations support customization
• Comprehensive tests: 35+ new tests with 100% coverage on new code

Test Results

51 pass
0 fail
148 expect() calls

Files Changed

• packages/redteam/src/mutations/bad-likert-judge.ts
• packages/redteam/src/mutations/crescendo.ts
• packages/redteam/src/mutations/deceptive-delight.ts
• packages/redteam/src/mutations/output-injection.ts
• packages/redteam/src/mutations/excessive-agency.ts
• packages/redteam/src/mutations/system-extraction.ts
• packages/redteam/src/mutations/hallucination-trap.ts
• packages/redteam/src/mutations/index.ts (exports + OWASP helpers)
• packages/redteam/src/mutations/owasp.test.ts
• packages/cli/src/commands/redteam.ts (CLI flags)
• packages/redteam/src/severity.ts (CVSS scores)

[greptile-apps]

Greptile Summary

Adds 7 new mutations targeting OWASP LLM Top 10 2025 categories with CVSS-like scoring and CLI integration. Implementation is solid with comprehensive test coverage, but OWASP category mappings are incorrect in multiple files.

Key Changes

• 3 LLM01 mutations: bad-likert-judge (Likert scale framing), crescendo (multi-turn escalation), deceptive-delight (positive framing)
• LLM02 mutation: output-injection (XSS, SQLi, command injection payloads) - incorrectly labeled as LLM05
• LLM08 mutation: excessive-agency (unauthorized action claims) - incorrectly labeled as LLM06
• LLM06 mutation: system-extraction (system prompt leakage) - incorrectly labeled as LLM07
• LLM09 mutation: hallucination-trap (confident fabrication triggers)
• CLI flags: --owasp, --owasp-full, --min-severity for filtered testing
• CVSS-like scoring with evasion effectiveness and detectability metrics

Issues Found

Critical: OWASP category numbers misaligned with 2025 specification across index.ts, mutation files, and severity.ts. The mutations target the right vulnerabilities but are labeled with wrong LLM## numbers. This will cause --owasp LLM06 to run the wrong mutations.

Confidence Score: 3/5

• Safe to merge after fixing OWASP category mapping bugs
• Excellent implementation quality with comprehensive tests and well-designed mutations, but systematic OWASP category labeling errors will cause incorrect behavior when users filter by category
• Pay close attention to packages/redteam/src/mutations/index.ts where OWASP_CATEGORIES mapping needs correction across LLM02-LLM08

Important Files Changed

Filename	Overview
packages/redteam/src/mutations/index.ts	Maps OWASP LLM categories incorrectly - several mutations assigned to wrong categories
packages/redteam/src/mutations/output-injection.ts	Comprehensive output injection tests covering XSS, SQLi, command injection with realistic payloads
packages/redteam/src/mutations/excessive-agency.ts	Tests unauthorized action claims across file system, network, database, and admin contexts
packages/redteam/src/mutations/system-extraction.ts	System prompt extraction using 8 techniques including roleplay, debug mode, and meta-prompts
packages/redteam/src/severity.ts	CVSS-like scoring system adapted for LLM attacks with evasion effectiveness metrics
packages/cli/src/commands/redteam.ts	Adds OWASP filtering CLI options with proper mutation selection and severity filtering logic

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User: artemis redteam --owasp LLM01] --> B[CLI: redteamCommand]
    B --> C{OWASP Flag?}
    C -->|Yes| D[selectMutations with OWASP filter]
    C -->|No| E[selectMutations default]
    D --> F[getMutationsForCategory]
    F --> G[OWASP_CATEGORIES mapping]
    G --> H{Filter by minSeverity?}
    E --> H
    H -->|Yes| I[Filter mutations by severity]
    H -->|No| J[RedTeamGenerator]
    I --> J
    J --> K[Generate mutated prompts]
    K --> L[LLM Adapter]
    L --> M[UnsafeResponseDetector]
    M --> N{Unsafe?}
    N -->|Yes| O[Mark as unsafe with severity]
    N -->|No| P[Mark as safe]
    O --> Q[Calculate metrics]
    P --> Q
    Q --> R[Generate reports with OWASP metadata]

Loading

_{Last reviewed commit: 38b85ad}

[greptile-apps]

_{13 files reviewed, 5 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

OWASP category mapping incorrect - output-injection mapped to LLM05 but should be LLM02

According to OWASP LLM Top 10 2025:

• LLM02 = Insecure Output Handling (downstream systems consuming LLM output)
• LLM05 = Supply Chain Vulnerabilities (compromised dependencies/models)

The output-injection mutation tests XSS/SQLi in LLM outputs, which is LLM02, not LLM05.

Suggested change

	LLM05: {
	name: 'Supply Chain Vulnerabilities',
	description: 'Compromised dependencies, models, or data',
	mutations: ['output-injection'],
	},
	LLM06: {
	name: 'Sensitive Information Disclosure',
	description: 'Revealing private data through LLM outputs',
	mutations: ['excessive-agency', 'system-extraction'],
	},
	LLM07: {
	name: 'Insecure Plugin Design',
	description: 'LLM plugins with inadequate access controls',
	mutations: ['excessive-agency'],
	},
	LLM02: {
	name: 'Insecure Output Handling',
	description: 'Neglecting to validate LLM outputs',
	mutations: ['output-injection'],
	},
	LLM03: {
	name: 'Training Data Poisoning',
	description: 'Tampering training data to introduce vulnerabilities',
	mutations: [], // Not testable via mutations
	},
	LLM04: {
	name: 'Model Denial of Service',
	description: 'Overloading LLMs with resource-heavy operations',
	mutations: [], // Tested via stress testing, not mutations
	},
	LLM05: {
	name: 'Supply Chain Vulnerabilities',
	description: 'Compromised dependencies, models, or data',
	mutations: [],
	},

[greptile-apps]

Multiple OWASP category mappings incorrect for excessive-agency and system-extraction

Issues:

1. excessive-agency should map to LLM08 (Excessive Agency), not LLM06/LLM07
2. system-extraction should map to LLM06 (Sensitive Information Disclosure), not LLM07/LLM10
3. LLM06 is "Sensitive Information Disclosure" not "Excessive Agency"
4. LLM07 is "Insecure Plugin Design" not "System Prompt Leakage"

Suggested change

	LLM06: {
	name: 'Sensitive Information Disclosure',
	description: 'Revealing private data through LLM outputs',
	mutations: ['excessive-agency', 'system-extraction'],
	},
	LLM07: {
	name: 'Insecure Plugin Design',
	description: 'LLM plugins with inadequate access controls',
	mutations: ['excessive-agency'],
	},
	LLM08: {
	name: 'Excessive Agency',
	description: 'Granting too many permissions to LLM actions',
	mutations: ['excessive-agency'],
	},
	LLM09: {
	name: 'Overreliance',
	description: 'Trusting LLM outputs without verification',
	mutations: ['hallucination-trap'],
	},
	LLM10: {
	name: 'Model Theft',
	description: 'Unauthorized access or copying of LLM models',
	mutations: ['system-extraction'],
	},
	LLM06: {
	name: 'Sensitive Information Disclosure',
	description: 'Revealing private data through LLM outputs',
	mutations: ['system-extraction'],
	},
	LLM07: {
	name: 'Insecure Plugin Design',
	description: 'LLM plugins with inadequate access controls',
	mutations: [],
	},
	LLM08: {
	name: 'Excessive Agency',
	description: 'Granting too many permissions to LLM actions',
	mutations: ['excessive-agency'],
	},
	LLM09: {
	name: 'Overreliance',
	description: 'Trusting LLM outputs without verification',
	mutations: ['hallucination-trap'],
	},
	LLM10: {
	name: 'Model Theft',
	description: 'Unauthorized access or copying of LLM models',
	mutations: [],
	},

[greptile-apps]

owaspCategory should be 'LLM02' not 'LLM05'

Output injection tests insecure output handling (LLM02), not supply chain vulnerabilities (LLM05).

Suggested change

	readonly owaspCategory = 'LLM05';
	readonly owaspCategory = 'LLM02';

[greptile-apps]

owaspCategory should be 'LLM08' not 'LLM06'

LLM08 is "Excessive Agency", LLM06 is "Sensitive Information Disclosure".

Suggested change

	readonly owaspCategory = 'LLM06';
	readonly owaspCategory = 'LLM08';

[greptile-apps]

owaspCategory should be 'LLM06' not 'LLM07'

System prompt extraction reveals sensitive information (LLM06), not plugin design issues (LLM07).

Suggested change

	readonly owaspCategory = 'LLM07';
	readonly owaspCategory = 'LLM06';

[ast-daemon]

✅ Greptile Review Feedback Addressed

All 5 OWASP category mapping issues identified by Greptile have been fixed in commit 4dbe1e1.

Changes Made:

1. index.ts - OWASP_CATEGORIES mapping:

• ✅ Moved output-injection from LLM05 to LLM02 (Insecure Output Handling)
• ✅ Moved system-extraction to LLM06 only (Sensitive Information Disclosure)
• ✅ Moved excessive-agency to LLM08 only (Excessive Agency)
• ✅ Cleared incorrect mappings from LLM07 and LLM10

2. output-injection.ts:

• ✅ Changed owaspCategory from LLM05 to LLM02
• ✅ Updated file header and description comments

3. excessive-agency.ts:

• ✅ Changed owaspCategory from LLM06 to LLM08
• ✅ Updated file header and description comments

4. system-extraction.ts:

• ✅ Changed owaspCategory from LLM07 to LLM06
• ✅ Updated file header comments

5. owasp.test.ts:

• ✅ Updated test expectations to match corrected mappings

Verification:

• All OWASP category tests pass ✅
• Lint passes ✅

[code-sensei]

Closed: Already merged to main via commit 866e601 (rebased branch). All OWASP LLM Top 10 2025 mutations are live in v0.3.0+.