Added initial draft of ASI08_Cascading_Failures.md #720
Conversation
Added Initial draft of ASI08 - Cascading Failures in Agentic AI Signed-off-by: dianamhenderson <[email protected]>
dianamhenderson
requested review from
guerilla7,
hoeg and
itskerenkatz
as code owners
|
|
||
| Description | ||
|
|
||
| Cascading Failures in agentic AI systems occur when a single error, whether a hallucination, malicious input, corrupted tool, or compromised memory, propagates from one AI agent to another, leading to compound failures and system-wide harm. Unlike traditional LLM applications that generate outputs for human review, agentic systems plan, persist, delegate, and act autonomously. These capabilities mean that a minor fault can become a multi-step chain of privileged operations affecting data, infrastructure, finances, or entire multi-agent ecosystems performing unintended actions. |
| Description | ||
|
|
||
| Cascading Failures in agentic AI systems occur when a single error, whether a hallucination, malicious input, corrupted tool, or compromised memory, propagates from one AI agent to another, leading to compound failures and system-wide harm. Unlike traditional LLM applications that generate outputs for human review, agentic systems plan, persist, delegate, and act autonomously. These capabilities mean that a minor fault can become a multi-step chain of privileged operations affecting data, infrastructure, finances, or entire multi-agent ecosystems performing unintended actions. | ||
| Root Causes of Cascading Failures |
There was a problem hiding this comment.
I think here are some of the vulnerabilities or suggested solution and less of the overview.
I think the overview above is super clear and maybe we can move what's written in here to the vulnerabilities or mitigation parts.
There was a problem hiding this comment.
Removing the Root Causes section and merging it into the existing Vulnerabilities list.
| 2. Overly trusting agents i.e. blind trust in outputs | ||
| 3. Lack of feedback loops with oversight i.e. no fail-safes | ||
| 4. | ||
| The impacts can escalate small faults into major incidents such as data corruption or widespread service failures across networks of connected agents or workflows. |
There was a problem hiding this comment.
Maybe in a broader sense we'd like to describe the impact as:
data leakage, output manipulation or workflow hijacking
To try and cover more holistically what could be the consequences
There was a problem hiding this comment.
removed root causes and merged impacts escalation into overview paragraph prior.
|
|
||
| Here are key fundamental differences between agentic cascade and traditional system failures: | ||
| 1. Autonomous Decision Propagation: Agents make decisions that directly influence other agents or systems without human validation at each step. | ||
| 2. Persistent State Corruption: Malicious or erroneous information can persist in agent memory, influencing future actions across sessions. |
There was a problem hiding this comment.
In the second one it's less clear to me why it is related to cascading failures and not to context injection (ASI06)
There was a problem hiding this comment.
Updated for better alignment to cascading failures instead of context injection
| Here are key fundamental differences between agentic cascade and traditional system failures: | ||
| 1. Autonomous Decision Propagation: Agents make decisions that directly influence other agents or systems without human validation at each step. | ||
| 2. Persistent State Corruption: Malicious or erroneous information can persist in agent memory, influencing future actions across sessions. | ||
| 3. Dynamic Tool Invocation: Agents can discover and execute tool combinations that weren't explicitly programmed, creating unpredictable attack surfaces. |
There was a problem hiding this comment.
In here too, more related to me to supply chain (ASI04)
There was a problem hiding this comment.
Updated for better alignment to cascading failures instead of supply chain
| A trading firm deploys an orchestrated system of specialized agents: Market Analysis, Risk Assessment, Position Management, and Execution agents. An attacker exploits LLM01:2025 Prompt Injection in the Market Analysis agent by embedding malicious instructions in a financial news feed. The compromised agent provides false market sentiment data to the Risk Assessment agent, which autonomously adjusts risk models and increases position limits. The Position Management agent approves larger trades based on corrupted assessments, while the Execution agent processes these trades without human approval. The compliance monitoring sees no violations because all agents operate within their adjusted parameters. The autonomous nature enables the attack to manipulate significant financial positions without triggering traditional oversight mechanisms. | ||
|
|
||
| Scenario #2: Healthcare Protocol Propagation | ||
| A hospital network uses coordinated agents for patient care: Pharmacy Management, Treatment Planning, and Care Coordination agents. A LLM03:2025 Supply Chain attack corrupts the Pharmacy Management agent through a malicious update to its drug interaction database. The compromised agent provides false drug compatibility data to the Treatment Planning agent, which autonomously adjusts treatment protocols. The Care Coordination agent propagates these corrupted protocols to other hospital systems, where local agents adopt the guidelines without human review. Unlike traditional healthcare systems requiring human approval for protocol changes, the autonomous agent coordination bypasses these safeguards, spreading potentially dangerous medical protocols across the entire network. |
There was a problem hiding this comment.
Let's quote ASI:04 Supply chain attack :)
There was a problem hiding this comment.
Replaced LLM03:2025 Supply Chain with ASI04: Supply Chain.
| An enterprise uses agents to manage cloud infrastructure: Resource Planning, Security Configuration, and Deployment Coordination agents. An attacker compromises the Resource Planning agent through a vulnerability in its forecasting tool, introducing LLM04:2025 Data and Model Poisoning. The corrupted agent begins generating resource allocation plans that include unauthorized access permissions and excessive resource provisioning. The Security Configuration agent, trusting the planning data, automatically implements the malicious security policies. The Deployment Coordination agent provisions resources based on the corrupted plans, creating backdoor access and escalating operational costs. The autonomous coordination means infrastructure changes happen without human authorization for each modification. | ||
|
|
||
| Scenario #4: Multi-Agent Security Operations Compromise | ||
| A security operations center uses autonomous agents for threat detection, incident response, and compliance monitoring. An attacker exploits shared service account credentials used by multiple security agents. This resembles LLM07:2025 System Prompt Leakage. The compromised credentials enable lateral movement through the agent network. The threat detection agent begins marking genuine alerts as false positives while approving malicious transactions. The incident response agent executes "remediation" actions that disable security controls and delete audit logs. The compliance agent reports false metrics based on corrupted data. The autonomous nature enables attacks to propagate at machine speed without traditional human oversight, effectively blinding the security operations while appearing to function normally. |
There was a problem hiding this comment.
Why is it relevant to: This resembles LLM07:2025 System Prompt Leakage? Are the credentials in the system prompts? less common
I think it worth explaining this one a bit better (this part)
There was a problem hiding this comment.
Good point. The LLM07 may not align as well since it’s more about extracting sensitive info from system prompts via prompt injection. The credentials in this scenario are more susceptible to infrastructure attacks vs system prompt being exposed. It’s probably closer to excessive agency and supply chain LLMTop10, so updated with this now, hopefully still showing the connection to cascade failure.
| A security operations center uses autonomous agents for threat detection, incident response, and compliance monitoring. An attacker exploits shared service account credentials used by multiple security agents. This resembles LLM07:2025 System Prompt Leakage. The compromised credentials enable lateral movement through the agent network. The threat detection agent begins marking genuine alerts as false positives while approving malicious transactions. The incident response agent executes "remediation" actions that disable security controls and delete audit logs. The compliance agent reports false metrics based on corrupted data. The autonomous nature enables attacks to propagate at machine speed without traditional human oversight, effectively blinding the security operations while appearing to function normally. | ||
|
|
||
| Scenario #5: Manufacturing Quality Control Chain Failure | ||
| A smart manufacturing facility uses agents to coordinate production: Quality Control, Inventory Management, and Production Scheduling agents. An attacker injects malicious context into the Quality Control agent's memory through processed inspection reports containing hidden instructions. This LLM08:2025 Vector and Embedding Weaknesses attack poisons the agent's quality assessment knowledge base. The contaminated agent begins approving products that don't meet specifications while rejecting compliant products. The Inventory Management agent adjusts stock levels based on false quality data, while the Production Scheduling agent optimizes workflows around incorrect quality metrics. The cascade results in shipping defective products while discarding good inventory, creating significant financial losses and potential safety issues. |
There was a problem hiding this comment.
Let's refer to ASI06 (memory and context injection)
There was a problem hiding this comment.
Added in context for ASI06 to scenario #5.
| 10. Deploy Behavioral Anomaly Detection: Monitor agent decision patterns and tool usage for deviations from expected behavior. Unlike traditional systems, agents can make semantically valid but contextually inappropriate decisions that require behavioral analysis. | ||
|
|
||
|
|
||
| Example Attack Scenarios |
There was a problem hiding this comment.
Scenarios are AMAZING - I truly loved it
| Key Insight: Across sectors, a single compromised agent can initiate a chain of autonomous, trusted actions like planning, approving, and executing changes, all without human checkpoints. As a result, several prominent agentic attributes (persistent memory, delegated authority, cross-agent coordination) unwittingly collaborate to allow a minor injection or poisoning escalate into systemic, business-critical failures. | ||
|
|
||
|
|
||
| Reference Links |
There was a problem hiding this comment.
Please provide the actual links, to the specific part - for example: GOVERN-1.1: AI governance structures adapted for multi-agent coordination - add a link specifically to it.
I think there is no need to actually refer to any prompt injection mention out there, but rather to what is parallel in other frameworks (of course we mention prompt injection in any of our entries)
There was a problem hiding this comment.
Updated with URLs for the references. NOTE that GOVERN-1.1, etc. are subcategories within the NIST AI RMF and don't have their own URL.
Updates to some of the comments. Signed-off-by: dianamhenderson <[email protected]>
Made additional changes based on review comments. Signed-off-by: dianamhenderson <[email protected]>
Hi, I have made the changes, but I'm unable to add updated document /.MD file here, kindly provide access. However, I have shared the updated document in the Slack Group. Changes made in following areas of the document:
Regards |
|
I dont have access to edit, so proposing cause and prevention strategy in cascading failure --> Cognitive Load Drift Detection: “Cognitive Drift” in agentic systems — where an agent’s understanding or memory gradually diverges from reality due to noise, incomplete rollback, or feedback loops. How? Why? Thanks |
Added headline, added sub categories for mitigation strategies, incorporated cognitive drift into existing strategy for Deploy Behavioral Anomaly Detection. Signed-off-by: dianamhenderson <[email protected]>
Added Initial draft of ASI08 - Cascading Failures in Agentic AI for the OWASP ASI Top 10 of Agentic AI.