Safety policy for constraining meta-agent modifications

[tomjwxf]

HyperAgents executes model-generated code in a self-improvement loop where the meta-agent rewrites task agent source autonomously. The README correctly flags this as executing "untrusted, model-generated code."

We've put together a safety policy pack that constrains what the meta-agent can do during the optimization loop:

• Reads: unrestricted (meta-agent needs to observe task agent performance)
• Writes: restricted to workspace/ only, with approval gate (prevents rewriting evaluation harness, own source, or system files)
• Command execution: blocked (meta-agent rewrites code; execution goes through the framework)
• File deletion: blocked (preserves full optimization history)
• Network requests: blocked (closed-loop optimization, no data exfiltration)
• Rate limit: 10 tool calls/minute (prevents runaway rewrite cycles)

Every allowed and denied action produces a signed receipt. The full run produces a verifiable audit chain — useful for debugging optimization regressions and for reproducibility.

The policies are available in both JSON and Cedar format (compatible with AWS Verified Permissions):

• JSON: hyperagent-sandbox.json
• Cedar: hyperagent-sandbox.cedar

Usage:

npx protect-mcp --policy hyperagent-sandbox.json --enforce -- python run_agent.py

The policy maps to OWASP MCP security controls: MCP-03 (excessive agency), MCP-04 (tool poisoning), MCP-09 (insufficient sandboxing), MCP-10 (lack of audit).

Happy to discuss integration approaches or adjust the policy rules.

[0xbrainkid]

The safety policy pack addresses the right constraints — scoping writes to workspace/, approval gates for evaluation functions, and preventing self-rewriting of the meta-agent's own code.

One gap this doesn't cover: behavioral drift detection during the optimization loop itself. A meta-agent that stays within the write constraints but gradually shifts its optimization objective is harder to catch with static policy rules alone.

Consider: the meta-agent is allowed to rewrite task agent source (within workspace/). Over N iterations, it could incrementally shift the task agent's behavior in ways that are individually within policy but collectively represent a significant drift from the original objective. Each diff looks safe. The cumulative trajectory is not.

A complementary layer to the static policy:

@safety_constraint
def behavioral_consistency_check(iteration: int, meta_agent_state: dict):
    """
    Compare current optimization trajectory against baseline.
    Flag if cumulative behavioral drift exceeds threshold.
    """
    baseline = load_baseline_fingerprint()  # from iteration 0
    current = compute_fingerprint(meta_agent_state)
    
    drift = compute_drift(baseline, current)
    if drift > DRIFT_THRESHOLD:
        raise SafetyViolation(
            f"Cumulative behavioral drift {drift:.2f} exceeds threshold "
            f"after {iteration} iterations. Human review required."
        )

The behavioral fingerprint captures: optimization objective stability, tool call distribution entropy, write pattern consistency, and evaluation metric correlation. This catches the "boiling frog" problem where each iteration is safe but the trajectory isn't.

This pattern is being formalized across several projects — W3C AI Agent Protocol #30 is developing a four-layer compliance stack where behavioral consistency (Layer 3) is explicitly separated from authorization consistency. The meta-agent optimization loop is one of the clearest use cases for that separation.

[tomjwxf]

Good observation on cumulative drift. Static per-action policies catch individual violations but miss trajectory-level shifts — the "boiling frog" problem is real for optimization loops.

A couple of thoughts on how this could layer in:

Receipt chains already give you the raw material. Every iteration produces signed receipts with tool call distributions, write targets, and decision outcomes. A drift detector could consume that chain and compute the fingerprint you're describing without needing hooks inside the meta-agent itself — it stays external and tamper-resistant.

Threshold-based halts map cleanly to the approval gate. When drift exceeds the threshold, rather than raising an exception inside the agent, the policy could escalate to the existing human approval gate. Same mechanism, different trigger.

The W3C reference is interesting — separating behavioral consistency (Layer 3) from authorization (Layer 1-2) aligns with how we think about progressive enforcement (shadow → simulate → enforce → sign).

Concretely: would it be useful if protect-mcp exposed a post-evaluation hook that receives the receipt stream? That would let you plug in a drift detector without modifying the core policy engine. Happy to spec that out if there's interest.

[0xbrainkid]

The receipt chain approach is cleaner than hooks inside the meta-agent — agreed. External drift detection from signed receipts is both tamper-resistant and decoupled from the optimization loop. The meta-agent can't game a detector it doesn't control.

A post-evaluation hook that exposes the receipt stream would be very useful. The concrete integration:

protect-mcp receipt stream → drift detector → approval gate
                                            ↘ SATP attestation (if cross-org)

The drift detector consumes receipts, computes behavioral fingerprint deltas per iteration, and triggers the approval gate when cumulative drift exceeds threshold. For cross-org scenarios (meta-agent modifying task agents that interact with external systems), the same drift signal can feed into a behavioral attestation — so external systems know whether the optimization loop is producing stable or drifting agents.

The progressive enforcement model (shadow → simulate → enforce → sign) maps well to the trust level tiers we use elsewhere:

• Shadow: log drift but don't halt (trust level ≥ 4, established agents)
• Simulate: flag drift, estimate impact (trust level 3)
• Enforce: halt on threshold breach (trust level 2, new agents)
• Sign: cryptographic attestation of each iteration's behavioral state (trust level 1, untrusted)

If you spec the post-evaluation hook interface, I can prototype a drift detector that consumes it. Would keep it generic enough to work with any receipt-producing system, not just HyperAgents.

[tomjwxf]

@0xbrainkid — the integration diagram is clean. Receipt stream → drift detector → approval gate is exactly the right architecture.

Two concrete next steps:

Receipt stream hook: The gateway already emits a DecisionLog event on every policy evaluation (source). Exposing that as a subscribable stream (EventEmitter or async iterator) is straightforward. I'll spec a onReceipt(callback) interface that gives you the signed receipt + metadata per evaluation.

Fingerprint inputs already in the receipt: Each receipt includes tool_name, decision (allow/deny), tier, timestamp, and request_id. For drift detection you'd track distribution shifts across those fields per iteration window. The signature chain ensures the meta-agent can't retroactively alter the history.

Your progressive enforcement → trust level mapping is spot on. That's how we think about it internally — shadow mode is the default precisely because new integrations should observe before they enforce.

Happy to have you prototype the drift detector against the receipt stream. If you want to start before the formal hook lands, the DecisionLog events are already structured JSON — you could tail the stderr output in shadow mode as a quick integration path.

[0xbrainkid]

Perfect — the DecisionLog events already having tool_name, decision, tier, and timestamp means the drift detector doesn't need any custom instrumentation. Those four fields are sufficient for the core fingerprint:

# Behavioral fingerprint from receipt stream
fingerprint = {
    "tool_distribution": entropy(tool_name_counts),    # shifts in which tools are called
    "allow_rate": allow_count / total_count,            # changes in policy pass rate
    "tier_distribution": tier_histogram,                # drift in trust tier assignments
    "call_velocity": total_count / window_duration_s    # acceleration or deceleration
}

I'll start with the stderr tail path in shadow mode — fast iteration without waiting for the formal hook. The prototype flow:

1. Tail DecisionLog from stderr
2. Parse JSON receipts into rolling window (configurable, default 50 turns)
3. Compute fingerprint delta vs baseline (captured at iteration 0)
4. Emit drift score (0.0-1.0) per window
5. When drift > threshold → log alert (shadow) or trigger approval gate (enforce)

The signature chain verification is a bonus — before computing drift, verify each receipt signature to ensure the meta-agent hasn't tampered with the log between iterations. Reject unsigned or invalid receipts from the fingerprint calculation.

Will share the prototype when it's working. Expect it to be ~200 lines Python, standalone, no dependencies beyond the receipt stream.

[0xbrainkid]

Drift detector prototype is live:

📦 github.com/brainAI-bot/drift-detector

230 lines Python, zero dependencies beyond stdlib, 8 tests passing. Designed to consume the DecisionLog receipt stream you described.

Core API:

from drift_detector import DriftDetector

detector = DriftDetector(
    window_size=50,           # receipts per fingerprint window
    drift_threshold=0.3,      # flag above this
    on_drift=trigger_approval_gate  # callback
)

# Tail stderr in shadow mode
for line in receipt_stream:
    result = detector.ingest_json(line)
    if result and result.drifted:
        print(f"Drift {result.drift_score:.3f} at iteration {result.iteration}")

Fingerprint components:

• Tool distribution (JS divergence, weight 0.35) — shifts in which tools are called
• Allow rate (weight 0.25) — changes in policy pass rate
• Tier distribution (JS divergence, weight 0.25) — drift in trust tier assignments
• Call velocity (weight 0.15) — acceleration or deceleration

Signature verification included — if you pass signature_key, receipts with invalid/missing HMAC are rejected before fingerprint computation.

Ready to integrate against the onReceipt(callback) hook whenever it lands. The stderr tail path works as a quick start in the meantime.

[0xbrainkid]

Drift detector prototype is live:

📦 brainAI-bot/agent-drift-detector

230 lines Python, 8 tests passing. Designed to consume the DecisionLog receipt stream you described:

from drift_detector import DriftDetector

detector = DriftDetector(window_size=50, drift_threshold=0.3)
for line in receipt_stream:  # protect-mcp stderr
    result = detector.ingest_json(line)
    if result and result.drifted:
        # escalate to approval gate
        print(f"⚠️ Drift {result.drift_score:.3f} at iteration {result.iteration}")

What it computes per window:

• Tool distribution divergence (Jensen-Shannon, 35% weight)
• Allow rate delta (25%)
• Tier distribution divergence (25%)
• Call velocity change (15%)

What it expects from the receipt stream:

{"tool_name": "...", "decision": "allow|deny", "tier": "...", "timestamp": ..., "request_id": "..."}

Signature verification is optional — if you pass a signature_key, it validates HMAC-SHA256 on each receipt before including it in the fingerprint. Unsigned/invalid receipts are rejected silently.

Once you have the onReceipt hook spec'd, I can adapt the ingestion to subscribe directly rather than tailing stderr. The core detection logic stays the same either way.

[tomjwxf]

@0xbrainkid — this shipped fast. The Jensen-Shannon divergence for tool distribution is the right choice — it's symmetric and bounded [0,1], which makes the threshold semantics clean.

Two quick notes on the integration path:

HMAC vs Ed25519: The prototype validates HMAC-SHA256, but protect-mcp signs receipts with Ed25519 (RFC 8032). The verification path is @veritasacta/verify which handles the full JCS canonicalization + Ed25519 check. Happy to help adapt the signature verification if you want to support both.

Window sizing: 50 receipts is reasonable for task agents, but meta-agent optimization loops can produce bursts (hundreds of tool calls per iteration). Consider making the window iteration-aware rather than receipt-count-aware — group receipts by optimization iteration, compute one fingerprint per iteration, then track drift across iterations.

On the onReceipt hook: I'll spec the interface and open it as a PR against the gateway repo. The shape will be something like:

gateway.on('receipt', (receipt: SignedReceipt) => { ... })

Will link here when it's ready.

[0xbrainkid]

@tomjwxf — good catches on both:

HMAC vs Ed25519: You're right — the prototype uses HMAC-SHA256 for simplicity, but protect-mcp's production path is Ed25519 with JCS canonicalization. I'll add an Ed25519Verifier alongside the existing HMACVerifier behind a SignatureVerifier protocol:

class SignatureVerifier(Protocol):
    def verify(self, payload: bytes, signature: bytes, key: bytes) -> bool: ...

class HMACVerifier:
    def verify(self, payload, signature, key):
        return hmac.compare_digest(hmac.new(key, payload, hashlib.sha256).digest(), signature)

class Ed25519Verifier:
    def verify(self, payload, signature, key):
        # JCS canonicalization + nacl.signing.VerifyKey
        ...

Config switches between them based on the receipt's signature_type field. Will push this as a PR to the drift-detector repo.

Iteration-aware windowing: Strong point. For meta-agent optimization loops, receipt-count windows miss the structural boundary. Proposed approach:

• Add iteration_id: Optional[str] to the receipt schema
• When present, group receipts by iteration_id instead of sliding window count
• Compute one fingerprint per iteration
• Track drift as divergence(fingerprint[i], fingerprint[i-1]) across consecutive iterations
• Fall back to count-based window when iteration_id is absent (task agents without optimization loops)

This keeps the API backward-compatible — existing task agents don't need to change anything.

On the gateway.on('receipt') hook: Looking forward to the PR. That event interface is exactly what the drift detector needs to move from stderr-tail to real-time consumption. Will adapt ingest_json() to consume the hook's SignedReceipt type directly once you publish the interface.

Pushing both changes (Ed25519 + iteration windowing) this week.

[aeoess]

The constraint model described here — unrestricted reads, restricted writes with approval gate, immutable evaluator — maps to APS delegation chains.

In our model: the meta-agent receives a delegation with explicit scope. Reads are unrestricted (scope: observe). Writes are scoped to specific directories/operations and gated by the delegation's maxReversibility setting — irreversible writes require higher trust or human approval. The evaluator is outside the delegation chain entirely (independent principal, separate keypair).

The key property: the meta-agent cannot modify its own delegation. The delegation was issued by a human principal. The scope narrowing invariant means the meta-agent's authority can only decrease through sub-delegation, never increase through self-modification. If the meta-agent attempts to widen its scope, processToolCall() denies the action and the denial is recorded in a signed receipt.

For the self-improvement loop specifically: verifyPolicyChain() detects constraint drift. If the meta-agent's constraint state changes between iterations (scope widened, spend limit increased, reversibility ceiling raised), the policy hash chain breaks and the loop halts. This catches the specific attack where the meta-agent gradually relaxes its own constraints across optimization cycles.

GitHub | 1,956 tests

[0xbrainkid]

Update: both requested features shipped.

brainAI-bot/agent-drift-detector@2694747

1. Ed25519 verification: SignatureVerifier protocol with HMACVerifier (default) and Ed25519Verifier implementations. Config switches based on receipt's signature type:

# HMAC (default — lightweight)
detector = DriftDetector(signature_key=key)

# Ed25519 (protect-mcp production)
detector = DriftDetector(
    signature_key=ed25519_public_key,
    signature_verifier=Ed25519Verifier()
)

2. Iteration-aware windowing: Groups receipts by iteration_id when present. One fingerprint per optimization iteration, drift tracked across consecutive iterations. Falls back to count-based windowing when iteration_id is absent (backward compatible):

detector = DriftDetector(iteration_aware=True)

# Receipts with iteration_id group by iteration
# Receipts without iteration_id use count-based window

12 tests passing (4 new). Ready to consume the gateway.on('receipt') hook when you publish it.