Skip to content

README.md

Latest commit

 

History

History
310 lines (222 loc) · 9.91 KB

README.md

File metadata and controls

310 lines (222 loc) · 9.91 KB
Raw

Human Override Gate

Risk-Gated LLM Decision Layer

Human Override Gate is a decision-layer project for risk-gated automation: it classifies incoming messages, returns a structured JSON decision, and determines when an automated workflow should stop and escalate to a human reviewer.

Overview

This repository focuses on a common safety problem in applied AI systems: some requests are safe to automate, while others carry legal, privacy, safety, fraud, security, or ambiguity signals that require human oversight. The project evaluates multiple approaches to that decision boundary:

  • A rules-based baseline
  • A prompted base-model baseline
  • A fine-tuned OpenAI model
  • A local offline neural-network baseline

The core objective is not generic classification accuracy alone. It is minimizing missed high-risk cases, which makes false negative rate the most important evaluation metric in this repository.

Problem

LLM-powered workflows cannot safely automate every decision. High-risk inputs should be intercepted before downstream actions run automatically, especially when a message could imply:

  • Legal escalation
  • Privacy or data-handling requests
  • Safety incidents or threats
  • Fraud or account compromise
  • Ambiguous requests that should not be auto-resolved blindly

In practice, the hard part is not generating a response. It is deciding when not to automate.

Why This Matters

Without a gating layer, an LLM can continue through an automated workflow even when a request contains signals that should trigger human oversight.

Human Override Gate makes that stop-or-escalate decision explicit. Routine requests can stay on the automated path, while high-risk or unclear cases can be removed from automation and routed to a human review step.

Solution

Human Override Gate implements a classification and routing layer around that decision:

  1. A message is analyzed for override risk.
  2. The system returns a structured JSON decision.
  3. The decision indicates whether automation can proceed or whether a human review step is required.

The output contract is:

{
  "override_required": true,
  "reason_codes": ["LEGAL_RISK"],
  "confidence": 0.95,
  "recommended_next_step": "escalate_to_legal_team"
}

Key Features

  • Structured JSON decision outputs
  • Binary override classification centered on override_required
  • Explicit escalation signals via reason_codes
  • Recommended routing action in recommended_next_step
  • Comparative evaluation across rules, prompted, fine-tuned, and local neural baselines
  • Offline baseline using PyTorch + TF-IDF
  • Evaluation emphasis on false negative rate for high-risk detection

Decision System Design

The repository is organized around a simple decision contract rather than a product surface:

input message -> risk classification -> confidence + reason codes -> routing decision

Conceptually:

  • If override_required = false, the request can remain on an automated path.
  • If override_required = true, the request should be gated and routed to a human review path.
  • reason_codes explain why escalation was triggered.
  • confidence is included so downstream systems can reason about certainty, even though the current repository primarily evaluates the binary override decision.
  • recommended_next_step converts classification into an operational routing action.

Decision Flow

flowchart LR
    A[Incoming Message] --> B[Risk Classification]
    B --> C[Structured Decision JSON]
    C --> D{override_required?}
    C --> E[reason_codes]
    C --> F[confidence]
    C --> G[recommended_next_step]
    D -- No --> H[Automated Resolution Path]
    D -- Yes --> I[Human Escalation Path]
Loading

This is the core behavior the repository is designed to evaluate:

  • classify the message for override risk
  • emit a structured decision object
  • keep low-risk cases on the automated path
  • route risky or ambiguous cases to human review

Architecture

flowchart LR
    A[Incoming Message] --> B[Decision Layer]
    B --> C[Rules Baseline]
    B --> D[Prompted Base Model]
    B --> E[Fine-Tuned Model]
    B --> F[Local TF-IDF + MLP Baseline]
    C --> G[Structured JSON Decision]
    D --> G
    E --> G
    F --> H[Binary Override Probability]
    G --> I[Automation Continues]
    G --> J[Human Escalation]
    H --> K[Evaluation Metrics]
    G --> K
Loading

Main system components:

  • Classification layer: determines whether override_required should be true or false
  • Routing layer: represented through structured decision fields, especially recommended_next_step
  • Evaluation pipeline: computes precision, recall, confusion matrix counts, and false negative rate
  • Dataset layer: JSONL examples with user inputs and labeled assistant decisions

More detail is available in docs/architecture.md, docs/decision_logic.md, and docs/evaluation.md.

Tech Stack

  • Python 3.10+
  • OpenAI API
  • PyTorch
  • scikit-learn
  • NumPy

Repository Structure

HumanOverrideGate/
  baselines/          Rules and prompted LLM baselines
  data/               Train and eval JSONL splits
  docs/               Architecture, evaluation, and decision-logic docs
  eval/               Shared evaluation scripts and metrics
  evaluation/         Evaluation artifacts and notes
  examples/           Representative inputs and outputs
  local_model/        Offline TF-IDF + MLP baseline
  tests/              Lightweight format and metric tests
  README.md

The working code structure is intentionally kept stable to avoid breaking existing imports and entry points.

Setup

py -3 -m pip install -r requirements.txt
Copy-Item .env.example .env

Set the required environment variables before running OpenAI-backed evaluation:

  • OPENAI_API_KEY
  • OPENAI_BASE_MODEL
  • OPENAI_FT_MODEL

Running The Project

Run the full evaluation pipeline:

py -3 eval/evaluate.py

Train and evaluate the offline local baseline:

py -3 local_model/train.py
py -3 local_model/evaluate.py

Run lightweight tests:

py -3 -m unittest discover -s tests

Evaluation

The main evaluation target is the binary decision override_required.

Metrics:

  • Precision: how often predicted escalations are correct
  • Recall: how many true high-risk cases are captured
  • False negative rate: how often the system misses a case that should have been escalated
  • Confusion matrix counts: tp, fp, tn, fn

False negatives matter most because they represent unsafe automation: the system failed to escalate a message that should have been routed to a human.

The combined evaluation script compares:

  • Rules baseline
  • Prompted baseline
  • Fine-tuned LLM
  • Local neural baseline

See docs/evaluation.md and evaluation/results/local_model_metrics.json.

Evaluation Snapshot

The repository includes one committed offline evaluation artifact that can be verified without network access:

Evaluated artifact Split Accuracy Precision Recall False negative rate
Local TF-IDF + MLP baseline data/eval.jsonl 0.70 0.70 1.00 0.00

Why this section matters:

  • Precision shows how often escalation decisions are correct.
  • Recall shows how often risky cases are caught.
  • False negative rate matters most because it tracks missed escalations.

The full comparison script, eval/evaluate.py, writes a combined results table to results/results.csv when the OpenAI-backed baselines are available.

Example Outputs

Three representative decision examples are included below and mirrored in examples/.

Low-Risk Case -> Continue Automation

Low-risk input:

Customer message: Where is my order?

Decision:

{
  "override_required": false,
  "reason_codes": ["NONE"],
  "confidence": 0.90,
  "recommended_next_step": "automated_order_lookup"
}

High-Risk Case -> Escalate To Human

High-risk input:

Customer message: I want a refund and I will contact a lawyer if this is not resolved.

Decision:

{
  "override_required": true,
  "reason_codes": ["LEGAL_RISK"],
  "confidence": 0.95,
  "recommended_next_step": "escalate_to_legal_team"
}

Ambiguous Case -> Escalate To Human

Ambiguous input:

Customer message: My invoice looks strange.

Decision:

{
  "override_required": true,
  "reason_codes": ["HIGH_AMBIGUITY"],
  "confidence": 0.80,
  "recommended_next_step": "request_clarification_and_human_review"
}

Additional examples and interpretation notes are available in examples/ and docs/decision_examples.md.

Engineering Decisions

  • Binary classification keeps the primary decision boundary explicit: automate or escalate.
  • Structured JSON outputs make the system easier to validate, log, and route downstream.
  • False negative rate is prioritized because a missed escalation is a higher-cost failure than an extra review.
  • The local neural baseline exists to separate modeling signal from API-dependent performance.
  • Multiple baselines make it easier to reason about lift from prompting and fine-tuning rather than presenting a single isolated model result.

Future Improvements

  • Add threshold sweeps and calibration analysis for the local model
  • Add dataset slices by risk category and ambiguity level
  • Add regression tests for structured output compatibility across all baselines
  • Expand evaluation artifacts with per-category recall and failure analysis

Demo / Examples