Human Override Gate

Risk-Gated LLM Decision Layer

Human Override Gate is a decision-layer project for risk-gated automation: it classifies incoming messages, returns a structured JSON decision, and determines when an automated workflow should stop and escalate to a human reviewer.

Overview

This repository focuses on a common safety problem in applied AI systems: some requests are safe to automate, while others carry legal, privacy, safety, fraud, security, or ambiguity signals that require human oversight. The project evaluates multiple approaches to that decision boundary:

• A rules-based baseline
• A prompted base-model baseline
• A fine-tuned OpenAI model
• A local offline neural-network baseline

The core objective is not generic classification accuracy alone. It is minimizing missed high-risk cases, which makes false negative rate the most important evaluation metric in this repository.

Problem

LLM-powered workflows cannot safely automate every decision. High-risk inputs should be intercepted before downstream actions run automatically, especially when a message could imply:

• Legal escalation
• Privacy or data-handling requests
• Safety incidents or threats
• Fraud or account compromise
• Ambiguous requests that should not be auto-resolved blindly

In practice, the hard part is not generating a response. It is deciding when not to automate.

Why This Matters

Without a gating layer, an LLM can continue through an automated workflow even when a request contains signals that should trigger human oversight.

Human Override Gate makes that stop-or-escalate decision explicit. Routine requests can stay on the automated path, while high-risk or unclear cases can be removed from automation and routed to a human review step.

Solution

Human Override Gate implements a classification and routing layer around that decision:

1. A message is analyzed for override risk.
2. The system returns a structured JSON decision.
3. The decision indicates whether automation can proceed or whether a human review step is required.

The output contract is:

{
  "override_required": true,
  "reason_codes": ["LEGAL_RISK"],
  "confidence": 0.95,
  "recommended_next_step": "escalate_to_legal_team"
}

Key Features

• Structured JSON decision outputs
• Binary override classification centered on override_required
• Explicit escalation signals via reason_codes
• Recommended routing action in recommended_next_step
• Comparative evaluation across rules, prompted, fine-tuned, and local neural baselines
• Offline baseline using PyTorch + TF-IDF
• Evaluation emphasis on false negative rate for high-risk detection

Decision System Design

The repository is organized around a simple decision contract rather than a product surface:

input message -> risk classification -> confidence + reason codes -> routing decision

Conceptually:

• If override_required = false, the request can remain on an automated path.
• If override_required = true, the request should be gated and routed to a human review path.
• reason_codes explain why escalation was triggered.
• confidence is included so downstream systems can reason about certainty, even though the current repository primarily evaluates the binary override decision.
• recommended_next_step converts classification into an operational routing action.

Decision Flow

flowchart LR
    A[Incoming Message] --> B[Risk Classification]
    B --> C[Structured Decision JSON]
    C --> D{override_required?}
    C --> E[reason_codes]
    C --> F[confidence]
    C --> G[recommended_next_step]
    D -- No --> H[Automated Resolution Path]
    D -- Yes --> I[Human Escalation Path]

Loading

This is the core behavior the repository is designed to evaluate:

• classify the message for override risk
• emit a structured decision object
• keep low-risk cases on the automated path
• route risky or ambiguous cases to human review

Architecture

flowchart LR
    A[Incoming Message] --> B[Decision Layer]
    B --> C[Rules Baseline]
    B --> D[Prompted Base Model]
    B --> E[Fine-Tuned Model]
    B --> F[Local TF-IDF + MLP Baseline]
    C --> G[Structured JSON Decision]
    D --> G
    E --> G
    F --> H[Binary Override Probability]
    G --> I[Automation Continues]
    G --> J[Human Escalation]
    H --> K[Evaluation Metrics]
    G --> K

Loading

Main system components:

• Classification layer: determines whether override_required should be true or false
• Routing layer: represented through structured decision fields, especially recommended_next_step
• Evaluation pipeline: computes precision, recall, confusion matrix counts, and false negative rate
• Dataset layer: JSONL examples with user inputs and labeled assistant decisions

More detail is available in docs/architecture.md, docs/decision_logic.md, and docs/evaluation.md.

Tech Stack

• Python 3.10+
• OpenAI API
• PyTorch
• scikit-learn
• NumPy

Repository Structure

HumanOverrideGate/
  baselines/          Rules and prompted LLM baselines
  data/               Train and eval JSONL splits
  docs/               Architecture, evaluation, and decision-logic docs
  eval/               Shared evaluation scripts and metrics
  evaluation/         Evaluation artifacts and notes
  examples/           Representative inputs and outputs
  local_model/        Offline TF-IDF + MLP baseline
  tests/              Lightweight format and metric tests
  README.md

The working code structure is intentionally kept stable to avoid breaking existing imports and entry points.

Setup

py -3 -m pip install -r requirements.txt
Copy-Item .env.example .env

Set the required environment variables before running OpenAI-backed evaluation:

• OPENAI_API_KEY
• OPENAI_BASE_MODEL
• OPENAI_FT_MODEL

Running The Project

Run the full evaluation pipeline:

py -3 eval/evaluate.py

Train and evaluate the offline local baseline:

py -3 local_model/train.py
py -3 local_model/evaluate.py

Run lightweight tests:

py -3 -m unittest discover -s tests

Evaluation

The main evaluation target is the binary decision override_required.

Metrics:

• Precision: how often predicted escalations are correct
• Recall: how many true high-risk cases are captured
• False negative rate: how often the system misses a case that should have been escalated
• Confusion matrix counts: tp, fp, tn, fn

False negatives matter most because they represent unsafe automation: the system failed to escalate a message that should have been routed to a human.

The combined evaluation script compares:

• Rules baseline
• Prompted baseline
• Fine-tuned LLM
• Local neural baseline

See docs/evaluation.md and evaluation/results/local_model_metrics.json.

Evaluation Snapshot

The repository includes one committed offline evaluation artifact that can be verified without network access:

Evaluated artifact	Split	Accuracy	Precision	Recall	False negative rate
Local TF-IDF + MLP baseline	data/eval.jsonl	0.70	0.70	1.00	0.00

Why this section matters:

• Precision shows how often escalation decisions are correct.
• Recall shows how often risky cases are caught.
• False negative rate matters most because it tracks missed escalations.

The full comparison script, eval/evaluate.py, writes a combined results table to results/results.csv when the OpenAI-backed baselines are available.

Example Outputs

Three representative decision examples are included below and mirrored in examples/.

Low-Risk Case -> Continue Automation

Low-risk input:

Customer message: Where is my order?

Decision:

{
  "override_required": false,
  "reason_codes": ["NONE"],
  "confidence": 0.90,
  "recommended_next_step": "automated_order_lookup"
}

High-Risk Case -> Escalate To Human

High-risk input:

Customer message: I want a refund and I will contact a lawyer if this is not resolved.

Decision:

{
  "override_required": true,
  "reason_codes": ["LEGAL_RISK"],
  "confidence": 0.95,
  "recommended_next_step": "escalate_to_legal_team"
}

Ambiguous Case -> Escalate To Human

Ambiguous input:

Customer message: My invoice looks strange.

Decision:

{
  "override_required": true,
  "reason_codes": ["HIGH_AMBIGUITY"],
  "confidence": 0.80,
  "recommended_next_step": "request_clarification_and_human_review"
}

Additional examples and interpretation notes are available in examples/ and docs/decision_examples.md.

Engineering Decisions

• Binary classification keeps the primary decision boundary explicit: automate or escalate.
• Structured JSON outputs make the system easier to validate, log, and route downstream.
• False negative rate is prioritized because a missed escalation is a higher-cost failure than an extra review.
• The local neural baseline exists to separate modeling signal from API-dependent performance.
• Multiple baselines make it easier to reason about lift from prompting and fine-tuning rather than presenting a single isolated model result.

Future Improvements

• Add threshold sweeps and calibration analysis for the local model
• Add dataset slices by risk category and ambiguity level
• Add regression tests for structured output compatibility across all baselines
• Expand evaluation artifacts with per-category recall and failure analysis

Demo / Examples

• examples/inputs/low_risk.txt
• examples/inputs/high_risk.txt
• examples/inputs/ambiguous_case.txt
• examples/outputs/low_risk.json
• examples/outputs/high_risk.json
• examples/outputs/ambiguous_case.json