Skip to content

README.md

Latest commit

 

History

History
206 lines (172 loc) · 8.85 KB

README.md

File metadata and controls

206 lines (172 loc) · 8.85 KB

OpenOperator Infra

Terraform Kubernetes Helm Azure Python FastAPI React TypeScript Vite MongoDB Docker Traefik Elasticsearch Grafana

AI-powered test automation and benchmarking platform running on Kubernetes.

OpenOperator is a cloud-native platform for testing AI agents against real operating systems (Windows, macOS, Linux). It provisions virtual computers on Kubernetes, deploys AI agents, records interactions via video, and provides real-time monitoring through a web dashboard. Designed for benchmarking AI agent performance at scale with full Azure cloud integration.

Features

  • Multi-OS Virtual Computers — Spin up Windows, macOS, and Linux VMs as Kubernetes pods with VNC/RDP access
  • AI Agent Orchestration — Deploy and manage AI agents that interact with virtual desktops
  • Test & Benchmark Execution — Run tests individually or as benchmark suites with configurable repetition
  • Video Recording — Capture and store agent interactions as recordings in Azure Blob Storage
  • Real-time Monitoring — Live desktop streaming via Guacamole/noVNC with WebSocket communication
  • LLM Observability — Trace and analyze LLM calls with Langfuse integration
  • Result Analytics — Track cost, time, and validation results per test step and test run
  • Auto-scaling Infrastructure — Node pool auto-scaling for agents, benchmarks, and user workloads

Architecture

graph TD
    subgraph Azure Cloud
        AKS[Azure Kubernetes Service]
        BLOB[Azure Blob Storage]
        UAMI[Managed Identity]
    end

    subgraph AKS Cluster
        subgraph OOAdmin[Admin Namespace]
            API[FastAPI API]
            UI[React Dashboard]
            WSB[WebSocket Broker]
            DB[(MongoDB)]
        end

        subgraph OOUser[User Namespace]
            WIN[Windows VM]
            MAC[macOS VM]
            LNX[Linux VM]
            AGT[AI Agents]
        end

        subgraph OOAnalytics[Analytics Namespace]
            ES[Elasticsearch]
            GF[Grafana]
            LF[Langfuse]
        end

        subgraph Infra
            TFK[Traefik Ingress]
            LH[Longhorn Storage]
        end
    end

    USER((User)) -->|HTTPS| TFK
    TFK --> UI
    TFK --> API
    TFK --> WSB
    UI -->|REST + WS| API
    API --> DB
    API -->|Manage| WIN & MAC & LNX
    API -->|Deploy| AGT
    AGT -->|Interact| WIN & MAC & LNX
    API -->|Upload recordings| BLOB
    API -->|Workload Identity| UAMI
    UAMI --> BLOB
Loading

Tech Stack

Layer Technology Purpose
Cloud Azure (AKS, Blob Storage, UAMI) Cloud infrastructure & identity
IaC Terraform (azurerm ~> 3.0) Infrastructure provisioning
Orchestration Kubernetes + Helm Container scheduling & packaging
Backend Python, FastAPI, Uvicorn REST API & job execution
Frontend React, TypeScript, Vite, Fluent UI Web dashboard
Database MongoDB Configuration & instance storage
Ingress Traefik v3.3 Routing, TLS termination
Remote Desktop Guacamole, noVNC, VNC/RDP proxies Live desktop access
Storage Longhorn Distributed persistent volumes
Logging Elasticsearch + Kibana Log aggregation
Monitoring Grafana Metrics dashboards
LLM Tracing Langfuse LLM observability

Project Structure

├── Infra-Terraform/            # Terraform modules for Azure infrastructure
│   ├── k8s/                    #   AKS cluster with multi-node-pool setup
│   ├── storage/                #   Azure Storage account
│   ├── security/               #   Managed identity & RBAC
│   ├── LLMs/                   #   LLM model infrastructure
│   └── omniparser-server/      #   Vision/UI parsing service
│
├── k8s-Helm/                   # Helm charts for Kubernetes deployments
│   ├── OOAdmin/                #   Admin services (API, UI, DB, WS-Broker, Guacamole, Proxy)
│   ├── OOUser/                 #   User workloads (Computers: Win/Mac/Linux, Agents)
│   ├── OOAnalytics/            #   Observability (Elasticsearch, Grafana, Langfuse)
│   ├── OOBenchmark/            #   Benchmark & test job runners
│   └── OOStorage/              #   Longhorn distributed storage
│
├── Projects/                   # Application source code
│   ├── OOAdmin/                #   Admin control plane
│   │   ├── api/                #     FastAPI REST API
│   │   ├── ui/                 #     React/TypeScript dashboard
│   │   ├── ws-broker/          #     WebSocket relay broker
│   │   ├── core/               #     Shared library (DB, K8s, storage, WS)
│   │   └── jobs/               #     Test & benchmark job runners
│   └── OOUser/                 #   User-facing workloads
│       └── computers/          #     VM init containers, VNC/RDP proxies
│
└── docs/                       # Documentation & architecture diagrams
    ├── *.drawio                #   Infrastructure & benchmarking diagrams
    └── img/                    #   Reference screenshots

Prerequisites

  • Terraform >= 1.0
  • Helm >= 3.x
  • kubectl configured for AKS
  • Azure subscription with AKS, Storage, and Managed Identity access
  • uv (Python package manager)
  • Node.js >= 18 (for UI development)

Getting Started

1. Provision Azure Infrastructure

cd Infra-Terraform/k8s
terraform init
terraform plan
terraform apply

2. Deploy Kubernetes Services

# Install storage layer
helm upgrade --install longhorn longhorn/longhorn -f k8s-Helm/OOStorage/longhorn/values.yaml

# Deploy admin services
helm upgrade --install oo-api ./k8s-Helm/OOAdmin/api
helm upgrade --install oo-ui ./k8s-Helm/OOAdmin/ui
helm upgrade --install oo-db ./k8s-Helm/OOAdmin/db
helm upgrade --install oo-ws-broker ./k8s-Helm/OOAdmin/ws-broker

# Deploy analytics
helm upgrade --install elasticsearch ./k8s-Helm/OOAnalytics/elasticsearch
helm upgrade --install grafana ./k8s-Helm/OOAnalytics/grafana
helm upgrade --install langfuse ./k8s-Helm/OOAnalytics/langfuse

3. Run the API Locally (Development)

cd Projects/OOAdmin/api
uv sync
uv run uvicorn server:app --reload

4. Run the UI Locally (Development)

cd Projects/OOAdmin/ui
npm install
npm run dev

Configuration

Variable Description Default
DB_URL MongoDB server hostname —
PORT API service port 8000
LOG_PATH Log directory path —
MY_BASE_URL Root path for API routing —
TEST_JOB_VERSION Test job container image version —
BENCHMARK_JOB_VERSION Benchmark job container image version —

Contributing

Contributions are welcome! Please follow these steps:

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request