🚀 Code Execution Engine (CEE)

A secure, distributed, container-isolated engine for executing untrusted code at scale.

---

📌 Overview

The Code Execution Engine (CEE) is a distributed, stateless, and language-agnostic platform designed to safely execute untrusted user code inside secure sandboxes. It uses Docker Swarm, gVisor, and Redis to achieve:

• 🔐 High security (kernel-level sandboxing via gVisor)
• ⚡ High scalability (Swarm micro-services model)
• 🧩 Language flexibility (Python, Node, Go, etc.)
• 🧱 Strong failure isolation (FMEA-driven design)

---

📑 Table of Contents

• Architecture Diagram
• Project Structure
• Key Components
• Development Workflow
• Data Flow
• Security Model
• Reliability & Failure Mitigation
• API Reference
• License

---

🏗 Architecture Diagram

flowchart LR
    A[Client API Request /exec] --> B[Orchestrator]
    B -->|Write Job| R[(Redis)]
    B -->|Push UUID| Q[queue:pending]

    B -->|Create Swarm Service| S[Worker Node + gVisor]
    S -->|Bootstrapper Fetches Job| R
    S -->|Executes User Code| S2((Sandbox))

    S -->|Write Result| R
    B -->|Webhook Callback| C[Client]

Loading

---

📂 Project Structure

codeexec-engine/
├── cmd/
│   ├── orchestrator/             # API, Scheduler, Reaper
│   └── bootstrapper/             # PID 1 inside container
│
├── internal/
│   ├── orchestrator/             # Hidden from container
│   └── bootstrapper/             # Hidden from host
│
├── pkg/
│   ├── types/                    # Shared Job/Result structs
│   └── store/                    # Redis key conventions
│
├── images/
│   ├── base/                     # Bootstrapper builder
│   └── python/                   # Python runner image
│
└── scripts/
    └── swarm_init.sh             # Networking + firewall setup

---

🧠 Key Components

1. Orchestrator (Host Binary)

• Exposes REST API (/exec)
• Writes job metadata to Redis
• Dispatches jobs as Docker Swarm Services
• Enforces concurrency limits
• Performs job cleanup & webhook dispatching

2. Bootstrapper (Container Binary)

• Runs as PID 1 inside sandbox

• Fetches job payload from Redis

• Executes code under:

  • unprivileged user
  • resource limits (setrlimit)
  • restricted filesystem

• Streams stdout/stderr back to Redis

3. Redis

• Indexed job metadata (job:{uuid})
• Pending and processing queues
• Optional Pub/Sub notifications

4. Docker Swarm + gVisor

• Creates one-off secure containers
• gVisor (runsc) isolates kernel surface
• Prevents container escape and syscall abuses

---

🛠 Development Workflow

1. Initialize Swarm & Restricted Network

./scripts/swarm_init.sh

2. Build Images

# Build bootstrapper
docker build -t cee-base -f images/base/Dockerfile .

# Build Python runtime
docker build -t registry/runner-python:3.11 -f images/python/Dockerfile .

3. Run Orchestrator

export REDIS_ADDR="localhost:6379"
go run cmd/orchestrator/main.go

---

🔄 Data Flow

Phase 1 — Ingestion

1. Client sends POST /exec

2. Orchestrator:

   • Generates UUID
   • Stores job metadata in Redis
   • Pushes UUID → queue:pending

3. Returns UUID immediately

Phase 2 — Scheduling

1. Orchestrator polls queue:pending

2. Resolves language → image tag

3. Creates a Swarm Service with:

   • JOB_ID
   • runtime=runsc (gVisor)

4. Moves UUID → queue:processing

Phase 3 — Execution

Inside container:

• Bootstrapper loads job
• Applies rlimits, user sandboxing
• Executes code
• Writes results back to Redis

Phase 4 — Finalization

• Orchestrator detects completion
• Sends webhook to client
• Removes Swarm service

---

🔐 Security Model

Layer	Security Mechanism
Kernel	gVisor (runsc) syscall interception
Network	Strict overlay network, outbound allowlist
User Privileges	Non-root (uid:1001)
Filesystem	Read-only root, writable /tmp only
Resource Limits	CPU, memory, and timeout enforcement

---

🛡 Reliability & Failure Mitigation

✔ Zombie Jobs

If a job remains in queue:processing beyond max timeout:

• Reaper checks Swarm task existence
• If missing → mark job as failed

✔ Swarm Saturation

Semaphore ensures max concurrent jobs threshold.

✔ Webhook Failures

Retry mechanism with exponential backoff.

---

📡 API Reference

POST /exec

{
  "codeid": "submission-8821",
  "code": "print(input())",
  "language": "python",
  "stdin": ["Test Case 1", "Test Case 2"],
  "limits": {
    "memory": 128,
    "cpu": 0.5,
    "timeout": 2000
  },
  "webhook_url": "https://client-api.com/callback"
}

Webhook Callback Example

{
  "id": "550e8400...",
  "status": "completed",
  "stdout": ["Test Case 1", "Test Case 2"],
  "metrics": {
    "cpu_usage": 12,
    "exec_time": 45
  }
}