v0,11.0: Shared storage as the backbone

[rogermt]

use Tailscale for everything.**

In fact, dropping Cloudflare Tunnels for your internal node-to-node communication is the best architectural decision you can make.

Cloudflare Tunnels are great for exposing a website to the public internet, but they are HTTP-first and add overhead. Tailscale creates a true, encrypted peer-to-peer Virtual Private Cloud (VPC). Your Laptop and Lightning AI will act exactly as if they are plugged into the same network switch, communicating over their private 100.x.x.x IP addresses.

Here is the v0.11.0 Architecture Diagram using 100% Tailscale.

🗺️ v0.11.0 Architecture Diagram (Tailscale-Only)

Note: In v0.11.0 (Phase A), Lightning AI is ONLY acting as your Data Lake. The compute (JobWorker) hasn't moved to Lightning yet—that happens in v0.12.0 (Phase B).

======================= TAILSCALE SECURE NETWORK (100.x.x.x) =======================
                      (Encrypted, Peer-to-Peer, No Open Ports)

      [ YOUR LAPTOP ]                                 [ LIGHTNING AI NODE ]
    (Tailscale IP: 100.x.x.A)                       (Tailscale IP: 100.x.x.B)
    Control Plane & Local Compute                   Storage Backbone
                                                
+-----------------------------------+           +----------------------------------+
|                                   |           |                                  |
|  💻 Web UI (React)                |           |   🗄️ MinIO Server (Docker)        |
|    |                              |           |      - Port 9000 (S3 API)        |
|    v                              |           |      - Port 9001 (Web Console)   |
|  ⚙️ FastAPI (Port 8000)           |           |                                  |
|    |-- /v1/video/submit           |           |   💾 Persistent Storage          |
|    |-- S3StorageService  ============(S3)=======>    /workspace/minio/data       |
|                                   |           |                                  |
|  🗃️ DuckDB (Local state)          |           +----------------------------------+
|    (Tracks job status:            |                           ^
|     pending -> running -> done)   |                           |
|                                   |                           |
|  👷 JobWorker (Python Thread)     |                           |
|    |-- Polls DuckDB               |                           |
|    |-- S3StorageService  ============(S3)=====================+
|    |-- Runs CPU Plugins locally   |
|                                   |
+-----------------------------------+

🔄 The v0.11.0 Data Flow (Step-by-Step)

When you upload a video in v0.11.0, here is exactly what happens over the Tailscale network:

1. Upload: You drop a video into the Web UI on your laptop.
2. Store to Lightning: FastAPI takes the file and, using boto3, streams it directly over Tailscale to Lightning AI (http://100.x.x.B:9000/forgesyte-jobs/video/input/job-123.mp4).
3. Database Update: FastAPI updates local DuckDB: status=pending.
4. Worker Picks Up: The JobWorker running in the background on your laptop sees the pending job.
5. Download to Temp: The Laptop's JobWorker downloads the MP4 from Lightning AI (via Tailscale) to a local /tmp/ file so OpenCV can read it.
6. Local Execution: The local CPU plugin runs.
7. Results Saved: The Laptop's JobWorker uploads the resulting output.json back to MinIO on Lightning AI over Tailscale.
8. UI Updates: The Web UI polls FastAPI, which fetches the JSON from MinIO and renders the bounding boxes.

🛠️ How to configure this right now

1. On Lightning AI:
Install Tailscale, log in, and start MinIO:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscaled &
sudo tailscale up  # Follow the link to log in

# Start MinIO mapping to port 9000
docker run -d --name minio --restart unless-stopped -p 9000:9000 -p 9001:9001 -v /workspace/minio/data:/data -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin minio/minio server /data --console-address ":9001"

Run tailscale ip -4 to get Lightning's IP (e.g., 100.100.100.5).

2. On Your Laptop (server/.env):
Update your environment variables to point boto3 to the Lightning Tailscale IP.

FORGESYTE_STORAGE_BACKEND=s3
S3_ENDPOINT_URL=http://100.100.100.5:9000   <--- Lightning's Tailscale IP
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET_NAME=forgesyte-jobs

Why this makes v0.12.0 (Ray) incredibly easy

By doing this now, your data gravity is already established on the GPU node.

When we move to Phase B (v0.12.0) and move the JobWorker to Lightning AI using Ray, the Ray worker will fetch the video from http://localhost:9000 (since it's on the same machine as MinIO). The download speed will be near-instantaneous (disk-speed), completely bypassing the network.

Are you ready for the s3_storage.py and factory.py code to lock this in?

[rogermt]

I have taken a step back, reviewed the entire architecture, and aligned this perfectly with your existing StorageService protocol.

Here is the exact, production-ready code for s3_storage.py and factory.py. It explicitly handles the file_exists method (which your video_submit.py relies on) and preserves file extensions when downloading to a temp file (which OpenCV requires for .mp4 parsing).

1. Create server/app/services/storage/s3_storage.py

Create this new file to implement the S3 backend. It downloads files to local temporary storage so that your OpenCV plugins (cv2.VideoCapture) continue to work exactly as they do today without any changes.

"""S3/MinIO storage implementation for Phase 11 (v0.11.0)."""

import os
import tempfile
from pathlib import Path
from typing import BinaryIO

import boto3
from botocore.exceptions import ClientError

from app.services.storage.base import StorageService


class S3StorageService(StorageService):
    """S3-compatible storage for distributed job processing."""

    def __init__(
        self,
        bucket_name: str,
        endpoint_url: str,
        access_key: str,
        secret_key: str,
        region_name: str = "us-east-1",
    ) -> None:
        self.bucket = bucket_name
        self.client = boto3.client(
            "s3",
            endpoint_url=endpoint_url,
            aws_access_key_id=access_key,
            aws_secret_access_key=secret_key,
            region_name=region_name,
        )

        # Ensure bucket exists on startup
        try:
            self.client.head_bucket(Bucket=self.bucket)
        except ClientError as e:
            error_code = int(e.response['Error']['Code'])
            if error_code == 404:
                self.client.create_bucket(Bucket=self.bucket)
            else:
                raise

    def save_file(self, src: BinaryIO, dest_path: str) -> str:
        """Upload file-like object to S3.
        
        Args:
            src: File-like object (BinaryIO) to save
            dest_path: Destination path relative to storage root
            
        Returns:
            Relative path where file was saved
        """
        src.seek(0)
        self.client.upload_fileobj(src, self.bucket, dest_path)
        return dest_path

    def load_file(self, path: str) -> Path:
        """Download file from S3 to a local temp file.
        
        Returns a local filesystem Path object so OpenCV and existing 
        file operations continue to work unmodified.
        
        Args:
            path: Path relative to storage root
            
        Returns:
            Local filesystem Path object
            
        Raises:
            FileNotFoundError: If file does not exist in S3
        """
        try:
            # Preserve the extension so OpenCV knows it's an .mp4
            suffix = Path(path).suffix
            tmp = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)
            
            self.client.download_fileobj(self.bucket, path, tmp)
            tmp.close()
            
            return Path(tmp.name)
        except ClientError as e:
            if e.response['Error']['Code'] == '404':
                raise FileNotFoundError(f"File not found in S3: {path}") from e
            raise

    def delete_file(self, path: str) -> None:
        """Delete a stored file, if it exists."""
        self.client.delete_object(Bucket=self.bucket, Key=path)

    def file_exists(self, path: str) -> bool:
        """Check if a file exists in storage.
        
        Note: This is required by video_submit.py
        """
        try:
            self.client.head_object(Bucket=self.bucket, Key=path)
            return True
        except ClientError:
            return False

2. Create server/app/services/storage/factory.py

This factory enforces the Singleton pattern (via @lru_cache) so we only establish the boto3 connection pool once.

"""Storage service factory for dependency injection."""

import os
from functools import lru_cache

from app.services.storage.base import StorageService
from app.services.storage.local_storage import LocalStorageService
from app.services.storage.s3_storage import S3StorageService

@lru_cache()
def get_storage_service() -> StorageService:
    """Returns the configured storage backend (Singleton)."""
    backend = os.getenv("FORGESYTE_STORAGE_BACKEND", "local").lower()

    if backend == "s3":
        return S3StorageService(
            bucket_name=os.getenv("S3_BUCKET_NAME", "forgesyte-jobs"),
            endpoint_url=os.getenv("S3_ENDPOINT_URL"),
            access_key=os.getenv("S3_ACCESS_KEY"),
            secret_key=os.getenv("S3_SECRET_KEY")
        )

    return LocalStorageService()

3. Wire It Into the App

You must update 6 files to replace the hardcoded LocalStorageService with the new factory.

Run these specific replacements in the following files:

A. In server/app/api_routes/routes/image_submit.py

B. In server/app/api_routes/routes/video_submit.py

C. In server/app/api_routes/routes/jobs.py

D. In server/app/api_routes/routes/job_results.py

Change this:

from app.services.storage.local_storage import LocalStorageService

router = APIRouter()
storage = LocalStorageService()

To this:

from app.services.storage.factory import get_storage_service

router = APIRouter()
storage = get_storage_service()

E. In server/app/workers/run_job_worker.py

(This is the background thread running inside FastAPI and the standalone CLI)

Change this:

from app.services.storage.local_storage import LocalStorageService

To this:

from app.services.storage.factory import get_storage_service

And in both run_worker_forever() and main() functions:
Change this:

storage = LocalStorageService()

To this:

storage = get_storage_service()

F. In server/tests/conftest.py (Update Mock Wiring)

To ensure your tests don't break because of the new SessionLocal dependency mocking context, you should leave your tests as they are for now, because tests use mock_session_local successfully. The get_storage_service will default to "local" in pytest unless overridden, which keeps tests green.

---

The Final Step

Run:

pip install boto3

Set your .env on your laptop to point to the Tailscale IP of your Lightning node:

FORGESYTE_STORAGE_BACKEND=s3
S3_ENDPOINT_URL=http://100.x.x.B:9000
S3_ACCESS_KEY=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET_NAME=forgesyte-jobs

Restart your FastAPI server. The abstraction is cleanly locked in. You are completely decoupled from the local filesystem.

Here is the definitive, step-by-step checklist for v0.11.0 (Phase A: Shared Storage).

Follow this strictly. Do not move to Ray (v0.12.0) until every box is checked.

🌐 1. Infrastructure (Tailscale & MinIO)

• Install Tailscale on Laptop: Authenticate and get your Laptop's 100.x.x.A IP.
• Install Tailscale on Lightning AI: Authenticate and get Lightning's 100.x.x.B IP.
• Verify Connection: Ping the Lightning Tailscale IP from your laptop terminal (ping 100.x.x.B).
• Start MinIO on Lightning AI:

  docker run -d --name minio --restart unless-stopped -p 9000:9000 -p 9001:9001 -v /workspace/minio/data:/data -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin minio/minio server /data --console-address ":9001"

• Verify MinIO Console: Open http://100.x.x.B:9001 in your laptop's browser and log in (minioadmin / minioadmin).

📦 2. Dependencies & Config

• Install boto3: Run uv add boto3 (or pip install boto3) on your laptop.
• Update .env: Add the S3 configuration to your laptop's server/.env file:

  FORGESYTE_STORAGE_BACKEND=s3
  S3_ENDPOINT_URL=http://100.x.x.B:9000  # Replace with actual Lightning Tailscale IP
  S3_ACCESS_KEY=minioadmin
  S3_SECRET_KEY=minioadmin
  S3_BUCKET_NAME=forgesyte-jobs

💻 3. Code Implementation

• Create s3_storage.py: Create server/app/services/storage/s3_storage.py and paste the S3StorageService class (from the previous message).
• Create factory.py: Create server/app/services/storage/factory.py and paste the get_storage_service() function.
• Refactor image_submit.py: Replace LocalStorageService() with get_storage_service().
• Refactor video_submit.py: Replace LocalStorageService() with get_storage_service().
• Refactor jobs.py: Replace LocalStorageService() with get_storage_service().
• Refactor job_results.py: Replace LocalStorageService() with get_storage_service().
• Refactor worker.py / run_job_worker.py: Replace LocalStorageService() with get_storage_service() in both run_worker_forever and main.

✅ 4. Acceptance Testing (End-to-End)

• Restart FastAPI: Restart your server on the laptop. Check the logs to ensure no startup errors.
• Submit a Job: Open the Web UI, upload a video (or image), and click "Run Job".
• Verify Upload: Check the MinIO Web Console (http://100.x.x.B:9001). Look inside the forgesyte-jobs bucket. You should see the .mp4 file in video/input/.
• Verify Worker Execution: Look at your FastAPI terminal logs. The JobWorker should detect the pending job, download the file to /tmp/, run the plugin, and log "saved results to video/output/...".
• Verify Result Output: Check the MinIO console again. You should now see a .json file in video/output/.
• Verify UI Render: The Web UI should fetch the completed status, download the JSON result from MinIO, and render the bounding boxes or results successfully.
• Run Test Suite: Run uv run pytest to ensure local tests still pass (they should automatically fall back to local storage since the test .env won't have FORGESYTE_STORAGE_BACKEND=s3).

---

Status Check: If all 4 phases are green, v0.11.0 is complete. Your application's state is now officially decoupled from the local filesystem, and you are ready for Phase B (Ray).

[rogermt]

I had changed force on me but Minio has to be on the laptop because Lighting overwrites the services because restarts with new vm

[rogermt]

That is a very smart pivot. Ephemeral cloud environments wiping out background services (like Docker containers) on reboot is a classic trap.

Moving MinIO to your laptop is actually the safest configuration. It turns your laptop into the permanent Control Plane & Data Lake, while Lightning AI becomes purely Stateless Ephemeral Compute. If Lightning restarts, you lose zero data and zero configuration.

The great news? Zero code changes are required. The only thing that changes is where you run the docker run command and what IP address goes into your .env.

Here is the updated architecture and the revised checklist.

---

🗺️ Revised v0.11.0 Architecture (MinIO on Laptop)

======================= TAILSCALE SECURE NETWORK (100.x.x.x) =======================

      [ YOUR LAPTOP ]                                 [ LIGHTNING AI NODE ]
  (Tailscale IP: 100.x.x.A)                         (Tailscale IP: 100.x.x.B)
  Control Plane & Data Lake                         Stateless Compute (Phase B)
                                                
+-----------------------------------+           +----------------------------------+
|                                   |           |                                  |
|  🗄️ MinIO Server (Docker)        |           |  (Idle in v0.11.0)               |
|      - Port 9000 (S3 API)         | <======   |  (Will run Ray Worker in v0.12.0)|
|      - Port 9001 (Web Console)    |  (Pull/   |                                  |
|                                   |   Push)   |                                  |
|  💻 Web UI (React)                |           +----------------------------------+
|    |                              |                           
|    v                              |                           
|  ⚙️ FastAPI (Port 8000)           |                           
|    |-- /v1/video/submit           |                           
|    |-- S3StorageService           |                           
|                                   |                           
|  🗃️ DuckDB (Local state)          |                           
|                                   |                           
|  👷 JobWorker (Python Thread)     |                           
|    |-- S3StorageService           |                           
|                                   |
+-----------------------------------+

⚠️ Crucial Configuration Detail

Because MinIO is on your laptop, you could use http://localhost:9000 in your .env. Do not do this.

If you use localhost, then in Phase B, when the Lightning AI worker tries to connect to S3, it will look at its own localhost and fail.
You must use your Laptop's Tailscale IP (e.g., http://100.x.x.A:9000) in the .env. This ensures both the laptop and the Lightning worker resolve the storage to the exact same place.

---

✅ The Revised v0.11.0 Checklist

🌐 1. Infrastructure (MinIO & Tailscale on Laptop)

• Install/Verify Tailscale on Laptop: Get your laptop's Tailscale IP (e.g., 100.x.x.A).
• Start MinIO locally on your Laptop:
  Run this in your laptop's terminal:

  docker run -d --name minio \
    --restart unless-stopped \
    -p 9000:9000 -p 9001:9001 \
    -v ${PWD}/minio_data:/data \
    -e MINIO_ROOT_USER=minioadmin \
    -e MINIO_ROOT_PASSWORD=minioadmin \
    minio/minio server /data --console-address ":9001"

• Verify MinIO Console: Open http://localhost:9001 in your browser and log in (minioadmin / minioadmin).

📦 2. Dependencies & Config (Laptop)

• Install boto3: Run uv pip install boto3 (or pip install boto3) on your laptop.
• Update .env: Add the S3 configuration to your laptop's server/.env file. Make sure to use the Tailscale IP:

  FORGESYTE_STORAGE_BACKEND=s3
  S3_ENDPOINT_URL=http://100.x.x.A:9000  # <-- YOUR LAPTOP'S TAILSCALE IP
  S3_ACCESS_KEY=minioadmin
  S3_SECRET_KEY=minioadmin
  S3_BUCKET_NAME=forgesyte-jobs

💻 3. Code Implementation (Laptop)

• Create s3_storage.py: Create server/app/services/storage/s3_storage.py (code from previous step).
• Create factory.py: Create server/app/services/storage/factory.py (code from previous step).
• Refactor Router endpoints: Replace LocalStorageService() with get_storage_service() in:
  • server/app/api_routes/routes/image_submit.py
  • server/app/api_routes/routes/video_submit.py
  • server/app/api_routes/routes/jobs.py
  • server/app/api_routes/routes/job_results.py
• Refactor worker.py: Replace LocalStorageService() with get_storage_service() in server/app/workers/run_job_worker.py (both run_worker_forever and main).

🧪 4. Acceptance Testing (End-to-End)

• Restart FastAPI: Restart your server on the laptop.
• Submit a Job: Open the Web UI, upload a video (or image), and click "Run Job".
• Verify Upload: Check the local MinIO Web Console (http://localhost:9001). You should see the .mp4 file in forgesyte-jobs/video/input/.
• Verify Worker Execution: Watch your FastAPI terminal. The JobWorker should download the file to a temp folder, run the plugin, and upload the output.json back to MinIO.
• Verify UI Render: The Web UI should fetch the completed status, download the JSON result from MinIO, and render the bounding boxes successfully.

Once this works, your local backend is completely agnostic to where files live. When Lightning AI boots up for Phase B, it will simply join the Tailscale network, ask the laptop for jobs, and pull the video bytes straight from your laptop's MinIO.