v0.11.0: How can I tell if the server has started OK after I update the server/.env file there is no log entries

[rogermt]

How can I tell if the server has started OK after I update the server/.env file there is no log entries to confirm the connection to S3 is actually working or falling back to local storage

THis is the server log....
Correct login nd incorrect login both fil
Traceback (most recent call last):
File "/teamspace/studios/this_studio/forgesyte/server/.venv/bin/uvicorn", line 10, in
sys.exit(main())
~~~~^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/click/core.py", line 1485, in call
return self.main(*args, **kwargs)
~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/click/core.py", line 1406, in main
rv = self.invoke(ctx)
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/main.py", line 424, in main
run(
~~~^
app,
^^^^
...<46 lines>...
h11_max_incomplete_event_size=h11_max_incomplete_event_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/main.py", line 594, in run
server.run()
~~~~~~~~~~^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/server.py", line 67, in run
return asyncio_run(self.serve(sockets=sockets), loop_factory=self.config.get_loop_factory())
File "/system/conda/miniconda3/uv/python/cpython-3.13.7-linux-x86_64-gnu/lib/python3.13/asyncio/runners.py", line 195, in run
return runner.run(main)
~~~~~~~~~~^^^^^^
File "/system/conda/miniconda3/uv/python/cpython-3.13.7-linux-x86_64-gnu/lib/python3.13/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/server.py", line 71, in serve
await self._serve(sockets)
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/server.py", line 78, in _serve
config.load()
~~~~~~~~~~~^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/config.py", line 439, in load
self.loaded_app = import_from_string(self.app)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/uvicorn/importer.py", line 19, in import_from_string
module = importlib.import_module(module_str)
File "/system/conda/miniconda3/uv/python/cpython-3.13.7-linux-x86_64-gnu/lib/python3.13/importlib/init.py", line 88, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1331, in _find_and_load_unlocked
File "", line 935, in _load_unlocked
File "", line 1026, in exec_module
File "", line 488, in _call_with_frames_removed
File "/teamspace/studios/this_studio/forgesyte/server/app/main.py", line 41, in
from .api_routes.routes.image_submit import router as image_submit_router
File "/teamspace/studios/this_studio/forgesyte/server/app/api_routes/routes/image_submit.py", line 28, in
storage = get_storage_service()
File "/teamspace/studios/this_studio/forgesyte/server/app/services/storage/factory.py", line 18, in get_storage_service
return S3StorageService(
bucket_name=os.getenv("S3_BUCKET_NAME", "forgesyte-jobs"),
...<2 lines>...
secret_key=os.getenv("S3_SECRET_KEY", "")
)
File "/teamspace/studios/this_studio/forgesyte/server/app/services/storage/s3_storage.py", line 41, in init
self.client.head_bucket(Bucket=self.bucket)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/botocore/client.py", line 602, in _api_call
return self._make_api_call(operation_name, kwargs)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/botocore/context.py", line 123, in wrapper
return func(*args, **kwargs)
File "/teamspace/studios/this_studio/forgesyte/server/.venv/lib/python3.13/site-packages/botocore/client.py", line 1078, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadBucket operation: Bad Request
(server) ⚡ v0.11.0 ~/forgesyte/server

FORGESYTE_STORAGE_BACKEND=s3

[rogermt]

This 400 Bad Request error is the "smoking gun" for Virtual Host Addressing.

When you connect to an S3 endpoint using an IP address (like your Tailscale IP http://100.x.x.B:9000), standard S3 clients try to be "smart" and rewrite the URL to http://forgesyte-jobs.100.x.x.B:9000.

This fails instantly because 100.x.x.B is an IP, not a domain you can prepend subdomains to. MinIO receives a request with a Host header it doesn't understand and rejects it as a Bad Request (400).

We need to force Path Style Addressing (e.g., http://100.x.x.B:9000/forgesyte-jobs).

Here is the fix. You need to update server/app/services/storage/s3_storage.py.

The Fix: server/app/services/storage/s3_storage.py

Update the file to import Config and pass it to the client.

"""S3/MinIO storage implementation for Phase 11 (v0.11.0)."""

import os
import tempfile
from pathlib import Path
from typing import BinaryIO

import boto3
from botocore.config import Config  # <--- NEW IMPORT
from botocore.exceptions import ClientError

from app.services.storage.base import StorageService


class S3StorageService(StorageService):
    """S3-compatible storage for distributed job processing."""

    def __init__(
        self,
        bucket_name: str,
        endpoint_url: str,
        access_key: str,
        secret_key: str,
        region_name: str = "us-east-1",
    ) -> None:
        self.bucket = bucket_name
        
        # FIX: Force Path Style addressing because we are using an IP address (Tailscale)
        # Also force s3v4 signature to satisfy MinIO requirements
        s3_config = Config(
            s3={'addressing_style': 'path'}, 
            signature_version='s3v4'
        )

        self.client = boto3.client(
            "s3",
            endpoint_url=endpoint_url,
            aws_access_key_id=access_key,
            aws_secret_access_key=secret_key,
            region_name=region_name,
            config=s3_config  # <--- Apply the config
        )

        # Ensure bucket exists on startup
        try:
            self.client.head_bucket(Bucket=self.bucket)
        except ClientError as e:
            # Handle standard 404 (Not Found)
            error_code = int(e.response['Error']['Code'])
            if error_code == 404:
                self.client.create_bucket(Bucket=self.bucket)
            else:
                raise

    def save_file(self, src: BinaryIO, dest_path: str) -> str:
        """Upload file-like object to S3."""
        src.seek(0)
        self.client.upload_fileobj(src, self.bucket, dest_path)
        return dest_path

    def load_file(self, path: str) -> Path:
        """Download file from S3 to a local temp file.
        Preserves suffix so OpenCV knows it is an .mp4.
        """
        try:
            suffix = Path(path).suffix
            tmp = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)
            
            self.client.download_fileobj(self.bucket, path, tmp)
            tmp.close()
            
            return Path(tmp.name)
        except ClientError as e:
            if e.response['Error']['Code'] == '404':
                raise FileNotFoundError(f"File not found in S3: {path}") from e
            raise

    def delete_file(self, path: str) -> None:
        """Delete a stored file, if it exists."""
        self.client.delete_object(Bucket=self.bucket, Key=path)

    def file_exists(self, path: str) -> bool:
        """Check if a file exists in storage."""
        try:
            self.client.head_object(Bucket=self.bucket, Key=path)
            return True
        except ClientError:
            return False

Why this fixes it

1. addressing_style='path': Tells boto3 to send requests to http://IP:PORT/bucket instead of trying http://bucket.IP:PORT.
2. signature_version='s3v4': Ensures we use the modern signature standard that newer MinIO versions enforce.

Apply this change to s3_storage.py and restart the server. It should connect instantly.

[rogermt]

The existing test needs updting https://raw.githubusercontent.com/rogermt/forgesyte/refs/heads/v0.11.0/server/tests/app/services/test_storage_s3.py

[rogermt]

These tests are excellent. They cover the exact edge cases we need (suffixes, non-existent files, factory defaults).

To ensure your Tests Pass (specifically handling the empty ENDPOINT_URL in the fixture) AND the Runtime Works (fixing the MinIO 400 Bad Request you saw earlier), I have refined the code slightly.

Here is the Final, Test-Compliant Code for v0.11.0.

1. server/app/services/storage/s3_storage.py

• Change: Added botocore.config.Config to force "path-style" addressing (Fixes the 400 error).
• Change: improved ClientError handling to catch both '404' and 'NoSuchKey' (Fixes test_load_nonexistent_file).

"""S3/MinIO storage implementation for Phase 11 (v0.11.0)."""

import os
import tempfile
from pathlib import Path
from typing import BinaryIO

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

from app.services.storage.base import StorageService


class S3StorageService(StorageService):
    """S3-compatible storage for distributed job processing."""

    def __init__(
        self,
        bucket_name: str,
        endpoint_url: str,
        access_key: str,
        secret_key: str,
        region_name: str = "us-east-1",
    ) -> None:
        self.bucket = bucket_name
        
        # FIX: Force Path Style addressing for IP-based MinIO URLs (Tailscale)
        # FIX: Force s3v4 signature for modern MinIO compatibility
        s3_config = Config(
            s3={'addressing_style': 'path'}, 
            signature_version='s3v4'
        )

        self.client = boto3.client(
            "s3",
            endpoint_url=endpoint_url,
            aws_access_key_id=access_key,
            aws_secret_access_key=secret_key,
            region_name=region_name,
            config=s3_config
        )

        # Ensure bucket exists on startup
        try:
            self.client.head_bucket(Bucket=self.bucket)
        except ClientError as e:
            # Check for 404 (Not Found)
            error_code = str(e.response.get('Error', {}).get('Code', ''))
            if error_code == '404':
                self.client.create_bucket(Bucket=self.bucket)
            else:
                raise

    def save_file(self, src: BinaryIO, dest_path: str) -> str:
        """Upload file-like object to S3."""
        src.seek(0)
        self.client.upload_fileobj(src, self.bucket, dest_path)
        return dest_path

    def load_file(self, path: str) -> Path:
        """Download file from S3 to a local temp file.
        Preserves suffix so OpenCV knows it is an .mp4.
        """
        try:
            suffix = Path(path).suffix
            # Delete=False is required so we can return the path and let the caller open it
            tmp = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)
            
            self.client.download_fileobj(self.bucket, path, tmp)
            tmp.close()
            
            return Path(tmp.name)
        except ClientError as e:
            error_code = str(e.response.get('Error', {}).get('Code', ''))
            # S3 returns '404' for HeadObject, 'NoSuchKey' for GetObject
            if error_code == '404' or error_code == 'NoSuchKey':
                raise FileNotFoundError(f"File not found in S3: {path}") from e
            raise

    def delete_file(self, path: str) -> None:
        """Delete a stored file, if it exists."""
        self.client.delete_object(Bucket=self.bucket, Key=path)

    def file_exists(self, path: str) -> bool:
        """Check if a file exists in storage."""
        try:
            self.client.head_object(Bucket=self.bucket, Key=path)
            return True
        except ClientError:
            return False

2. server/app/services/storage/factory.py

• Change: Added .lower() to backend check for robustness.
• Change: Added or None to endpoint. This is crucial for your test s3_env fixture which sets S3_ENDPOINT_URL="". Boto3 will error on an empty string, but works fine with None.

"""Storage service factory for dependency injection."""

import os
import logging
from functools import lru_cache
from dotenv import load_dotenv

# Ensure env vars are loaded
load_dotenv()

from app.services.storage.base import StorageService
from app.services.storage.local_storage import LocalStorageService
from app.services.storage.s3_storage import S3StorageService

logger = logging.getLogger(__name__)

@lru_cache()
def get_storage_service() -> StorageService:
    """Returns the configured storage backend (Singleton)."""
    backend = os.getenv("FORGESYTE_STORAGE_BACKEND", "local").lower()

    if backend == "s3":
        # Handle empty string from tests by converting to None
        endpoint = os.getenv("S3_ENDPOINT_URL") or None
        
        logger.info(f"🔌 Storage Backend: S3 (MinIO) enabled at {endpoint}")
        
        return S3StorageService(
            bucket_name=os.getenv("S3_BUCKET_NAME", "forgesyte-jobs"),
            endpoint_url=endpoint,
            access_key=os.getenv("S3_ACCESS_KEY", ""),
            secret_key=os.getenv("S3_SECRET_KEY", "")
        )

    logger.info("📁 Storage Backend: Local Filesystem")
    return LocalStorageService()

3. Run the Tests

Now you can execute the test suite you provided. It should pass green.

cd server
uv run pytest tests/services/test_storage_factory.py -v

Once the tests pass, restart your server. The Config change in s3_storage.py will fix the "Bad Request" error, and your distributed storage will be live!