fix: fail loudly on non-1536-dim local embeddings (#84)

.env.example

@@ -52,12 +52,22 @@ EMBEDDINGS_PROVIDER=disabled
 
 # Embedding model (used when provider is openai or local).
 # OpenAI: text-embedding-3-small
-# Local:  a sentence-transformers model name (requires raglogs[local-embeddings])
+# Local:  a sentence-transformers model name (requires raglogs[local-embeddings]).
+#         IMPORTANT: the stored embedding column is Vector(1536), so a local
+#         model MUST emit exactly 1536 dims. The popular models do NOT
+#         (all-MiniLM-L6-v2 = 384, all-mpnet-base-v2 = 768) and are rejected
+#         rather than silently producing no embeddings (see EMBEDDINGS_DIMENSIONS
+#         below for exactly how). Use OpenAI or disabled unless you have a
+#         1536-dim local model. Note: provider=local loads the model at startup
+#         and before ingest, which may download it on first use.
 EMBEDDINGS_MODEL=text-embedding-3-small
 
 # Embedding vector dimensions — must match the model output AND the stored
-# log_embeddings column (Vector(1536)). Persist and semantic ask skip when this
-# is not 1536. text-embedding-3-small = 1536, text-embedding-3-large = 3072
+# log_embeddings column (Vector(1536)). Any value other than 1536 with a
+# non-disabled provider is a configuration error (nothing could be persisted).
+# It is surfaced, not silent: `ingest --with-embeddings` hard-exits with an
+# actionable message, and API startup logs it at `error` and keeps serving the
+# non-embedding features. text-embedding-3-small = 1536, text-embedding-3-large = 3072.
 EMBEDDINGS_DIMENSIONS=1536
 
 # Merge fingerprint clusters whose representative messages have cosine

README.md

@@ -591,7 +591,7 @@ Populate vectors at ingest time:
 raglogs ingest ./logs --with-embeddings
 ```
 
-Vectors are stored in `log_embeddings` (1536-d). If `EMBEDDINGS_DIMENSIONS` is not 1536, persist is skipped so the schema stays valid. Provider failures during ingest are skipped; the log lines still land.
+Vectors are stored in `log_embeddings` (1536-d). Because that column width is fixed, a non-disabled provider with `EMBEDDINGS_DIMENSIONS ≠ 1536` — or a `local` model whose native width isn't 1536 — is a configuration error that can never persist a vector. It is surfaced, not silently skipped: **`ingest --with-embeddings` hard-exits** (non-zero) with an actionable message before ingesting, and **API startup logs the misconfig at `error` and keeps serving** the non-embedding features (explain / timeline / compare / keyword `ask`) — coupling the whole API's availability to an optional feature would be worse, especially during an incident. Transient provider failures *during* ingest are still skipped fail-open; the log lines land regardless. (With `EMBEDDINGS_PROVIDER=local`, both the startup check and ingest load the SentenceTransformer model, which may download it on first use.)
 
 **Example output**
 
@@ -702,7 +702,7 @@ All settings are read from `.env`, environment variables, or CLI flags. Priority
 | `ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | Anthropic Messages API host |
 | `EMBEDDINGS_PROVIDER` | `disabled` | `disabled`, `openai`, `local`. Cluster merge, semantic `ask`, and `/similar` ANN skip when `disabled` |
 | `EMBEDDINGS_MODEL` | `text-embedding-3-small` | Embeddings model name |
-| `EMBEDDINGS_DIMENSIONS` | `1536` | Vector size passed to the OpenAI embeddings API. Persist/ask skip unless this is 1536 (stored column width) |
+| `EMBEDDINGS_DIMENSIONS` | `1536` | Vector size. Must be 1536 (the stored column width) for any non-disabled provider; any other value is a hard config error surfaced at startup / before ingest |
 | `CLUSTER_MERGE_SIMILARITY_THRESHOLD` | `0.92` | Cosine similarity at or above which fingerprint clusters merge. High on purpose so distinct errors stay separate |
 | `CLUSTER_MERGE_MIN_COUNT` | `1` | Minimum `count` for a cluster to participate in a merge |
 | `ASK_SEMANTIC_TOP_K` | `100` | Max log lines returned by semantic `ask` |
@@ -913,7 +913,7 @@ Fingerprinting can still split one incident across multiple clusters when wordin
 - **Cluster template persist.** After clustering, raglogs upserts each cluster's representative template into `cluster_embeddings` keyed by `(scope, fingerprint)` when the embeddings provider is available. Similar-incident search queries those rows (not raw `log_embeddings`). Skip happens automatically when `EMBEDDINGS_PROVIDER=disabled`.
 - **Ask vs merge vs similar.** Semantic `ask` uses the *stored* `log_embeddings` table (populated by `raglogs ingest --with-embeddings`) and `ASK_SEMANTIC_MIN_SIMILARITY` (default **0.75**). Cluster merge still uses its own in-memory pass and threshold. Similar-incident search (`POST /v1/query/similar`) uses the `cluster_embeddings` table (upserted at analysis time when an embeddings provider is available) and `SIMILAR_SEMANTIC_MIN_SIMILARITY` (default **0.80**). Compare still applies its own heuristic collapse for webhook retries / queue growth after clustering.
 
-Local embeddings require the optional extra: `pip install 'raglogs[local-embeddings]'` (`sentence-transformers`). If that import fails, merge is skipped.
+Local embeddings require the optional extra: `pip install 'raglogs[local-embeddings]'` (`sentence-transformers`) **and** a model that emits exactly 1536 dims to match the stored column. The common models don't (`all-MiniLM-L6-v2` = 384, `all-mpnet-base-v2` = 768), so `EMBEDDINGS_PROVIDER=local` with one of them is rejected rather than silently producing no embeddings: `ingest --with-embeddings` hard-exits, and API startup logs it at `error` and continues. On the analysis-time merge hot path a missing extra or bad config degrades to skip (logged at `error`), never a crash.
 
 ### Baseline comparison
 

src/api/app.py

@@ -51,6 +51,25 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
     setup_observability(log_format=settings.log_format)
     warn_if_insecure_bind(settings.api_bind_host, settings)
 
+    from src.core.embeddings.provider import (
+        EmbeddingsConfigError,
+        validate_embeddings_config,
+    )
+
+    # A broken embeddings config is logged loudly but does NOT abort startup:
+    # embeddings are an optional feature, and coupling the whole API's
+    # availability to them (taking down explain/timeline/compare/keyword-ask,
+    # especially during an incident) is worse than serving without them. The
+    # `ingest --with-embeddings` path DOES hard-exit on the same error, where a
+    # vector would actually be persisted. Keep this in sync with the README /
+    # .env.example wording.
+    try:
+        validate_embeddings_config(settings)
+    except EmbeddingsConfigError as exc:
+        import structlog
+
+        structlog.get_logger().error("embeddings_config_invalid", reason=str(exc))
+
     if not check_connection():
         import structlog
         log = structlog.get_logger()

src/cli/commands/ingest.py

@@ -55,12 +55,23 @@ def ingest_cmd(
 
     if with_embeddings:
         from src.config import get_settings
+        from src.core.embeddings.provider import (
+            EmbeddingsConfigError,
+            validate_embeddings_config,
+        )
 
-        if get_settings().embeddings_provider == "disabled":
+        settings = get_settings()
+        if settings.embeddings_provider == "disabled":
             console.print(
                 "[yellow]Warning:[/yellow] --with-embeddings requested but "
                 "EMBEDDINGS_PROVIDER=disabled; ingesting without embeddings."
             )
+        else:
+            try:
+                validate_embeddings_config(settings)
+            except EmbeddingsConfigError as e:
+                console.print(f"[red]Error:[/red] {e}")
+                raise typer.Exit(1)
 
     with Progress(
         SpinnerColumn(),

src/core/embeddings/provider.py

@@ -8,14 +8,31 @@
 from typing import Protocol, runtime_checkable
 
 import httpx
+import structlog
 from tenacity import retry, stop_after_attempt, wait_exponential
 
 from src.config import get_settings
 from src.config.settings import Settings
 
+log = structlog.get_logger()
+
 # OpenAI allows large batches; keep requests modest for local-compatible APIs.
 _OPENAI_BATCH_SIZE = 96
 
+# The persisted pgvector column is fixed at this width (migration 0001,
+# ``log_embeddings.embedding`` / ``cluster_embeddings.embedding``). Any provider
+# that emits a different dimension cannot be stored, so we refuse it loudly
+# rather than silently dropping every vector.
+STORED_EMBEDDING_DIMS = 1536
+
+
+class EmbeddingsConfigError(RuntimeError):
+    """A configured embeddings provider cannot possibly persist vectors.
+
+    Raised loudly (at startup / before ingest) instead of letting the storage
+    layer silently skip every write — see issue #84.
+    """
+
 
 @runtime_checkable
 class EmbeddingsProvider(Protocol):
@@ -86,9 +103,16 @@ def _embed_batch(self, texts: list[str]) -> list[list[float]]:
 
 
 class LocalEmbeddingsProvider:
-    """Optional local encoder. Construction raises ImportError if the extra is missing."""
+    """Optional local encoder.
+
+    Construction raises ``ImportError`` if the extra is missing and
+    ``EmbeddingsConfigError`` if the model's native embedding width does not
+    match the stored ``Vector`` column — the common case, since the popular
+    local models (``all-MiniLM-L6-v2`` = 384, ``all-mpnet-base-v2`` = 768) do
+    not emit 1536 dims and would otherwise be dropped silently at persist time.
+    """
 
-    def __init__(self, model: str) -> None:
+    def __init__(self, model: str, expected_dimensions: int = STORED_EMBEDDING_DIMS) -> None:
         try:
             from sentence_transformers import SentenceTransformer
         except ImportError as exc:
@@ -97,6 +121,16 @@ def __init__(self, model: str) -> None:
                 "Install with: pip install 'raglogs[local-embeddings]'"
             ) from exc
         self._model = SentenceTransformer(model)
+        actual = self._model.get_sentence_embedding_dimension()
+        if actual != expected_dimensions:
+            raise EmbeddingsConfigError(
+                f"EMBEDDINGS_PROVIDER=local model '{model}' emits {actual}-dim vectors, "
+                f"but the stored embedding column is Vector({expected_dimensions}). "
+                f"Local embeddings only work with a model that outputs exactly "
+                f"{expected_dimensions} dims. Choose such a model, or use "
+                f"EMBEDDINGS_PROVIDER=openai (text-embedding-3-small = 1536), or "
+                f"EMBEDDINGS_PROVIDER=disabled."
+            )
 
     def is_available(self) -> bool:
         return True
@@ -108,8 +142,15 @@ def embed_texts(self, texts: list[str]) -> list[list[float]]:
         return [vector.tolist() for vector in vectors]
 
 
-def get_embeddings_provider(settings: Settings | None = None) -> EmbeddingsProvider:
-    """Factory: build the configured embeddings provider. Fail-open to disabled."""
+def build_embeddings_provider(settings: Settings | None = None) -> EmbeddingsProvider:
+    """Strict factory: build the configured provider or raise on a broken config.
+
+    Raises ``ImportError`` (local extra missing) or ``EmbeddingsConfigError``
+    (local model dimension mismatch). ``openai`` with no key returns the
+    disabled provider — that is "not configured yet", not a broken config.
+    Use :func:`validate_embeddings_config` at startup and
+    :func:`get_embeddings_provider` on the hot path.
+    """
     if settings is None:
         settings = get_settings()
 
@@ -124,9 +165,54 @@ def get_embeddings_provider(settings: Settings | None = None) -> EmbeddingsProvi
         )
 
     if settings.embeddings_provider == "local":
-        try:
-            return LocalEmbeddingsProvider(model=settings.embeddings_model)
-        except ImportError:
-            return DisabledEmbeddingsProvider()
+        return LocalEmbeddingsProvider(
+            model=settings.embeddings_model,
+            expected_dimensions=STORED_EMBEDDING_DIMS,
+        )
 
     return DisabledEmbeddingsProvider()
+
+
+def get_embeddings_provider(settings: Settings | None = None) -> EmbeddingsProvider:
+    """Hot-path factory: fail-open to disabled so a misconfig never crashes ingest.
+
+    A broken configuration is logged at ``error`` (never silently swallowed) and
+    degrades to the disabled provider. Startup calls
+    :func:`validate_embeddings_config` first, so in practice this path only sees
+    a healthy config or an already-surfaced one.
+    """
+    try:
+        return build_embeddings_provider(settings)
+    except (ImportError, EmbeddingsConfigError) as exc:
+        log.error("embeddings_provider_disabled", reason=str(exc))
+        return DisabledEmbeddingsProvider()
+
+
+def validate_embeddings_config(settings: Settings | None = None) -> None:
+    """Fail loudly if the embeddings configuration can never persist a vector.
+
+    Call at startup / before ingest. Raises :class:`EmbeddingsConfigError` when
+    a non-disabled provider is configured with a dimension the stored column
+    cannot hold, or (for ``local``) a model whose native width does not match.
+    ``disabled`` is always valid. See issue #84.
+    """
+    if settings is None:
+        settings = get_settings()
+
+    if settings.embeddings_provider == "disabled":
+        return
+
+    if settings.embeddings_dimensions != STORED_EMBEDDING_DIMS:
+        raise EmbeddingsConfigError(
+            f"EMBEDDINGS_DIMENSIONS={settings.embeddings_dimensions} but the stored "
+            f"embedding column is Vector({STORED_EMBEDDING_DIMS}). No embeddings can be "
+            f"persisted with this setting. Set EMBEDDINGS_DIMENSIONS={STORED_EMBEDDING_DIMS} "
+            f"(e.g. text-embedding-3-small) or EMBEDDINGS_PROVIDER=disabled."
+        )
+
+    if settings.embeddings_provider == "local":
+        # Constructing loads the model and verifies its real output dimension.
+        try:
+            build_embeddings_provider(settings)
+        except ImportError as exc:
+            raise EmbeddingsConfigError(str(exc)) from exc

src/core/embeddings/store.py

@@ -19,7 +19,11 @@
 
 from src.config import get_settings
 from src.config.settings import Settings
-from src.core.embeddings.provider import EmbeddingsProvider, get_embeddings_provider
+from src.core.embeddings.provider import (
+    STORED_EMBEDDING_DIMS,
+    EmbeddingsProvider,
+    get_embeddings_provider,
+)
 from src.db.models import (
     CLUSTER_EMBEDDING_UNIQUE,
     ClusterEmbedding,
@@ -32,8 +36,8 @@
 
 log = structlog.get_logger()
 
-# Must match ``log_embeddings.embedding Vector(1536)`` in the applied schema.
-STORED_EMBEDDING_DIMS = 1536
+# ``STORED_EMBEDDING_DIMS`` is defined in provider.py (the canonical source) and
+# re-exported here for the existing call sites below.
 
 
 def ingest_embeddings_provider(

tests/unit/test_embeddings_provider.py

@@ -1,16 +1,46 @@
 """Unit tests for the embeddings provider factory. No live API calls."""
 
+import sys
+import types
+from contextlib import contextmanager
 from unittest.mock import MagicMock, patch
 
+import pytest
+
 from src.config.settings import Settings
 from src.core.embeddings.provider import (
+    STORED_EMBEDDING_DIMS,
     DisabledEmbeddingsProvider,
+    EmbeddingsConfigError,
     LocalEmbeddingsProvider,
     OpenAIEmbeddingsProvider,
     get_embeddings_provider,
+    validate_embeddings_config,
 )
 
 
+@contextmanager
+def _fake_sentence_transformers(dim: int):
+    """Patch ``sentence_transformers`` with a stub model of a given output dim."""
+    module = types.ModuleType("sentence_transformers")
+
+    class _FakeSentenceTransformer:
+        def __init__(self, model: str) -> None:
+            self._model = model
+
+        def get_sentence_embedding_dimension(self) -> int:
+            return dim
+
+        def encode(self, texts, convert_to_numpy=True):
+            import numpy as np
+
+            return np.zeros((len(texts), dim))
+
+    module.SentenceTransformer = _FakeSentenceTransformer
+    with patch.dict(sys.modules, {"sentence_transformers": module}):
+        yield
+
+
 def test_disabled_is_unavailable() -> None:
     provider = DisabledEmbeddingsProvider()
     assert provider.is_available() is False
@@ -48,7 +78,7 @@ def test_factory_local_import_error_is_disabled() -> None:
     settings = Settings(_env_file=None, embeddings_provider="local")
 
     class Boom:
-        def __init__(self, model: str) -> None:
+        def __init__(self, model: str, expected_dimensions: int = 1536) -> None:
             raise ImportError("sentence-transformers missing")
 
     with patch("src.core.embeddings.provider.LocalEmbeddingsProvider", Boom):
@@ -69,6 +99,65 @@ def test_local_provider_missing_extra() -> None:
     assert raised is True
 
 
+def test_local_provider_dimension_mismatch_raises() -> None:
+    # 384-dim model (all-MiniLM-L6-v2) can't fill the Vector(1536) column.
+    with _fake_sentence_transformers(dim=384):
+        with pytest.raises(EmbeddingsConfigError) as exc:
+            LocalEmbeddingsProvider(model="all-MiniLM-L6-v2")
+    msg = str(exc.value)
+    assert "384" in msg and str(STORED_EMBEDDING_DIMS) in msg
+
+
+def test_local_provider_matching_dimension_constructs() -> None:
+    with _fake_sentence_transformers(dim=STORED_EMBEDDING_DIMS):
+        provider = LocalEmbeddingsProvider(model="some-1536-model")
+    assert provider.is_available() is True
+
+
+def test_factory_local_dimension_mismatch_fails_open_to_disabled() -> None:
+    # The hot-path factory never raises: it degrades to disabled (logged).
+    settings = Settings(_env_file=None, embeddings_provider="local")
+    with _fake_sentence_transformers(dim=768):
+        provider = get_embeddings_provider(settings)
+    assert isinstance(provider, DisabledEmbeddingsProvider)
+
+
+def test_validate_disabled_is_ok() -> None:
+    validate_embeddings_config(Settings(_env_file=None))  # no raise
+
+
+def test_validate_rejects_non_1536_dimensions() -> None:
+    settings = Settings(
+        _env_file=None,
+        embeddings_provider="openai",
+        openai_api_key="sk-test",
+        embeddings_dimensions=384,
+    )
+    with pytest.raises(EmbeddingsConfigError) as exc:
+        validate_embeddings_config(settings)
+    assert "384" in str(exc.value)
+
+
+def test_validate_local_dimension_mismatch_raises() -> None:
+    settings = Settings(
+        _env_file=None,
+        embeddings_provider="local",
+        embeddings_model="all-mpnet-base-v2",
+        embeddings_dimensions=1536,
+    )
+    with _fake_sentence_transformers(dim=768):
+        with pytest.raises(EmbeddingsConfigError):
+            validate_embeddings_config(settings)
+
+
+def test_validate_local_missing_extra_raises() -> None:
+    settings = Settings(_env_file=None, embeddings_provider="local")
+    with patch.dict(sys.modules, {"sentence_transformers": None}):
+        with pytest.raises(EmbeddingsConfigError) as exc:
+            validate_embeddings_config(settings)
+    assert "sentence-transformers" in str(exc.value)
+
+
 def test_openai_embed_texts_restores_index_order() -> None:
     fake_response = MagicMock()
     fake_response.json.return_value = {