Skip to content

Add TwelveLabs Marengo embedding backend for clip back - #406

Open
mohit-twelvelabs wants to merge 2 commits into
rom1504:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

Add TwelveLabs Marengo embedding backend for clip back#406
mohit-twelvelabs wants to merge 2 commits into
rom1504:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

What this adds

An opt-in TwelveLabs Marengo embedding backend for clip-retrieval back, alongside the existing CLIP / open_clip / hf_clip / DeepSparse backends. Marengo is a video-native multimodal embedding model that embeds text and images into the same vector space as the video segments it produces — useful when your retrieval corpus is video rather than still images.

It follows the same prefix-based selection pattern as the DeepSparse (nm:) and open_clip (open_clip:) backends: pass a clip_model prefixed with twelvelabs:, e.g. --clip_model "twelvelabs:marengo3.0".

Why it helps

clip-retrieval already supports multiple embedding providers behind one knn service. Marengo extends that to a video-native model served via API, so a backend can answer text/image queries against Marengo embeddings without a local GPU.

Opt-in / non-breaking

  • New module clip_retrieval/marengo.py; nothing is imported unless a twelvelabs: model is requested.
  • twelvelabs SDK is an optional dependency (pip install clip-retrieval[twelvelabs]); core install is unchanged.
  • Default CLIP path and all existing behavior are untouched — the Marengo branch only activates when marengo_model is set.
  • Vectors are L2-normalized to match the existing faiss inner-product convention.

How it was tested

  • No-network unit tests in tests/test_marengo.py (model selection, name parsing, encode shape/normalization, missing-key error) — pass.
  • A gated live smoke test (skipped unless TWELVELABS_API_KEY is set) that does a real marengo3.0 text embedding. Verified locally against the API with twelvelabs==1.2.8: returns a (1, 512) unit-norm float32 vector.
  • black==23.12.1 (repo-pinned) and pylint (10.00/10) clean on the changed files.

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

@mohit-twelvelabs

Copy link
Copy Markdown
Author

Pushed ae258a3 to fix the CI failures on tests (3.10/3.11/3.12).

Root cause: MarengoModel.__init__ imported the optional twelvelabs SDK before validating the API key. Since the SDK isn't installed in the default test env, test_missing_api_key_raises got a ModuleNotFoundError instead of the expected ValueError.

Fix: moved the API-key check ahead of the from twelvelabs import TwelveLabs line, so a missing key raises ValueError regardless of whether the optional SDK is present (also a nicer error for users who haven't installed clip-retrieval[twelvelabs] yet). One-line reorder, no behavior change to the happy path. black + pylint (10.00/10) still clean on the changed file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant