Add TwelveLabs Marengo embedding backend for clip back - #406
Open
mohit-twelvelabs wants to merge 2 commits into
Open
Add TwelveLabs Marengo embedding backend for clip back#406mohit-twelvelabs wants to merge 2 commits into
mohit-twelvelabs wants to merge 2 commits into
Conversation
Author
|
Pushed Root cause: Fix: moved the API-key check ahead of the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
What this adds
An opt-in TwelveLabs Marengo embedding backend for
clip-retrieval back, alongside the existing CLIP / open_clip / hf_clip / DeepSparse backends. Marengo is a video-native multimodal embedding model that embeds text and images into the same vector space as the video segments it produces — useful when your retrieval corpus is video rather than still images.It follows the same prefix-based selection pattern as the DeepSparse (
nm:) and open_clip (open_clip:) backends: pass aclip_modelprefixed withtwelvelabs:, e.g.--clip_model "twelvelabs:marengo3.0".Why it helps
clip-retrieval already supports multiple embedding providers behind one knn service. Marengo extends that to a video-native model served via API, so a backend can answer text/image queries against Marengo embeddings without a local GPU.
Opt-in / non-breaking
clip_retrieval/marengo.py; nothing is imported unless atwelvelabs:model is requested.twelvelabsSDK is an optional dependency (pip install clip-retrieval[twelvelabs]); core install is unchanged.marengo_modelis set.How it was tested
tests/test_marengo.py(model selection, name parsing, encode shape/normalization, missing-key error) — pass.TWELVELABS_API_KEYis set) that does a realmarengo3.0text embedding. Verified locally against the API withtwelvelabs==1.2.8: returns a(1, 512)unit-norm float32 vector.black==23.12.1(repo-pinned) andpylint(10.00/10) clean on the changed files.You can grab a free API key at https://twelvelabs.io — there's a generous free tier.