Skip to content

Declare embedding_dim and drop the default tag vocabulary for OpenCLIP RN101 - #43

Merged
andriiryzhkov merged 2 commits into
darktable-org:masterfrom
andriiryzhkov:embed_dim
Sep 6, 2026
Merged

andriiryzhkov merged 2 commits into
darktable-org:masterfrom
andriiryzhkov:embed_dim

Conversation

@andriiryzhkov

Copy link
Copy Markdown
Collaborator

Two changes to the OpenCLIP RN101 package, both about making the manifest describe the model instead of leaving darktable to assume things.

The first declares embedding_dim: 512. darktable sizes its embedding buffers and the vector index column from the model's output width but had no way to learn it, so it hardcoded 512 – which is how a model of a different width could be truncated into an index built for another embedding space. darktable now reads this attribute and refuses a model that does not declare it rather than falling back to a default. That is a coordinated change: the darktable side is a separate PR, and packages built before this key need rebuilding or the updated darktable will refuse them.

The second removes the 86-tag default vocabulary. It was the cold-start path for auto-tagging, but darktable replaced it with centroids built from a user's own tagged images, which describe what a photographer means by a tag far better than a generic label does. The model centroids are no longer applied, and importing them only added 86 unused names to the tag dictionary. Dropping the vocabulary takes the text encoder out of conversion entirely, so convert.py exports the image encoder alone and tags.json is gone from the package. The demo has nothing to overlay without tags, so it writes the embedding vector as JSON instead, and the SDK gains a .json output extension for the task. samples/embed also moves to samples/embedding, which the earlier task rename missed – run_demo resolves samples/<task>, so it had been finding no samples for this model and skipping every image without failing.

I ran the rewritten demo against the already-built model.onnx and it returns dim 512 at norm 1.0000, which confirms both the declared dimension and the baked-in L2 normalisation. Both scripts compile. I did not re-run the conversion, so the ONNX produced by the edited convert.py is unverified, and I did not run the SDK demo runner end to end. No dependency changes.

Written with AI assistance.

@andriiryzhkov
andriiryzhkov merged commit bf878a3 into darktable-org:master Sep 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant