fix: prefer native sparse embeddings when available

[rogernogueira]

Summary

This PR fixes Qdrant hybrid retrieval behavior by preferring embedder-native sparse vectors when available, while preserving the existing FastEmbed BM25 fallback.

fixes #7432

Problem

Currently, the Agno Qdrant integration always instantiates FastEmbed's SparseTextEmbedding (BM25) when search_type=hybrid`, even when the configured embedder already provides sparse vectors natively.

This creates two issues:

1. hybrid retrieval may combine dense vectors from the configured embedder with sparse vectors from a different representation space
2. FastEmbed becomes an unnecessary dependency for embedders that already support sparse output

A real example is bge-m3, which can produce sparse vectors natively.

Type of change

• [ X ] Bug fix
• New feature
• Breaking change
• Improvement
• Model update
• Other:

---

Summary of Changes

• The system now checks for native sparse embedding support first. If found, it skips the FastEmbed setup to save resources.
• Added _get_sparse_vector(), which prioritizes native embeddings and only uses FastEmbed as a fallback.
• Both hybrid and keyword search paths now use this new logic to retrieve sparse vectors.
• Updated _build_search_results to handle custom metadata (like doc_id) without triggering KeyError crashes.
• Added a cookbook example showing a custom embedder with native sparse vectors in Qdrant hybrid search.
• Updated document insert/upsert paths to use the same native-sparse-first resolution as search.

Checklist

• Code complies with style guidelines
• Ran format/validation scripts (./scripts/format.sh and ./scripts/validate.sh)
• Self-review completed
• Documentation updated (comments, docstrings)
• Examples and guides: Relevant cookbook examples have been included or updated (if applicable)
• Tested in clean environment
• Tests added/updated (if applicable)

Duplicate and AI-Generated PR Check

• I have searched existing open pull requests and confirmed that no other PR already addresses this issue
• If a similar PR exists, I have explained below why this PR is a better approach
• Check if this PR was entirely AI-generated (by Copilot, Claude Code, Cursor, etc.)

---

Validation

Added unit tests covering:

• native sparse priority in _get_sparse_vector()
• FastEmbed fallback when native sparse output is unavailable
• empty input handling
• keyword search using the sparse helper
• hybrid search using the sparse helper
• clear errors when sparse vectors cannot be generated

These tests validate the new native-sparse-first behavior while preserving the existing FastEmbed fallback.

[github-actions]

PR Triage

A few things to address before this PR can be reviewed:

Missing issue link: Please link the issue this PR addresses using fixes #<issue_number>, closes #<issue_number>, or resolves #<issue_number> in the PR description. If there is no existing issue, please create one first.

---

Missing tests: This PR modifies source code but does not include any test changes. Please add or update tests to cover your changes.

[rogernogueira]

Add tests

[sannya-singal]

Please make sure to fix the failing pipeline as well @rogernogueira.

[rogernogueira]

add codebook

[sannya-singal]

@rogernogueira please make sure that gh pipeline is green.

[rogernogueira]

updated the PR to keep sparse vector resolution consistent across the Qdrant paths.

Changes included:

_get_sparse_vector() helper
helper usage in hybrid search
helper usage in keyword search
helper usage in insert / async_insert
unit tests covering native sparse priority and fallback behavior

The branch has also been updated with the latest changes and tests are passing locally.

[rogernogueira]

I tried to simplify the PR.

[rogernogueira]

I tried to simplify the PR.

[sannya-singal]

Why did we revert the cookbook?

Please ensure a green CI pipeline @rogernogueira.

[rogernogueira]

I’ve added a cookbook for this change.

[rogernogueira]

add test

[rogernogueira]

Pls review the changes