`insert_chunks` should use upsert semantics — providers silently duplicate or error on repeated chunk IDs

[jakub-walaszczyk]

System Info

• OGX version: current main (commit 57886c8bd)
• Python version: 3.12+
• Affected components: src/ogx_api/vector_io/api.py, all VectorIO provider implementations

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

The VectorIO protocol defines insert_chunks (src/ogx_api/vector_io/api.py:58) with no documented contract for duplicate chunk_id handling. Five providers already implement upsert semantics (PGVector, Qdrant, Elasticsearch, OCI 26AI, Infinispan), but the remaining five use pure insert — causing silent data duplication (FAISS, Milvus, Weaviate), errors (ChromaDB), or inconsistent internal state (SQLite-vec) when chunks with existing IDs are re-inserted.

This is a portability bug: the VectorIO abstraction should guarantee consistent behavior regardless of the backend. Since generate_chunk_id (src/ogx/providers/utils/vector_io/vector_utils.py:15) produces deterministic UUIDs from sha256(document_id:chunk_text) — clearly designed for idempotent re-ingestion — upsert is the intended and correct semantic.

All five misaligned backends natively support upsert or can trivially emulate it, so alignment is a matter of calling the right API method, not a design limitation.

Current provider behavior

Provider	Current semantics	Duplicate chunk_id behavior	Fix complexity
PGVector	UPSERT	Replaces existing chunk	Already correct
Qdrant	UPSERT	Replaces existing point	Already correct
Elasticsearch	UPSERT	Replaces existing document	Already correct
OCI 26AI	UPSERT	Replaces existing row	Already correct
Infinispan	UPSERT	Replaces existing entry	Already correct
ChromaDB	INSERT (collection.add())	Raises DuplicateIDError	One-line: swap add() to upsert()
Milvus	INSERT (client.insert())	Creates duplicate entries	One-line: swap insert() to upsert()
SQLite-vec	Partial upsert	Metadata upserted, vec0 vector table duplicated	Swap INSERT to INSERT OR REPLACE on vec0 table
FAISS	INSERT (append-only)	Creates duplicate entries	Wrap IndexFlatL2 with IndexIDMap2, use remove_ids() + add_with_ids()
Weaviate	INSERT (insert_many(), auto-gen UUIDs)	Creates duplicate objects	Pass deterministic UUIDs and use batch REST API with replace, or delete-then-insert

Milvus — easiest and most impactful fix

The Milvus provider (src/ogx/providers/remote/vector_io/milvus/milvus.py:150) calls client.insert() which silently creates duplicate entries on repeated chunk_id values. Milvus natively supports client.upsert() with identical signature and semantics — it checks the primary key and replaces existing entities. The fix is a single method swap:

# Current (line 150):
await asyncio.to_thread(self.client.insert, self.collection_name, data=data)

# Fixed:
await asyncio.to_thread(self.client.upsert, self.collection_name, data=data)

Note: Milvus upsert() requires the collection to be loaded and has higher memory usage than insert() for large-scale ingestion. This is an acceptable trade-off for correctness, and aligns with how PGVector/Qdrant already behave.

Relevant code paths

• Protocol definition: src/ogx_api/vector_io/api.py:58 — insert_chunks, no duplicate-handling contract specified
• Chunk ID generation: src/ogx/providers/utils/vector_io/vector_utils.py:15 — deterministic UUIDs from sha256(document_id:chunk_text), designed for idempotent re-ingestion
• Milvus (pure insert): src/ogx/providers/remote/vector_io/milvus/milvus.py:150 — client.insert() creates duplicates
• ChromaDB (error on dup): src/ogx/providers/remote/vector_io/chroma/chroma.py:86 — collection.add() raises on duplicate IDs
• SQLite-vec (partial upsert): src/ogx/providers/inline/vector_io/sqlite_vec/sqlite_vec.py:250 — vec0 table does plain INSERT, metadata table correctly upserts at line 236
• FAISS (pure insert): src/ogx/providers/inline/vector_io/faiss/faiss.py:182 — index.add() always appends, no ID-based dedup
• Weaviate (pure insert): src/ogx/providers/remote/vector_io/weaviate/weaviate.py:87 — insert_many() with auto-generated UUIDs, ignores chunk_id as object ID

Reproduction

from ogx_client import OgxClient

client = OgxClient(base_url="http://localhost:8321")

chunk = {
    "content": "test content",
    "chunk_id": "same-id-twice",
    "metadata": {},
    "embedding": [0.1] * 384,
    "embedding_model": "all-MiniLM-L6-v2",
    "embedding_dimension": 384,
}

# Insert the same chunk twice
client.vector_io.insert(vector_store_id="my-store", chunks=[chunk])
client.vector_io.insert(vector_store_id="my-store", chunks=[chunk])

# Result depends on backend:
# - PGVector/Qdrant/ES/OCI/Infinispan: 1 chunk (upserted) — correct
# - Milvus/FAISS/Weaviate: 2 chunks (silently duplicated) — bug
# - ChromaDB: DuplicateIDError on second call — bug
# - SQLite-vec: 1 metadata row, 2 vector entries — bug

Error logs

# ChromaDB raises on duplicate:
chromadb.errors.DuplicateIDError: ...

# Milvus/FAISS/Weaviate: no error, but silent data duplication affecting search quality
# SQLite-vec: no error, but inconsistent state between metadata and vector tables

Expected behavior

Expected behavior

insert_chunks should guarantee upsert semantics across all providers: inserting a chunk with an existing chunk_i d replaces the previous entry. Specifically:

1. Document the contract. The insert_chunks protocol docstring should explicitly state that duplicate chunk_id
   values result in replacement, not duplication or error.

2. Align all providers to upsert. Recommended fix per provider:

   • Milvus: swap client.insert() to client.upsert() (one-line fix)
   • ChromaDB: swap collection.add() to collection.upsert() (one-line fix)
   • SQLite-vec: change INSERT INTO [vec_table] to INSERT OR REPLACE INTO [vec_table] for the vec0 virtual ta
     ble
   • FAISS: wrap IndexFlatL2 with IndexIDMap2 to enable ID-based operations, implement upsert as remove_ids()

• add_with_ids()
  • Weaviate: use chunk_id as the Weaviate object UUID (deterministic) instead of auto-generating, and use the b
    atch REST API with replace semantics or delete-then-insert

3. Prioritize Milvus and ChromaDB. These are one-line fixes with no trade-offs that affect common production deploy
   ments.