docs/how_to/indexing/

[giscus[bot]]

docs/how_to/indexing/

Here, we will look at a basic indexing workflow using the LangChain indexing API.

https://python.langchain.com/docs/how_to/indexing/

[vshanbha]

Thank you for this. I tried getting this to work with a local FAISS store.
I am able to use this API to upload documents and run similarity searches against the store.

However this document does not cover any definitive way to delete a single Document from the Vector Store. I can't find any links in the langchain pages on deleting a document that has been added to the Vector store using the indexing method specified here.

The main challenge is that when we retrieve documents from the Vector store they do not have any Ids.
I already looked at docs for the [Index] ( https://api.python.langchain.com/en/latest/core/indexing/langchain_core.indexing.api.index.html
and [VectorStore] (https://api.python.langchain.com/en/latest/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html#langchain_core.vectorstores.base.VectorStore)

[DeezNutzHimself]

Watch https://www.youtube.com/watch?app=desktop&v=6XG4mfUVTe4

[vshanbha]

Thank you @DeezNutz6942O . This is the same thing as the documentation page now in a video and your explanation. I appreciate you taking the time to create the video. However it still does not answer how to remove a single document from the vector store.
I did figure out a hacky solution and have it available in a project https://github.com/vshanbha/RAGPoC.
I do not like my solution because it involves updating the index in "Full" mode by only sending the documents that I want to keep.
For the relevant stuff look at (https://github.com/vshanbha/RAGPoC/blob/main/persistence/vector_db.py) there is a method called delete_document.
This solution is acceptable for a proof of concept but it is not going to scale if I have a Vector DB with 100s of documents.

I am probably missing something super obvious. Happy to learn.

[alex-feel]

How does it work with ParentDocumentRetriever? Can it help in managing parent documents?

[Revigatech]

I want to report issue with index when using RedisVectorStore

• the problem lies with method add_documentsin in RedisVectorStore. when we pass ids to add_documents, it doesn't store passed ids as keys instead it generate its own keys, so record_manger store keys that doesn't exist redis

• the problem lies in the following lines in index function from indexing api
  if docs_to_index:
  if isinstance(destination, VectorStore):
  destination.add_documents(
  docs_to_index,
  ids=uids,
  batch_size=batch_size,
  **(upsert_kwargs or {}),
  )
  print(uids)
  elif isinstance(destination, DocumentIndex):
  destination.upsert(
  docs_to_index,
  **(upsert_kwargs or {}),
  )

        num_added += len(docs_to_index) - len(seen_docs)
        num_updated += len(seen_docs)
  
    # And only then update the record store.
    # Update ALL records, even if they already exist since we want to refresh
    # their timestamp.
    record_manager.update(
        [doc.uid for doc in hashed_docs],
        group_ids=source_ids,
        time_at_least=index_start_dt,
    )
  
            destination.add_documents(
                docs_to_index,
                ids=uids,
                batch_size=batch_size,
                **(upsert_kwargs or {}),
            )
  

[Revigatech]

    if docs_to_index:
        if isinstance(destination, VectorStore):
            destination.add_documents(
                docs_to_index,
                ids=uids,
                batch_size=batch_size,
                **(upsert_kwargs or {}),
            )
            print(uids)
        elif isinstance(destination, DocumentIndex):
            destination.upsert(
                docs_to_index,
                **(upsert_kwargs or {}),
            )

        num_added += len(docs_to_index) - len(seen_docs)
        num_updated += len(seen_docs)

    # And only then update the record store.
    # Update ALL records, even if they already exist since we want to refresh
    # their timestamp.
    record_manager.update(
        [doc.uid for doc in hashed_docs],
        group_ids=source_ids,
        time_at_least=index_start_dt,
    )

[alberto-agudo]

This solution is interesting but what if your original document directory has thousands of documents and you only want to process the ones that have changed in source and not after processing?

The ideal thing here would be to first select the documents you know have changed and then load them and use the incremental index. However, this wouldn't allow you to delete documents that have been eliminated from source.

It would be great to have some way of indexing both the vector store and the original document directory (my example is in S3) so that the whole document lifecycle is managed.