[Question]: Adding new summary to existing DocumentSummaryIndex

[adas-cpp]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

[Question]: Hi, I am using the DocumentSummaryIndex functionality. Once I create a DocumentSummaryIndex, how can I add more documents to it? It says that I can add nodes using the _insert, or _add_nodes_to_index function. I tried using that but it doesn;t seem to work as it doesn't insert the text of each node. It just inserts the summary and the list of questions as 1 node. Can you please give me an example of how to 1. get a new summary for a document without creating a new DocumentSummaryIndex and 2. How to add that summary and all the corresponding nodes to an existing Index. Thank you.

[dosubot[bot]]

🤖

Hi there,

To add more documents to an existing DocumentSummaryIndex, you can use the add_summary_and_nodes method of the IndexDocumentSummary class. This method allows you to add a summary node and its associated nodes to the index. Here's an example of how you can use it:

# Assuming `index_doc_summary` is an instance of IndexDocumentSummary
# `summary_node` is a new BaseNode instance representing the summary
# and `new_nodes` is a list of new BaseNode instances you want to add

index_doc_summary.add_summary_and_nodes(summary_node, new_nodes)

Please note that the ref_doc_id of the summary node cannot be None when building a document summary index.

The add_summary_and_nodes method works as follows:

1. It retrieves the document ID (doc_id) from the summary_node and checks if the ref_doc_id (reference document ID) of the summary_node is not None. If it is None, it raises a ValueError because a ref_doc_id is required when building a document summary index.

2. It maps the ref_doc_id of the summary_node to the summary_id in the doc_id_to_summary_id dictionary.

3. For each node in the nodes list, it retrieves the node's ID (node_id) and adds it to the list of node IDs associated with the summary_id in the summary_id_to_node_ids dictionary. If the summary_id is not already a key in the dictionary, it first initializes it with an empty list.

4. It also maps the node_id to the summary_id in the node_id_to_summary_id dictionary.

5. Finally, it returns the summary_id.

You can find more details in the source code of the add_summary_and_nodes method in the IndexDocumentSummary class here.

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

Sources

• [Question]: Is it possible to append new nodes to an existing index without overriding the whole index?
• Adding documents to a Composible graph after creating the graph
• llama_index/indices/document_summary/base.py
• llama_index/data_structs/document_summary.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[adas-cpp]

Can you tell me the difference between the DocumentSummaryIndex and IndexDocumentSummary class please? What I get is a DocumentSummaryIndex class, how can I add nodes to it using the IndexDocumentSummary class?

[adas-cpp]

Lets say that I have an index named "doc_summary_index" which I loaded from a file and is of the type DocumentSummaryIndex class. How can I add new summaries to it. The solution you provided earlier uses the IndexDocumentSummary class. How are they related. Can you please write the code I would need to add a new summary and all the nodes to a DocumentSummaryIndex class. Also, how do I generate new summaries from Documents without creating a new index.

storage_context = StorageContext.from_defaults(persist_dir="doc_summary_index")
doc_summary_index = load_index_from_storage(storage_context)

[dosubot[bot]]

🤖

Hello,

The DocumentSummaryIndex and IndexDocumentSummary classes are closely related. The DocumentSummaryIndex class is a data structure where LlamaIndex stores the summary per document, and maps the summary to the underlying Nodes. This summary can be used for retrieval. On the other hand, the IndexDocumentSummary class is used as the index structure for the DocumentSummaryIndex class.

To add a new summary and all the nodes to a DocumentSummaryIndex class, you can use the _add_nodes_to_index method. Here is the code:

def _add_nodes_to_index(
    self,
    index_struct: IndexDocumentSummary,
    nodes: Sequence[BaseNode],
    show_progress: bool = False,
) -> None:
    """Add nodes to index."""
    doc_id_to_nodes = defaultdict(list)
    for node in nodes:
        if node.ref_doc_id is None:
            raise ValueError(
                "ref_doc_id of node cannot be None when building a document "
                "summary index"
            )
        doc_id_to_nodes[node.ref_doc_id].append(node)

    summary_node_dict = {}
    items = doc_id_to_nodes.items()
    iterable_with_progress = get_tqdm_iterable(
        items, show_progress, "Summarizing documents"
    )
    for doc_id, nodes in iterable_with_progress:
        nodes_with_scores = [NodeWithScore(node=n) for n in nodes]
        # get the summary for each doc_id
        summary_response = self._response_synthesizer.synthesize(
            query=self._summary_query,
            nodes=nodes_with_scores,
        )
        summary_response = cast(Response, summary_response)
        summary_node_dict[doc_id] = TextNode(
            text=summary_response.response,
            relationships={
                NodeRelationship.SOURCE: RelatedNodeInfo(node_id=doc_id)
            },
        )
        self.docstore.add_documents([summary_node_dict[doc_id]])
        logger.info(
            f"> Generated summary for doc {doc_id}: " f"{summary_response.response}"
        )

    for doc_id, nodes in doc_id_to_nodes.items():
        index_struct.add_summary_and_nodes(summary_node_dict[doc_id], nodes)

To generate new summaries from Documents without creating a new index using the IndexDocumentSummary class, you can use the _response_synthesizer.synthesize method. This method takes a query and a list of nodes as arguments, and returns a summary response. Here is the code:

summary_response = self._response_synthesizer.synthesize(
    query=self._summary_query,
    nodes=nodes_with_scores,
)
summary_response = cast(Response, summary_response)

I hope this helps! Let me know if you have any other questions.

Sources

• llama_index/indices/document_summary/base.py
• llama_index/data_structs/document_summary.py
• tests/indices/document_summary/test_index.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.