Dynamic Top-k Retrieval Chunks

[Malak-Alaabiad]

Suggest me a specific node or flow to reduce the number of tokens going into the LLM model, considering that my data is stored in a Qdrant collection, and I'm using a custom retriever node to pull only the necessary metadata. This custom retriever node is connected to the Conversational Retriever QA Chain, which then passes the data directly to the LLM.

Now, I want to implement a Dynamic Top-k Retrieval Chunks or a similar flow to achieve the same goal—reducing the tokens sent to the model, which would help minimize the associated costs.

[Ganesh-403]

@Malak-Alaabiad
If your main objective is reducing the number of tokens sent to the LLM, I'd focus on improving the retrieval pipeline rather than trying to make the Conversational Retrieval QA Chain dynamically adjust top_k.

From what I can tell, Flowise doesn't currently provide a dedicated "Dynamic Top-k" node that automatically changes the number of retrieved chunks per query. I couldn't find any documentation describing such a feature. :contentReference[oaicite:0]{index=0}

Instead, I'd recommend one (or a combination) of these approaches:

1. Use a Similarity Score Threshold Retriever (Recommended)

Instead of always returning a fixed number of chunks (e.g. Top-10), use the Similarity Score Threshold Retriever.

This returns only documents above a relevance threshold:

Qdrant
    ↓
Similarity Score Threshold Retriever
    ↓
Conversational Retrieval QA Chain

For simple questions you may end up with only 2–3 chunks, while more complex questions can still retrieve more. This is often the closest behavior to "dynamic top-k". Flowise includes this retriever out of the box. :contentReference[oaicite:1]{index=1}

---

2. Add a Reranker

If you currently retrieve a larger candidate set (for example Top-20), add a reranker such as:

• Cohere Rerank Retriever
• Voyage AI Rerank Retriever

A common pattern is:

Qdrant (Top 20)
        ↓
Reranker
        ↓
Top 3–5 most relevant chunks
        ↓
Conversational Retrieval QA Chain

This usually gives better relevance while reducing the amount of context passed to the LLM. Flowise provides built-in reranker retrievers for this purpose. :contentReference[oaicite:2]{index=2}

---

3. Reduce chunk size and tune Top-K

Token usage is also heavily influenced by your indexing strategy.

The Flowise documentation recommends experimenting with:

• chunk size
• chunk overlap
• Top-K

to balance retrieval quality against token cost. Smaller, more focused chunks often allow you to lower Top-K without hurting answer quality. :contentReference[oaicite:3]{index=3}

---

4. Keep your Custom Retriever lean

Since you're already using a Custom Retriever, make sure you're only sending the information the LLM actually needs.

For example, instead of:

Document
Metadata
Source
Author
Timestamp
Category
...

return only something like:

{{context}}

or include only the metadata that contributes to answering the question (e.g. source for citations). The Custom Retriever is specifically intended to control the context formatting sent to the LLM. :contentReference[oaicite:4]{index=4}

---

Suggested pipeline

Qdrant
    ↓
Similarity Score Threshold Retriever
    ↓
(Optional) Cohere/Voyage Reranker
    ↓
Custom Retriever
    ↓
Conversational Retrieval QA Chain

This typically achieves lower token usage than a fixed Top-K approach while maintaining answer quality.

One question: is your goal to optimize for cost, latency, or both? That will help determine whether a threshold retriever, reranker, or another retrieval strategy is the best fit.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.