How to pause document ingestion after processing a set number of Tokens using Kernel Memory ImportDocumentAsync?

[mahadzk]

Hello community,

I'm working on a project where we process large documents using the Microsoft.KernelMemory library. What I'm trying to figure out is if there is a way to pause file processing once a certain number of tokens has been processed.

Current Approach:
I’m using the KernelMemoryBuilder with WithCustomTextPartitioningOptions to handle the chunking automatically:

.WithCustomTextPartitioningOptions(new TextPartitioningOptions { MaxTokensPerParagraph = _chunkingOptions.MaxTokensPerParagraph, MaxTokensPerLine = _chunkingOptions.MaxTokensPerLine, OverlappingTokens = _chunkingOptions.OverlappingTokens })

I call ImportDocumentAsync() to ingest the document, assuming it would process it in chunks internally:

await memory.ImportDocumentAsync(stream, fileName, indexName, documentId, tags);

What I Need:
I need to ensure that the ingestion process pauses after 100,000 tokens have been processed for 60 seconds before continuing. This is due to the 250,000 tokens per minute (TPM) restriction for OpenAI.

Questions:
Does ImportDocumentAsync() allow for mid-process control (e.g., tracking real-time chunks and pausing)?
If not, is there a recommended way to achieve this without manually splitting the document into smaller pieces beforehand?
Are there any built-in methods or extension points that provide visibility into the individual chunks created by WithCustomTextPartitioningOptions?
Context:
I’m using version 0.62.240605.1 of Microsoft.KernelMemory.Core.
The use case is to avoid OpenAI’s API rate limits, which enforces a 250,000-token per minute restriction.
Any guidance or recommendations would be highly appreciated. Thank you for your help!

[dluc]

KM automatically handles TPM quota with Azure OpenAI and OpenAI, pausing requests as instructed by HTTP 429 response headers.

When the KM code reaches the quota, OpenAI service returns a response with HTTP Status code 429, and a header saying something like "max quota reached, wait N seconds". As long as this header is returned, KM will honor it.

Code here: https://github.com/microsoft/kernel-memory/blob/main/extensions/OpenAI/OpenAI/Internals/ClientSequentialRetryPolicy.cs

[mahadzk]

Thanks @dluc! Will try this approach