Proposal: semantics for optimized payload buffering

[usize]

Problem

The current body processing modes (Stream, Buffer, StreamBuffer) don't support a common payload processing pattern: making decisions on partial data while buffers accumulate, and exiting early once a decision is reached.

Today a filter must choose between:

• Buffer: wait for the entire body, then process once. Correct but slow: wastes time and memory when the decision could have been made from the first few chunks.
• Stream: see each chunk independently with no accumulation. Fast but no context across chunks.
• StreamBuffer: see each chunk and can Release the buffer, but doesn't expose the running accumulation to the filter and can't signal "skip me for remaining chunks."

Proposed semantics

Two capabilities that build on StreamBuffer:

1. Early decision on partial data

A filter receives each chunk as it arrives and can short-circuit with Reject or Continue before the full body is buffered. Once the filter signals a final decision, it is skipped for all remaining chunks.

We currently have Release for short circuiting StreamBuffer, but when a filter returns Release currently, the accumulated buffer is forwarded upstream and the buffer is cleared. Here, we propose holding onto the accumulated buffer for use by other policies in the chain (freeing only at the end).

Example: A guardrails filter scans each incoming chunk for policy violations. If it detects a violation in the first 2 KiB of a 100 KiB body, it rejects immediately without buffering the rest.

Open Question

• Should we maintain semantics for forcing a buffer to flush from within a single filter?
• Do we also need a way of signaling that a filter only intends to operate on the first N bytes of a stream and doesn't need the whole buffer?

2. Progressive accumulation with early exit

The filter sees both the current chunk AND the accumulated buffer so far, allowing decisions based on the full context seen to date. When the filter has enough signal, it marks itself "done" and is skipped for subsequent chunks, but other filters in the pipeline that need the full body continue accumulating.

Example: An AI inference guardrails filter (like LlamaStack) evaluates each chunk against content policy using the running context. Once it can make a confident allow/deny decision, it exits early while a downstream body-rewriting filter continues buffering to completion.

An example of this pattern in practice is here: https://github.com/ogx-ai/ogx/blob/main/src/ogx/providers/inline/responses/builtin/responses/streaming.py#L1239

Though it only applies to guardrails. This proposal is about generalizing this optimization across all payload processors which opt in.

Design considerations

• Filters should be able to express "I'm done, skip me" without affecting other filters in the pipeline that still need chunks.
• The accumulated view should be available without requiring each filter to maintain its own buffer, the pipeline manages one shared accumulation.
• This is an optimization over Buffer mode, not a replacement. Filters that truly need the complete body before acting (e.g., JSON schema validation) should continue using Buffer.
• Early rejection should stop both accumulation and upstream forwarding.

Motivation

This is a key differentiator for Praxis over Envoy's ext_proc model, where processors typically slurp the entire body regardless of what they need. Optimized payload processing reduces latency and memory for the common case where a decision can be made from partial data.

Related

• StreamBuffer pre-read body forwarding limited to 64 KiB #75 — StreamBuffer pre-read body forwarding (current workaround)
• PR feat: add per-filter failure_mode (open/closed) #62 — failure_mode (discussion on filter processing semantics)

[shaneutt]

The proposal frames the problem around choosing between Buffer and Stream, but we don't have a Buffer mode any more. How do you feel the recent promotion of StreamBuffer affects this initiative?

Also it sounds like you're stating that we need the ability to do early decisions on partial data, but we already have ways to express this pattern with JSON_RPC and the other JSON filters, using streaming JSON, can you help me to understand better what's missing there, what you're looking for?

[usize]

This is stale. You're right. BodyDone in particular solves the core problem here. Closing this!