Skip to content

Feat/add streaming chunks based guardrails - #6

Open
vrushabh-akto wants to merge 18 commits into
masterfrom
feat/add-streaming-chunks-based-guardrails
Open

Feat/add streaming chunks based guardrails#6
vrushabh-akto wants to merge 18 commits into
masterfrom
feat/add-streaming-chunks-based-guardrails

Conversation

@vrushabh-akto

@vrushabh-akto vrushabh-akto commented May 28, 2026

Copy link
Copy Markdown

Summary

  • Async prefetching pipeline for streaming guardrails — while batch N's Akto API call is in-flight, batch N+1 accumulates in parallel. Result collection is near-instant when the API responds within the batch fill window (pipeline ok), eliminating per-batch stall latency for the agent.
  • Separate thread pools — hook-level checks (request/response) and stream batch checks run on independent pools so streaming load never starves prompt-level guardrails.
  • Fixed stream response payload formatresponsePayload is now JSON-encoded ({"content": [{"type": "text", "text": "..."}]}) matching the structure of requestPayload, fixing guardrail evaluation on stream batches.
  • 403 block responses — blocked requests/responses now return 403 with {"error": "<reason>"} instead of a graceful SSE message. Mid-stream blocks send data: {"error": "..."}\n\ndata: [DONE].
  • Configurable thresholdAKTO_TEXT_THRESHOLD env var (default 600 chars) controls batch size; tunable after measuring API latency from logs.
  • Structured logging — latency on every Akto API call, full payload and response logging behind AKTO_LOG_PAYLOADS flag, clear REQUEST / STREAM / RESPONSE prefixes, pipeline status (pipeline ok vs waited Xms) per batch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants