test: expand sequential response coverage across providers and edge cases#52
Merged
test: expand sequential response coverage across providers and edge cases#52
Conversation
commit: |
…mbined matching, malformed)
…ues, cross-provider truncation)
jpr5
force-pushed
the
test/sequential-robustness
branch
from
d77ddce to
6097baa
Compare
Add ChaosConfig with dropRate, malformedRate, disconnectRate. Chaos can be configured at server level, fixture level, or per-request via headers (X-LLMock-Chaos-*). Fixture overrides server, headers override both. Actions are journaled with chaosAction field. Also fix setChaos/clearChaos to propagate to running server by exposing defaults on ServerInstance.
Anthropic (/v1/messages, Bedrock) now returns { type: 'error', error: { type, message } }.
Gemini now returns { error: { code, message, status } }.
OpenAI Chat unchanged (already correct).
Previously all providers returned identical OpenAI-style error JSON.
- Cross-provider error format conformance and streaming error tests - Azure streaming through deployment path - Error fixture with sequenceIndex (success/error/success) - onToolResult and onToolCall live server verification - setChaos/clearChaos integration tests - Fixture loader streamingProfile/chaos passthrough and validation boundaries - Groq streaming through proxy path - Error status codes (401, 503, default 500)
…i-tool coverage - Concurrent message serialization: verify no event interleaving on rapid sends - WS frame fragmentation: continuation frame reassembly - Multiple tool calls in single response (Realtime + Responses WS)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Comprehensive test coverage expansion and two implementation improvements across the entire llmock codebase.
Implementation Changes
Chaos injection feature — New fault tolerance testing capability:
ChaosConfigwithdropRate,malformedRate,disconnectRate(all 0-1 probability)X-LLMock-Chaos-*)--chaos-drop,--chaos-malformed,--chaos-disconnectchaosActionfieldsetChaos()/clearChaos()on LLMock class (propagates to running server)Provider-specific error formats — Previously all providers returned identical OpenAI-style error JSON. Now:
{ type: "error", error: { type, message } }{ error: { code, message, status } }Test Coverage (848 → 906 tests, +58)
Sequential Responses (+9): error responses, tool calls, gap indices, Anthropic/Gemini sequences, predicate+sequenceIndex, model matching, resetMatchCounts, concurrent requests
Embeddings (+14): base64 encoding_format, empty input, sequential embeddings, Unicode, error status codes (401, 503)
Structured Output (+9): streaming json_object, json_schema matching, combined responseFormat+model+userMessage routing, malformed response_format, sequenceIndex+error interleaving
Streaming Physics (+6): streamingProfile+truncateAfterChunks combined, streamingProfile+disconnectAfterMs, truncateAfterChunks:1 boundary, large chunkSize, empty content streaming, Anthropic truncation
Chaos (+12): server-level drop/malformed on Anthropic/Gemini/Bedrock, fixture-level chaos scoping, journal recording, setChaos/clearChaos integration
Error Format (+6): Anthropic/Gemini/Bedrock format conformance, streaming error returns JSON not SSE, cross-provider format verification
Provider Coverage (+4): Azure streaming, onToolResult live verification, onToolCall response verification, Groq streaming through proxy
Fixtures (+7): streamingProfile/chaos JSON passthrough, validation boundary conditions (status 99/100/599), error status default 500
WebSocket (+5): concurrent message serialization (no interleaving), frame fragmentation (continuation frames), multiple tool calls (Realtime + Responses WS)
Bug Fixes Found During Testing
setChaos()/clearChaos()silently failed on running servers (captured snapshot at startup, not live reference)Test plan