Skip to content

Clarify live API conformance gap in Future Direction #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jpr5 merged 2 commits into from
Mar 11, 2026
Merged

Clarify live API conformance gap in Future Direction #23

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bcfc7c7
docs: clarify live API conformance gap in Future Direction
jpr5 Mar 11, 2026
b169bad
Merge branch 'main' into docs/future-direction-fix
jpr5 Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,7 +631,7 @@ Areas where llmock could grow, and explicit non-goals for the current scope.

### Testing

- **E2E SDK tests**: The test suite uses raw HTTP and WebSocket frames, not real OpenAI/Anthropic/Gemini client SDKs.
- **Live API conformance**: The `api-conformance` tests validate response format structure but do not run against real LLM APIs. A subset of tests that hit actual OpenAI/Anthropic/Gemini endpoints (gated behind API keys) would catch format drift as providers evolve their APIs.
- **Token counts**: Usage fields are always zero across all providers.
- **Vision/image content**: Image content parts are not handled by any provider.

Expand Down
Loading