feat(guardrails): add commercial-API guardrails (Watsonx Guardian, Patronus, Qualifire, Pioneer) - #184
Merged
Merged
Conversation
The REST-based API guardrails (Alinia, Lakera, and the new Patronus/Qualifire/Pioneer) import requests directly, but it was not guaranteed transitively by the core deps (any-llm-sdk/pydantic) -- declare it explicitly. Add the optional 'watsonx' extra (ibm-watsonx-ai) for the WatsonxGuardian guardrail and include it in the 'all' aggregate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hosted, pay-per-use counterpart to the local GraniteGuardian: wraps IBM watsonx.ai's Text Detection / Guardian moderation API (ibm-watsonx-ai SDK) against the Granite Guardian risk catalogue. Maps detections onto GuardrailOutput categories + spans and fails closed on unparsable responses. Unit tests exercise post-processing in isolation (no SDK install needed); the e2e test is gated on watsonx credentials. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hosted counterpart to the local Glider/Lynx models: wraps Patronus' /v1/evaluate managed-evaluator API. Combines per-evaluator pass/score_raw into GuardrailOutput (canonical risk = 1 - score_raw) per a configurable success strategy, and fails closed. Request/response shape verified against NeMo Guardrails' Patronus integration source. Adds unit + key-gated e2e tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hosted counterpart to the local Sentinel prompt-injection model: wraps Qualifire's multi-check evaluation API (prompt-injection, PII, hallucination, grounding, content-moderation). The verdict is driven by the per-check 'flagged' booleans (the top-level 'status' is a lifecycle value, e.g. 'completed'); Qualifire's 0-100 higher-is-safer scores are converted to canonical risk (1 - score/100). Shape verified against the MIT-licensed Qualifire SDK. Adds unit + key-gated e2e tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hosted counterpart to the open-weight GLiGuard model: wraps Fastino's Pioneer inference API running fastino/gliguard-llm-guardrails-300m for schema-conditioned safety classification. Pioneer's /inference response body is not publicly documented, so it is parsed defensively (envelope unwrapping, argmax over score dicts, fail-closed on unknown shapes) with the full payload preserved in raw. Adds unit + key-gated e2e tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire WatsonxGuardian, Patronus, Qualifire, and Pioneer into the GuardrailName enum, the API-docs generator (GUARDRAILS list + install-extra override), the GitBook nav (SUMMARY.md), and the factory test's load skip-list (they are API/SDK-based, not provider.load_model-based). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds four new hosted/commercial guardrail integrations to any-guardrail, matching the existing “API guardrail” pattern (direct Guardrail subclasses, env-var auth, GuardrailOutput mapping) and wiring them into docs + unit/e2e coverage.
Changes:
- Added new guardrails:
WatsonxGuardian(watsonx SDK),Patronus,Qualifire,Pioneer(REST viarequests). - Expanded documentation generation/navigation to include the new guardrails.
- Added unit tests (mocked) and integration/e2e tests (skipped when API keys are missing); added
requeststo core deps andwatsonxextra.
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_unit_watsonx_guardian.py | Unit tests for WatsonxGuardian post-processing + credential validation behavior. |
| tests/unit/test_unit_qualifire.py | Unit tests for Qualifire request/response mapping and failure modes. |
| tests/unit/test_unit_pioneer.py | Unit tests for Pioneer request/response parsing across multiple response shapes. |
| tests/unit/test_unit_patronus.py | Unit tests for Patronus request/response mapping + success strategies. |
| tests/unit/test_api.py | Ensures new guardrails resolve via factory and are excluded from provider load expectations. |
| tests/integration/test_watsonx_guardian.py | Live watsonx Guardian e2e test (skipped without creds). |
| tests/integration/test_qualifire.py | Live Qualifire e2e test (skipped without key). |
| tests/integration/test_pioneer.py | Live Pioneer e2e test (skipped without key). |
| tests/integration/test_patronus.py | Live Patronus e2e test (skipped without key). |
| src/any_guardrail/guardrails/watsonx_guardian/watsonx_guardian.py | New watsonx Guardian SDK-based guardrail implementation. |
| src/any_guardrail/guardrails/watsonx_guardian/init.py | Package export for WatsonxGuardian. |
| src/any_guardrail/guardrails/qualifire/qualifire.py | New Qualifire REST guardrail implementation. |
| src/any_guardrail/guardrails/qualifire/init.py | Package export for Qualifire. |
| src/any_guardrail/guardrails/pioneer/pioneer.py | New Pioneer REST guardrail + defensive envelope parsing. |
| src/any_guardrail/guardrails/pioneer/init.py | Package export for Pioneer. |
| src/any_guardrail/guardrails/patronus/patronus.py | New Patronus REST guardrail implementation. |
| src/any_guardrail/guardrails/patronus/init.py | Package export for Patronus. |
| src/any_guardrail/base.py | Added new GuardrailName enum entries. |
| scripts/generate_api_docs.py | Added new guardrails to generated API docs/index; added watsonx extra override. |
| pyproject.toml | Added requests to core deps; added watsonx extra and included it in all. |
| docs/SUMMARY.md | Added new guardrails to GitBook navigation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Per Copilot review: a non-dict item in the response 'results' list would hit AttributeError on .get() and bypass fail-closed handling. Treat any non-dict result entry as a failed/triggered evaluator so all_pass cannot fail open on a malformed response. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Copilot review: a null / non-list / empty 'evaluationResults', or entries that carry no parseable check, previously fell through to valid=True (fail open). Require a non-empty list and at least one parsed check, else fail closed (extra={'parse_failure': True}).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Copilot review: a task prediction present but yielding no labels (e.g. {} or an unexpected structure) made the task non-flagged and could return valid=True. For the undocumented Pioneer response schema, fail closed on such values; an explicitly empty list (multi-label task with no categories detected) is still treated as safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per Copilot review: the skip condition also accepts WATSONX_SPACE_ID, so the reason string now notes it alongside WATSONX_PROJECT_ID. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts from the #179 open-weight guardrail merge (PR #183): union the GuardrailName enum (13 #179 entries + the 4 commercial-API entries), the generate_api_docs GUARDRAILS list, the SUMMARY.md nav (alphabetized), and the 'all' aggregate extra (lettucedetect/gliner + watsonx). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
I'd rather use httpx than requests, but otherwise this is fine for me as long as tests pass. |
javiermtorres
approved these changes
Jun 23, 2026
Contributor
Author
|
Thanks @javiermtorres! I'm waiting on Mario and Carlos to resolve this issue: https://github.com/mozilla-ai/infrastructure/issues/311 Then I can run integration tests against these commercial guardrails to make sure all the piping works. |
Drop the Qualifire and Pioneer commercial-API guardrails (no API keys / not pursued), keeping Patronus and WatsonxGuardian. Removes the two guardrail packages, their unit + integration tests, and their entries in the GuardrailName enum, the API-docs GUARDRAILS list, the SUMMARY.md nav, and the test_api load skip-list. requests stays a core dep (Alinia/Lakera/Patronus still use it); comment updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose PATRONUS_API_KEY and WATSONX_APIKEY/WATSONX_URL/WATSONX_PROJECT_ID from repo secrets so the Patronus and watsonx Guardian e2e tests run when the secrets are configured (they skip otherwise). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 tasks
dni138
added a commit
that referenced
this pull request
Jul 6, 2026
* feat(guardrails): add Qwen3Guard (Qwen3Guard-Gen) safety moderation Generative safety classifier with a three-level severity verdict (Safe / Controversial / Unsafe). The chat template embeds the classifier instruction, so prompt moderation is a single user message and response moderation adds the assistant turn. Severity maps onto the canonical risk score (0.0 / 0.5 / 1.0) and is surfaced verbatim in extra["severity"]; strict=True (default) passes only Safe verdicts. In response mode the model's Refusal verdict is surfaced as a "refusal" category. Fails closed on unparseable output. Part of #93. The Qwen3Guard-Stream variants land separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(guardrails): add Qwen3GuardStream token-level streaming classifier Qwen3Guard-Stream loads its classification heads as remote code (AutoModel + trust_remote_code) and judges the user prompt as a whole plus every assistant response token individually. validate() is a non-streaming facade over the streaming API: it aggregates the worst severity across all judged positions onto the same strict/score/extra contract as Qwen3Guard, and returns runs of flagged response tokens as character spans into output_text (offset-mapping based; degrades to no spans on tokenizers without offset support). HuggingFace-only; a user-supplied provider must set trust_remote_code=True. Part of #93. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(guardrails): unit and integration coverage for Qwen3Guard family Gen: severity/score/extra mapping, strict vs lenient Controversial, category parsing scoped to the Categories line (no substring matches), refusal surfacing rules per mode, think-block stripping, fail-closed. Stream: worst-severity aggregation, category dedup, span merge/split semantics, offsetless-token handling, fail-closed on missing risk levels. Integration: both 0.6B defaults added to the non-heavy parametrized HuggingFace suite. Part of #93. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: register Qwen3Guard family and regenerate API reference Adds both guardrails to the API-docs generator, GitBook navigation (SUMMARY.md), and the CLAUDE.md guardrail-shape lists, then runs scripts/generate_api_docs.py. The regeneration also emits the patronus.md / watsonx-guardian.md pages and their index rows that PR #184 forgot to commit (the index table is derived from the enum, so the catch-up cannot be split out). Part of #93. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: add missing PolyGuard entry to SUMMARY.md PolyGuard's generated API page has existed since it landed, but the GitBook navigation never linked it, so it was absent from production docs navigation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(guardrails): gate Qwen3GuardStream on transformers<5 The Qwen3Guard-Stream model repos ship remote modeling code written against transformers 4.x; transformers 5 removed APIs it relies on (the implicit pad_token_id config default, ROPE_INIT_FUNCTIONS ["default"], the old rotary-embedding weight-init interface), so the checkpoint cannot load there. Construction now raises an actionable ImportError on transformers >= 5, the integration test skips on such environments, and test_model_load asserts the gate. Verified end to end on transformers 4.57.6: prompt and response moderation on the real 0.6B weights, including span offsets into output_text. Part of #93. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the four accepted commercial pay-per-use guardrail APIs from the #181 sweep, each the hosted counterpart to an open-weight model
any-guardrailalready integrates. All follow the existing API-guardrail shape (Alinia/LakeraGuard): inheritGuardraildirectly, env-var auth, map ontoGuardrailOutput, fail closed on unparsable responses.WatsonxGuardianGraniteGuardianGuardianText Detection API (ibm-watsonx-aiSDK)PatronusGlider+ LynxQualifireSentinelPioneerfastino/gliguard-LLMGuardrails-300M)Closes #181 (the four accepted candidates; the near-misses remain tracked there for re-evaluation).
Verifying correctness without API keys
We don't have keys yet, so every request/response shape was verified against authoritative source, not just docs prose:
ibm-watsonx-aiSDKGuardian.detect()({"detections": [...]}).POST /v1/evaluate,X-API-KEY,results[].evaluation_result.{pass,score_raw,explanation}).types.py/client.py+ README):POST /api/v1/evaluation/evaluate,X-Qualifire-API-Key; verdict is the per-checkflaggedboolean (statusis a lifecycle value like"completed"), and the 0-100 higher-is-safer scores are converted to canonical risk (1 - score/100).docs.pioneer.ai(POST /inference,X-API-Key, modelfastino/gliguard-llm-guardrails-300m) + the GLiGuard model card (gliner2 schema/labels).The implementation was then put through an adversarial multi-agent review (per-guardrail review → independent verification of each finding), which surfaced 12 real bugs — all fixed in this PR, the two highest-impact ones (Qualifire's
valid-from-statusand its score scale/inversion) re-confirmed by hand against the Qualifire README.Notable / reviewer attention
requestsadded to core dependencies. It isn't a transitive dep ofany-llm-sdk/pydantic, yet the existingAlinia/LakeraGuard(and the three new REST guardrails) import it at module top level — a core-only install wouldImportError. This is the minimal correct fix./inferenceresponse body is not publicly documented (request/auth/model-id/pricing are). It's parsed defensively with fullrawpassthrough and the uncertainty is called out in the docstring — please confirm against a livePIONEER_API_KEY. Happy to split Pioneer into a follow-up if you'd prefer to ship only the three fully-live-verifiable guardrails first.Tests
e2eintegration tests for all four, skipped without keys (gated onPATRONUS_API_KEY/QUALIFIRE_API_KEY/ IBM IAM creds /PIONEER_API_KEY).pre-commit(ruff, ruff-format, mypy strict on 117 files, codespell) green;generate_api_docs.pyproduces all four pages.New CI keys needed to exercise the e2e tests:
PATRONUS_API_KEY,QUALIFIRE_API_KEY,WATSONX_APIKEY/WATSONX_URL/WATSONX_PROJECT_ID,PIONEER_API_KEY(all have free/low-cost tiers).🤖 Generated with Claude Code