Skip to content
Open
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
650369c
added toxicity detection validators
rkritika1508 Apr 1, 2026
949647d
fixed import error
rkritika1508 Apr 1, 2026
da50537
removed redundant validators
rkritika1508 Apr 2, 2026
9ab64c7
Added NSFW text validator
rkritika1508 Apr 2, 2026
b64d0e9
fixed test
rkritika1508 Apr 2, 2026
57d97b2
Merge branch 'feat/toxicity-hub-validators' into feat/toxicity-huggin…
rkritika1508 Apr 2, 2026
09b6a05
fix: profanity free validator description
dennyabrain Apr 6, 2026
f4a11fa
doc: updated details of sentence parameter
dennyabrain Apr 7, 2026
f330f1b
fix: remove vscode files
dennyabrain Apr 7, 2026
51c9266
Added integration tests
rkritika1508 Apr 7, 2026
141e5fc
Merge branch 'main' into feat/toxicity-hub-validators
rkritika1508 Apr 7, 2026
c76f829
added integration tests
rkritika1508 Apr 7, 2026
baac9e4
fix: profanity free validator description
dennyabrain Apr 6, 2026
627fb4f
Added integration tests
rkritika1508 Apr 7, 2026
8b3da89
validator config: add name to config (#79)
nishika26 Apr 7, 2026
cc0bb14
added integration tests
rkritika1508 Apr 7, 2026
3037eb8
Merge branch 'feat/toxicity-hub-validators' into feat/toxicity-huggin…
rkritika1508 Apr 7, 2026
b69883d
added integration tests
rkritika1508 Apr 7, 2026
8f67176
updated readme
rkritika1508 Apr 7, 2026
affe72d
Added installation of huggingface model in dockerfile
rkritika1508 Apr 7, 2026
8b0a183
resolved comment
rkritika1508 Apr 7, 2026
14f6dc1
removed blank line
rkritika1508 Apr 7, 2026
74f8a82
updated policies for llama guard
rkritika1508 Apr 7, 2026
6676414
fixed tests
rkritika1508 Apr 7, 2026
0d15d0c
Merge branch 'feat/toxicity-hub-validators' into feat/toxicity-huggin…
rkritika1508 Apr 7, 2026
6443c1b
updated readme and fixed llama guard inference
rkritika1508 Apr 8, 2026
af933ef
fixed test
rkritika1508 Apr 8, 2026
9b6616a
Merge branch 'feat/toxicity-hub-validators' into feat/toxicity-huggin…
rkritika1508 Apr 9, 2026
9aca5f2
Merge branch 'main' into feat/toxicity-hub-validators
rkritika1508 Apr 10, 2026
664ded8
resolved comments
rkritika1508 Apr 10, 2026
0ce6ebb
Added evaluation readme (#82)
rkritika1508 Apr 10, 2026
ba27b80
resolved comments
rkritika1508 Apr 10, 2026
d7c5eba
resolved comments
rkritika1508 Apr 10, 2026
02fd043
fixed llama guard
rkritika1508 Apr 10, 2026
d9569ba
Merge branch 'feat/toxicity-hub-validators' into feat/toxicity-huggin…
rkritika1508 Apr 10, 2026
31af2f6
Toxicity Detection validators (#80)
rkritika1508 Apr 10, 2026
a061af8
Merge branch 'main' into feat/toxicity-huggingface-model
rkritika1508 Apr 10, 2026
88c1b56
removed unnecessary changes
rkritika1508 Apr 10, 2026
5b2fe3b
fix: update default nsfw_text model to michellejieli/NSFW_text_classi…
rkritika1508 Apr 10, 2026
fd3cddc
fix: use textdetox/xlmr-large-toxicity-classifier as default nsfw_tex…
rkritika1508 Apr 10, 2026
7264771
updated readme
rkritika1508 Apr 10, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions backend/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,14 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Install pinned spaCy model in the final environment used at runtime.
RUN python -m pip install --no-deps "${SPACY_MODEL_WHEEL_URL}"

# Set HuggingFace cache directory
ENV HF_HOME=/app/hf_cache

# Pre-download HuggingFace model
RUN /app/.venv/bin/python -c "from transformers import AutoTokenizer, AutoModelForSequenceClassification; \
AutoTokenizer.from_pretrained('textdetox/xlmr-large-toxicity-classifier', cache_dir='/app/hf_cache'); \
AutoModelForSequenceClassification.from_pretrained('textdetox/xlmr-large-toxicity-classifier', cache_dir='/app/hf_cache')"

# -------------------------------
# Entrypoint (runtime setup)
# -------------------------------
Expand Down
31 changes: 8 additions & 23 deletions backend/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,39 +272,24 @@ If verification succeeds, tenant's scope (`organization_id`, `project_id`) is re
> Set `OPENAI_API_KEY` in your `.env` / `.env.test` before using these validators.
> If the key is missing, `llm_critic` will raise a `ValueError` at build time and `topic_relevance` will return a validation failure with an explicit error message.

1. Ensure that the .env file contains the correct value from `GUARDRAILS_HUB_API_KEY`. The key can be fetched from [here](https://hub.guardrailsai.com/keys).
1. Ensure that the `.env` file contains the correct value for `GUARDRAILS_HUB_API_KEY`. The key can be fetched from [here](https://hub.guardrailsai.com/keys).

2. Make the `install_guardrails_from_hub.sh` script executable using this command (run this from the `backend` folder) -
2. Make the `install_guardrails_from_hub.sh` script executable (run from the `backend` folder):

```bash
chmod +x scripts/install_guardrails_from_hub.sh
```
3. Run this command to configure Guardrails AI -

```bash
scripts/install_guardrails_from_hub.sh;
```

### Alternate Method
Run the following commands inside your virtual environment:
3. Run the script to configure Guardrails and install all hub validators:

```bash
uv sync
guardrails configure

Enable anonymous metrics reporting? [Y/n]: Y
Do you wish to use remote inferencing? [Y/n]: Y
Enter API Key below leave empty if you want to keep existing token [HBPo]
👉 You can find your API Key at https://hub.guardrailsai.com/keys
GUARDRAILS_HUB_API_KEY=<your-key> bash scripts/install_guardrails_from_hub.sh
```

To install any validator from Guardrails Hub:
```bash
guardrails hub install hub://guardrails/<validator-name>

Example -
guardrails hub install hub://guardrails/ban_list
```
> **Remote inferencing is enabled by default.** The script sets `ENABLE_REMOTE_INFERENCING=true` unless overridden. This is required for `llamaguard_7b`, which runs inference on the Guardrails Hub. You can disable it explicitly if needed:
> ```bash
> GUARDRAILS_HUB_API_KEY=<your-key> ENABLE_REMOTE_INFERENCING=false bash scripts/install_guardrails_from_hub.sh
> ```

## Adding a new validator from Guardrails Hub
To add a new validator from the Guardrails Hub to this project, follow the steps below.
Expand Down
5 changes: 4 additions & 1 deletion backend/app/api/API_USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Endpoint:
Optional filters:
- `ids=<uuid>&ids=<uuid>`
- `stage=input|output`
- `type=uli_slur_match|pii_remover|gender_assumption_bias|ban_list|llm_critic|topic_relevance`
- `type=uli_slur_match|pii_remover|gender_assumption_bias|ban_list|llm_critic|topic_relevance|llamaguard_7b|profanity_free|nsfw_text`

Example:

Expand Down Expand Up @@ -442,6 +442,9 @@ From `validators.json`:
- `ban_list`
- `llm_critic`
- `topic_relevance`
- `llamaguard_7b`
- `profanity_free`
- `nsfw_text`

Source of truth:
- `backend/app/core/validators/validators.json`
Expand Down
10 changes: 10 additions & 0 deletions backend/app/api/docs/guardrails/run_guardrails.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,16 @@ Behavior notes:
- For `ban_list`, `ban_list_id` can be resolved to `banned_words` from tenant ban list configs.
- For `topic_relevance`, `topic_relevance_config_id` is required and is resolved to `configuration` + `prompt_schema_version` from tenant topic relevance configs in `guardrails.py`. Requires `OPENAI_API_KEY` to be configured; returns a validation failure with an explicit error if missing.
- For `llm_critic`, `OPENAI_API_KEY` must be configured; returns `success=false` with an explicit error if missing.
- For `llamaguard_7b`, `policies` accepts human-readable policy names (see table below). If omitted, all policies are enforced by default.

| `policies` value | Policy enforced |
|-----------------------------|----------------------------------|
| `no_violence_hate` | No violence or hate speech |
| `no_sexual_content` | No sexual content |
| `no_criminal_planning` | No criminal planning |
| `no_guns_and_illegal_weapons` | No guns or illegal weapons |
| `no_illegal_drugs` | No illegal drugs |
| `no_encourage_self_harm` | No encouragement of self-harm |
- `rephrase_needed=true` means the system could not safely auto-fix the input/output and wants the user to retry with a rephrased query.
- When `rephrase_needed=true`, `safe_text` contains the rephrase prompt shown to the user.

Expand Down
3 changes: 3 additions & 0 deletions backend/app/api/routes/guardrails.py
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,9 @@ def add_validator_logs(
for log in iteration.outputs.validator_logs:
result = log.validation_result

if result is None:
continue

if suppress_pass_logs and isinstance(result, PassResult):
continue

Expand Down
4 changes: 4 additions & 0 deletions backend/app/core/enum.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,7 @@ class ValidatorType(Enum):
GenderAssumptionBias = "gender_assumption_bias"
BanList = "ban_list"
TopicRelevance = "topic_relevance"
LLMCritic = "llm_critic"
LlamaGuard7B = "llamaguard_7b"
ProfanityFree = "profanity_free"
NSFWText = "nsfw_text"
Loading