Commit 8034e8a
committed
fix(e2e): enable LoRA PII detection for aibrix profile
This commit fixes PII detection test failures for the aibrix profile by switching
from the non-functional ModernBERT PII model to LoRA-based auto-detection, matching
the configuration already proven to work in dynamic-config and ai-gateway profiles.
## Problem
The aibrix profile PII detection test was failing with 0% accuracy (0/100 tests passed).
All 100 PII test requests were passing through without being blocked, even though they
contained sensitive data like credit cards, SSNs, emails, and IP addresses.
## Root Causes
**Same issues as ai-gateway had before the previous fix**:
1. **Using outdated ModernBERT PII model**:
- Profile was using `models/pii_classifier_modernbert-base_presidio_token_model`
- ModernBERT classifier initialized but never detected any PII entities
- No "Detected PII" or "PII token classification" logs during test runs
- Result: 0% detection accuracy
2. **Missing default_decision fallback**:
- No catch-all decision for PII policy fallback mechanism
- PII policy code (src/semantic-router/pkg/utils/pii/policy.go) falls back to "default_decision"
- Without it, edge cases with empty decision names would disable PII detection
3. **No profile-specific model configuration**:
- Test was using inherited E2E_TEST_MODEL from previous test runs
- Not explicitly configured for aibrix's model: vllm-llama3-8b-instruct
- Could cause inconsistent behavior across test runs
## Solution
### 1. Switch AIBrix to LoRA PII Detection (deploy/kubernetes/aibrix/semantic-router-values/values.yaml)
**Change 1**: Updated pii_model configuration (lines 459-466)
```yaml
pii_model:
# Support both traditional (modernbert) and LoRA-based PII detection
# When model_type is "auto", the system will auto-detect LoRA configuration
model_id: "models/lora_pii_detector_bert-base-uncased_model"
model_type: "auto" # Enables LoRA auto-detection
threshold: 0.7
use_cpu: true
pii_mapping_path: "models/lora_pii_detector_bert-base-uncased_model/pii_type_mapping.json"
```
**Why**:
- ModernBERT PII model was not detecting any PII (0% accuracy in tests)
- LoRA PII model proven to work in both dynamic-config and ai-gateway (100% accuracy)
- `model_type: "auto"` enables automatic LoRA model detection
- Uses same battle-tested model across all profiles for consistency
- Aligns aibrix configuration with the working profiles
**Change 2**: Added default_decision for fallback (lines 386-401)
```yaml
- name: default_decision
description: "Default catch-all decision - blocks all PII for safety"
priority: 0
rules:
operator: "OR"
conditions:
- type: "domain"
name: "other"
modelRefs:
- model: vllm-llama3-8b-instruct
plugins:
- type: "pii"
configuration:
enabled: true
pii_types_allowed: []
```
**Why**:
- PII policy code falls back to "default_decision" when decision lookup fails
- Priority 0 ensures it's only used as last resort
- Blocks all PII types for maximum safety
- Prevents edge cases where PII detection would be disabled
- Required by fallback mechanism in src/semantic-router/pkg/utils/pii/policy.go
### 2. Configure AIBrix Profile Test Model (e2e/profiles/aibrix/profile.go)
**Change**: Set environment variable in Setup() method (lines 84-85)
```go
// Configure PII test to use vllm-llama3-8b-instruct model
os.Setenv("E2E_TEST_MODEL", deploymentDemoLLM)
```
where `deploymentDemoLLM = "vllm-llama3-8b-instruct"`
**Why**:
- AIBrix uses different model names than dynamic-config/ai-gateway
- Ensures test explicitly uses the correct aibrix model
- Prevents reliance on environment variable inheritance from other tests
- Matches the approach used in dynamic-config profile (sets E2E_TEST_MODEL=MoM)
- Makes test behavior predictable and independent
## Testing
**Before Fix**:
- aibrix: 0/100 PII tests passed (0% accuracy) ❌
**After Fix**:
- aibrix: 100/100 PII tests passed (100% accuracy) ✅
**Test Command**:
```bash
make e2e-cleanup && make e2e-test E2E_PROFILE=aibrix E2E_VERBOSE=true E2E_KEEP_CLUSTER=true
```
**Verified**: No impact on other profiles (dynamic-config and ai-gateway) as changes are isolated to aibrix-specific files only.
## Files Changed
1. **deploy/kubernetes/aibrix/semantic-router-values/values.yaml** (Lines 386-401, 459-466)
- Added default_decision for PII policy fallback
- Switched pii_model from ModernBERT to LoRA auto-detection
- Aligned with dynamic-config and ai-gateway working configuration
2. **e2e/profiles/aibrix/profile.go** (Lines 84-85)
- Sets E2E_TEST_MODEL=vllm-llama3-8b-instruct in Setup()
- Ensures profile-specific model configuration
- Makes test behavior independent and predictable
## Why This Works
**AIBrix flow**:
1. Test uses model="vllm-llama3-8b-instruct" (via E2E_TEST_MODEL env var)
2. Routes to decision (either matched or falls back to default_decision)
3. Decision has PII plugin enabled → PII detection runs
4. LoRA PII classifier detects entities (credit cards, SSNs, emails, etc.)
5. Policy blocks request → 100% accuracy ✅
All three profiles (dynamic-config, ai-gateway, aibrix) now use the same proven
LoRA PII detection model with 100% accuracy across all E2E tests.
## Summary of All Profiles
| Profile | PII Detection | Configuration |
|---------|--------------|---------------|
| dynamic-config | 100/100 (100%) ✅ | LoRA auto-detection, model=MoM |
| ai-gateway | 100/100 (100%) ✅ | LoRA auto-detection, model=general-expert |
| aibrix | 100/100 (100%) ✅ | LoRA auto-detection, model=vllm-llama3-8b-instruct |
Signed-off-by: Yossi Ovadia <[email protected]>1 parent fbd782f commit 8034e8a
File tree
2 files changed
+28
-3
lines changed- deploy/kubernetes/aibrix/semantic-router-values
- e2e/profiles/aibrix
2 files changed
+28
-3
lines changedLines changed: 25 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
380 | 380 | | |
381 | 381 | | |
382 | 382 | | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
383 | 403 | | |
384 | 404 | | |
385 | 405 | | |
| |||
437 | 457 | | |
438 | 458 | | |
439 | 459 | | |
440 | | - | |
441 | | - | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
442 | 464 | | |
443 | 465 | | |
444 | | - | |
| 466 | + | |
445 | 467 | | |
446 | 468 | | |
447 | 469 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
84 | 87 | | |
85 | 88 | | |
86 | 89 | | |
| |||
0 commit comments