Skip to content

CHANGELOG.md

Latest commit

 

History

History
55 lines (31 loc) · 1.96 KB

CHANGELOG.md

File metadata and controls

55 lines (31 loc) · 1.96 KB
Raw

PINT Benchmark Changelog

Any significant changes to the benchmark evaluation code, benchmark output, benchmark results table, or the underlying dataset will be noted here.

2025-12-16

Changed

  • Merged public_prompt_injection and internal_prompt_injection categories into a single prompt_injection

2024-04-12

Benchmark Results Table

  • Added updated results for Epivolis/Hyperion to the benchmark results table due to error in previous results due to incorrect injection_label value

2024-04-09

Benchmark Results Table

  • Added updated results for Lakera Guard to the benchmark results table

2024-04-05

Benchmark Results Table

  • Added Azure AI Prompt Shields to the benchmark results table
  • Updated protectai/deberta-v3-base-prompt-injection results to account for slight increase in score after stride was added to chunking code

Benchmark Output

  • Added the date the benchmark was run to the benchmark output

Benchmark Evaluation Code

  • Added stride parameter of 25% of the model's max_length to the HuggingFaceModelEvaluation implementation of tokenizing and chunking long text for classification

Note: Any model's scores affected by this change were updated in the benchmark results table.

2024-04-04

Benchmark Results Table

  • Added results for Lakera Guard to the benchmark results table
  • Added results for protectai/deberta-v3-base-prompt-injection to the benchmark results table
  • Added results for WhyLabs LangKit to the benchmark results table
  • Added results for fmops/distilbert-prompt-injection to the benchmark results table
  • Added results for deepset/deberta-v3-base-injection to the benchmark results table
  • Added results for Myadav/setfit-prompt-injection-MiniLM-L3-v2 to the benchmark results table
  • Added results for Epivolis/Hyperion to the benchmark results table

Benchmark Evaluation Code

  • Added HuggingFaceModelEvaluation class for conveniently evaluating Hugging Face models