Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Help contribute by opening a pull request to add more resources and tools!
* [Content Safety API by Google](https://protectingchildren.google/tools-for-partners/#learn-about-our-tools)
* uses machine learning to detect novel CSAM, nudity, and sexually explicit content in images and videos
* free service, but requires registration
* not open source itself, but can be [used via Coop](https://roostorg.github.io/coop/SIGNALS.html#content-safety-api-by-google), which is open source
* not open source itself, but can be [used via Coop](https://roostorg.github.io/coop/latest/integrations/google-content-safety.html), which is open source
* [CoPE by Zentropi](https://huggingface.co/zentropi-ai/cope-a-9b)
* small language model trained for accurate, fast, steerable content classification based on developer-defined content policies
* [Detoxify by Unitary AI](https://github.com/unitaryai/detoxify)
Expand Down Expand Up @@ -123,7 +123,7 @@ Help contribute by opening a pull request to add more resources and tools!
* Tool for testing prompt injection vulnerabilities in AI systems
* [Promptfoo](https://github.com/promptfoo/promptfoo)
* Automated LLM evaluations, report generations, several ready-to-use attack strategies
* [PyRIT by Microsoft](https://github.com/Azure/PyRIT)
* [PyRIT by Microsoft](https://github.com/microsoft/PyRIT)
* Microsoft’s Python-based tool for AI red teaming and security testing
* [Socketteer](https://github.com/socketteer?tab=repositories)
* Allows AI models to interact, helping test conversational weaknesses
Expand Down Expand Up @@ -233,12 +233,10 @@ Help contribute by opening a pull request to add more resources and tools!
* Questions across multiple risk areas and harm types to test LLM safety and refusal behavior
* [Forbidden Questions dataset](https://huggingface.co/datasets/TrustAIRLab/forbidden_question_set)
* Questions adopted from OpenAI Usage Policy
* [HackAPrompt Jailbreak Dataset](https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset/viewer/default/train?p=1&row=137)
* [HackAPrompt Jailbreak Dataset](https://huggingface.co/datasets/hackaprompt/hackaprompt-dataset)
* dataset for testing AI vulnerability to prompt-based jailbreaking
* [HarmBench by Center for AI Safety](https://github.com/centerforaisafety/HarmBench)
* Evaluation dataset for automated red teaming
* [HiroKachi Jailbreak Dataset](https://sizu.me/love)
* dataset focused on adversarial AI prompt attacks
* [Jailbreak Prompt Generator AI Model](https://huggingface.co/tsq2000/Jailbreak-generator)
* AI model that generates jailbreak-style prompts
* [JailbreakBench](https://huggingface.co/datasets/JailbreakBench/JBB-Behaviors)
Expand All @@ -251,7 +249,7 @@ Help contribute by opening a pull request to add more resources and tools!
* Medical safety prompts to evaluate LLM safety in medical contexts
* [Multilingual Vulnerability dataset](https://github.com/CarsonDon/Multilingual-Vuln-LLMs)
* Multilingual prompts demonstrating LLM vulnerabilities
* [Red Team Resistance Leaderboard](https://huggingface.co/spaces/HaizeLabs/red-teaming-resistance-benchmark)
* [Red Team Resistance Leaderboard](https://github.com/haizelabs/redteaming-resistance-benchmark)
* rankings of AI models based on resistance to adversarial attacks
* [Rentry Jailbreak Datasets](https://rentry.org/gpt0721)
* collection of datasets related to jailbreak attempts on AI models
Expand Down