Black-box attack

• [paper
• [paper
• [paper
• [paper - jailbreak/tree/main)
• [paper
• [paper - AI/do-not-answer) [[dataset]](https://huggingface.co/datasets/LibrAI/do-not-answer)
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper - jailbreak/tree/main)
• [paper
• [paper
• [paper
• [paper
• [paper - Prompt-Injection)
• [paper
• [paper
• [paper
• [paper - AI/do-not-answer) [[dataset]](https://huggingface.co/datasets/LibrAI/do-not-answer)
• [paper
• [paper
• [paper - Tuning-Safety/LLMs-Finetuning-Safety) [[site]](https://llm-tuning-safety.github.io/) [[dataset]](https://huggingface.co/datasets/LLM-Tuning-Safety/HEx-PHI)
• [paper
• [paper
• [paper - NLP-SG/multilingual-safety-for-LLMs)
• [paper
• [paper - group/DeepInception) [[site]](https://deepinception.github.io/)
• [paper
• [paper
• [paper
• [paper - evaluation)
• [paper
• [paper
• [paper
• [paper - jailbreak/tree/main)
• [paper
• [paper
• [paper
• [paper
• [paper - Prompt-Injection)
• [paper
• [paper
• [paper
• [paper - AI/do-not-answer) [[dataset]](https://huggingface.co/datasets/LibrAI/do-not-answer)
• [paper
• [paper - NLP-SG/multilingual-safety-for-LLMs)
• [paper
• [paper - group/DeepInception) [[site]](https://deepinception.github.io/)
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper - Tuning-Safety/LLMs-Finetuning-Safety) [[site]](https://llm-tuning-safety.github.io/) [[dataset]](https://huggingface.co/datasets/LLM-Tuning-Safety/HEx-PHI)
• [paper
• [paper
• [paper
• [paper - evaluation)
• [paper - ai/JOOD)

• BITE: Textual Backdoor Attacks with Iterative Trigger Injection

Backdoor attack

• [paper
• [paper - prompt-injection) [[site]](https://poison-llm.github.io/)
• [paper
• [paper - prompt-injection) [[site]](https://poison-llm.github.io/)
• [paper
• [paper
• [paper
• [paper - prompt-injection) [[site]](https://poison-llm.github.io/)

Defense

• [paper - self-defense) [[site]](https://mphute.github.io/papers/llm-self-defense)
• [paper - defenses)
• [paper
• [paper
• [paper
• [paper
• [paper - liu/IB4LLMs)
• [paper - Zh/PARDEN)
• [paper - liu/IB4LLMs)
• [paper - breakers)
• [paper - defenses)
• [paper - self-defense) [[site]](https://mphute.github.io/papers/llm-self-defense)
• [paper
• [paper
• [paper
• [paper - liu/IB4LLMs)
• [paper - Zh/PARDEN)
• [paper
• [paper - breakers)

Platform Security

• [paper - platform-security/chatgpt-plugin-eval)
• [paper - platform-security/chatgpt-plugin-eval)
• [paper - platform-security/chatgpt-plugin-eval)

Survey

• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper

White-box attack

• [paper - Adversarial-Examples-Jailbreak-Large-Language-Models)
• [paper
• [paper
• [paper - to-strong)
• [paper - Adversarial-Examples-Jailbreak-Large-Language-Models)
• [paper
• [paper
• [paper - attacks/llm-attacks) [[page]](https://llm-attacks.org/)
• [paper
• [paper - hijacks) [[site]](https://image-hijacks.github.io)
• [paper - to-strong)
• [paper - Adversarial-Examples-Jailbreak-Large-Language-Models)
• [paper
• [paper
• [paper - attacks/llm-attacks) [[page]](https://llm-attacks.org/)
• [paper
• [paper - hijacks) [[site]](https://image-hijacks.github.io)
• [paper - to-strong)

Fingerprinting

• [paper - dario/LLMmap)
• [paper - Fingerprint) [[site]](https://cnut1648.github.io/Model-Fingerprint/)
• [paper
• [paper - Fingerprint) [[site]](https://cnut1648.github.io/Model-Fingerprint/)
• [paper
• [paper - dario/LLMmap)

Survey

• Prompt Injection Cheat Sheet: How To Manipulate AI Language Models
• Hacking Auto-GPT and escaping its docker container
• Prompt Injection Cheat Sheet: How To Manipulate AI Language Models
• Indirect Prompt Injection Threats
• The AI Attack Surface Map v1.0
• LLM Evaluation metrics, frmaework, and checklist
• How RAG Poisoning Made Llama3 Racist!
• Prompt injection: What’s the worst that can happen?
• OWASP Top 10 for Large Language Model Applications
• PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news
• ChatGPT Plugins: Data Exfiltration via Images & Cross Plugin Request Forgery
• Jailbreaking GPT-4's code interpreter
• Adversarial Attacks on LLMs
• How Anyone can Hack ChatGPT - GPT4o

• Securing LLM Systems Against Prompt Injection

• PromptBounty.io

Survey

• PALLMs (Payloads for Attacking Large Language Models)
• LLM Security startups
• Gandalf
• LangChain vulnerable to code injection - CVE-2023-29374
• Jailbreak Chat
• Adversarial Prompting
• Epivolis
• LLM Security Problems at DEFCON31 Quals

Survey

• @llm_sec
• LLM Security
• Embrace The Red
• Kai's Blog
• AI safety takes
• @llm_sec

• Hackstery

Survey

• [paper
• [paper
• [paper - spylab/agentdojo) [[site]](https://agentdojo.spylab.ai/)
• [paper
• [paper
• [paper - spylab/agentdojo) [[site]](https://agentdojo.spylab.ai/)
• [paper - Prompt-Injection)
• [paper

Survey

• Open-Prompt-Injection - source tool to evaluate prompt injection attacks and defenses on benchmark datasets. 
• Rebuff - hardening prompt injection detector 
• LLMFuzzer
• Vigil - llm?style=social)
• jailbreak-evaluation - to-use Python package for language model jailbreak evaluation 
• Prompt Fuzzer - source tool to help you harden your GenAI applications 
• WhistleBlower - source tool designed to infer the system prompt of an AI agent based on its generated text outputs. 
• Plexiglass - labs/plexiglass?style=social)
• Garak
• LLM Guard - ai/llm-guard?style=social)
• Agentic Radar - source CLI security scanner for agentic workflows.