Articles

Survey

• Hacking Auto-GPT and escaping its docker container
• Indirect Prompt Injection Threats
• Prompt injection: What’s the worst that can happen?
• OWASP Top 10 for Large Language Model Applications
• PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news
• ChatGPT Plugins: Data Exfiltration via Images & Cross Plugin Request Forgery
• Jailbreaking GPT-4's code interpreter
• Adversarial Attacks on LLMs
• LLM Evaluation metrics, frmaework, and checklist
• How RAG Poisoning Made Llama3 Racist!
• Prompt Injection Cheat Sheet: How To Manipulate AI Language Models
• How Anyone can Hack ChatGPT - GPT4o
• Securing LLM Systems Against Prompt Injection

Benchmark

Survey

• [paper
• [paper - spylab/agentdojo) [site]
• [paper - Prompt-Injection)
• [paper

Other Awesome Projects

Survey

• Gandalf
• LangChain vulnerable to code injection - CVE-2023-29374
• Adversarial Prompting
• Epivolis
• LLM Security Problems at DEFCON31 Quals
• PALLMs (Payloads for Attacking Large Language Models)
• LLM Security startups
• PromptBounty.io

Other Useful Resources

Survey

• LLM Security
• Embrace The Red
• Kai's Blog
• AI safety takes
• Hackstery

Papers

Backdoor attack

• [paper
• [paper
• [paper - prompt-injection) [site]

Black-box attack

• [paper
• [paper
• [paper - jailbreak/tree/main)
• [paper
• [paper
• [paper
• [paper
• [paper - Prompt-Injection)
• [paper
• [paper
• [paper
• [paper - AI/do-not-answer) [dataset]
• [paper
• [paper
• [paper - Tuning-Safety/LLMs-Finetuning-Safety) [site] [dataset]
• [paper
• [paper
• [paper - NLP-SG/multilingual-safety-for-LLMs)
• [paper
• [paper - group/DeepInception) [site]
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper
• [paper - evaluation)
• [paper
• [paper - ai/JOOD)

Defense

• [paper - defenses)
• [paper - self-defense) [site]
• [paper
• [paper
• [paper
• [paper - liu/IB4LLMs)
• [paper - Zh/PARDEN)
• [paper
• [paper - breakers)

Fingerprinting

• [paper - Fingerprint) [site]
• [paper
• [paper - dario/LLMmap)

Platform Security

• [paper - platform-security/chatgpt-plugin-eval)

Survey

• [paper
• [paper
• [paper
• [paper

White-box attack

• [paper - Adversarial-Examples-Jailbreak-Large-Language-Models)
• [paper
• [paper
• [paper - attacks/llm-attacks) [page]
• [paper
• [paper - hijacks) [site]
• [paper - to-strong)

Tools

Survey

• Rebuff - hardening prompt injection detector
• LLMFuzzer
• Vigil - llm?style=social)
• jailbreak-evaluation - to-use Python package for language model jailbreak evaluation
• Prompt Fuzzer - source tool to help you harden your GenAI applications
• WhistleBlower - source tool designed to infer the system prompt of an AI agent based on its generated text outputs.
• Open-Prompt-Injection - source tool to evaluate prompt injection attacks and defenses on benchmark datasets.
• Agentic Radar - source CLI security scanner for agentic workflows.
• LLM Guard - ai/llm-guard?style=social)
• PurpleLlama
• Garak
• UTCP - calling to any native endpoint for your AI agent