https://github.com/wearetyomsmnv/awesome-llm-agent-security

All about llm-agents security,attack,vulnerabilities and how to do them for cybersecurity.
https://github.com/wearetyomsmnv/awesome-llm-agent-security

List: awesome-llm-agent-security

Last synced: 2 months ago
JSON representation

All about llm-agents security,attack,vulnerabilities and how to do them for cybersecurity.

• Host: GitHub
• URL: https://github.com/wearetyomsmnv/awesome-llm-agent-security
• Owner: wearetyomsmnv
• License: unlicense
• Created: 2025-01-20T13:21:50.000Z (4 months ago)
• Default Branch: main
• Last Pushed: 2025-01-21T14:06:38.000Z (4 months ago)
• Last Synced: 2025-03-15T01:01:43.176Z (2 months ago)
• Size: 16.6 KB
• Stars: 6
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• ultimate-awesome - awesome-llm-agent-security - All about llm-agents security,attack,vulnerabilities and how to do them for cybersecurity. (Other Lists / Julia Lists)

README

        
Awesome LLM Agent Security




  





  

  





A curated list of resources about LLM agent security, vulnerabilities, attacks, and their applications in cybersecurity.



📚 Contents




| 🔍 [Research](#research) | 🎯 [Threat Model](#threat-model) | 🛠️ [Build Agent](#build-agent) | 📊 [Results](#results) | 💼 [Solutions](#solutions) |

|:-----------------------:|:--------------------------------:|:------------------------------:|:---------------------:|:------------------------:|

| _Papers & Publications_ | _Threats & Vulnerabilities_ | _Development Guides for cybersecurity_ | _Successful cases_ | _Tools Overview_ |



🔰 Fundamentals


Basic Concepts


### LLM Agent - is an intelligent system used by a large language model to perform tasks.

> [!NOTE]

> **MultiAgent**: When there is an ensemble of many agents that perform one large task - this is called a multi-agent system. It is based on a large language model that is responsible for task planning and decision making.

---

### Memory

Memory in autonomous agents can be categorized into two main types:

• Short-Term Memory: This refers to the agent's ability to utilize in-context learning, where it retains information temporarily during a single interaction. This is often limited by the model's context window, which restricts the amount of information it can process at once.

• Long-Term Memory: This allows the agent to store and recall information over extended periods. Long-term memory is typically managed through external vector stores, enabling the agent to retrieve relevant information quickly. This capability is crucial for tasks that require knowledge accumulation and recall over time.

Memory plays a vital role in how agents learn from past experiences, refine their actions, and improve their performance in future tasks.

---

### Planning

Planning involves the agent's ability to break down complex tasks into manageable subgoals. This process can be enhanced through various techniques:

Task Decomposition: The agent can decompose a large task into smaller, more manageable steps. Techniques like Chain of Thought (CoT) prompting encourage the model to think step-by-step, making it easier to tackle complex problems.

Self-Reflection: Agents can evaluate their past actions, learn from mistakes, and refine their strategies. This iterative process helps improve decision-making and planning over time.

External Planning Tools: Some agents may utilize classical planning methods, such as the Planning Domain Definition Language (PDDL), to generate structured plans based on predefined domains.

Effective planning is essential for agents to navigate complex tasks and adapt to changing circumstances.

---

### Action

Action refers to the execution of tasks based on the agent's planning and memory. This involves:

Tool Use: Agents can interact with external APIs and tools to gather information, execute code, or perform specific functions that extend their capabilities beyond what is encoded in their model weights.

ReAct Framework: This framework integrates reasoning and action, allowing agents to generate reasoning traces alongside their actions. This helps in understanding the rationale behind decisions and improving future actions.

Dynamic Interaction: Agents can adapt their actions based on real-time observations and feedback from their environment, allowing for more responsive and intelligent behavior.



  



## 🔄 Framework Comparison

🔄 Framework Comparison




| Framework | Key Features | Focus Area | License | Language Support | Distributed Systems |

|:---------:|:------------|:-----------|:--------|:-----------------|:-------------------|

| **LangChain** | • Chain-based architecture
• Memory management
• Tool integration | General purpose LLM apps | MIT | Python, JavaScript | Limited |

| **AutoGPT** | • Autonomous goal pursuit
• Long-term memory
• Self-prompting | Autonomous agents | MIT | Python | No |

| **AgentGPT** | - Web-based interface
- Task decomposition
- Visual workflow | Task automation | MIT | TypeScript | Yes |

| **BabyAGI** | - Task prioritization
- Simple architecture
- Learning focus | Research & Education | MIT | Python | No |

| **Lyzr** | - Enterprise security
- Scalable architecture
- Pre-built agents | Enterprise solutions | Commercial | Python | Yes |

| **CrewAI** | - Multi-agent collaboration
- Role-based agents
- Team coordination | Complex workflows | Apache 2.0 | Python | Yes |



### Key Differences:



| Category | Features | Examples/Details |

|:---------|:---------|:----------------|

| **Architecture Focus** | • Task-oriented
• Chain-based
• Multi-agent
• Enterprise-grade | • BabyAGI, AgentGPT
• LangChain
• CrewAI
• Lyzr |

| **Use Case Optimization** | • Research & Experimentation
• Enterprise Applications
• Personal Automation
• Educational Purposes | • Academic projects
• Business solutions
• Individual tools
• Learning platforms |

| **Development Approach** | • Low-code solutions
• Programming-intensive
• Visual builders
• API-first design | • No-code platforms
• Custom development
• Drag-and-drop interfaces
• API integration |

| **Deployment Options** | • Cloud-native
• Self-hosted
• Hybrid deployment
• Edge computing support | • Cloud platforms
• On-premise solutions
• Mixed environments
• Edge devices |

| **Integration Capabilities** | • API connectivity
• Database support
• Third-party tools
• Custom extensions | • REST/GraphQL APIs
• Various DB systems
• External services
• Custom plugins |



## 🤖 LLM Agents in Cybersecurity

Agent Core Properties


| Essential Components | Description |

|:--------------------|:------------|

| **Role Definition** | Specific security function and responsibilities |

| **Goal Setting** | Clear security objectives and success criteria |

| **Backstory** | Detailed capabilities and operational context |

| **Tools Access** | Integration with security tools and APIs |

| Key Capabilities | Description |

|:-----------------|:------------|

| **Task Inheritance** | Ability to receive and delegate security tasks |

| **Boundary Management** | Operating within defined security constraints |

| **Tool Utilization** | Leveraging security tools and APIs effectively |

| **Collaborative Assessment** | Evaluating and coordinating with other agents |

Security Applications & Benefits


| Category | Features |

|:---------|:---------|

| **Threat Detection & Response** | • Real-time monitoring and alert triage
• Automated incident response workflows |

| **Security Operations** | • 24/7 autonomous security monitoring
• Automated routine security tasks |

| Category | Features |

|:---------|:---------|

| **Vulnerability Management** | • Continuous security assessment
• Automated vulnerability scanning |

| **Incident Investigation** | • Automated evidence collection
• Timeline reconstruction |

Advantages in Security Context


| Category | Capabilities |

|:---------|:-------------|

| **Enhanced Efficiency** | • Continuous operation without fatigue
• Rapid processing of security data |

| **Improved Accuracy** | • Reduced human error
• Standardized analysis methods |

| Category | Capabilities |

|:---------|:-------------|

| **Scalability** | • Handling multiple tasks simultaneously
• Easy deployment across systems |

| **Advanced Capabilities** | • Complex pattern recognition
• Real-time threat analysis |

Creation Methods




| Method | Description |

|:-------|:------------|

| **Human-designed** | Agents with specific security roles |

| **Auto-generated** | Agents created by LLMs for specialized tasks |

| **Hybrid** | Approaches combining human expertise and AI capabilities |



## Security Landscape

### 🔒 OWASP Top 10 for AI Agents (Non official)



| Category | Risk | Description |

|:---------|:-----|:------------|

| [AAC-01](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-auth-control-01.md) | Authorization & Control Hijacking | Unauthorized control of agent actions |

| [ACS-02](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-critical-systems-02.md) | Critical Systems Interaction | Unsafe interaction with critical systems |

| [AGI-03](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-goal-instruction-03.md) | Goal & Instruction Manipulation | Malicious modification of agent objectives |

| [AHE-04](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-hallucination-04.md) | Hallucination Exploitation | Exploitation of agent's false assumptions |

| [AIC-05](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-impact-chain-05.md) | Impact Chain & Blast Radius | Cascading effects of agent actions |

| [AMC-06](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-memory-context-06.md) | Memory & Context Manipulation | Tampering with agent's memory systems |

| [AOR-07](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-orchestration-07.md) | Orchestration Exploitation | Multi-agent system vulnerabilities |

| [ARE-08](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-resource-exhaustion-8.md) | Resource Exhaustion | DoS and resource depletion attacks |

| [ASC-09](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-supply-chain-09.md) | Supply Chain Attacks | Compromised dependencies and components |

| [AKP-10](https://github.com/precize/OWASP-Agentic-AI/blob/main/agent-knowledge-poisoning-10.md) | Knowledge Base Poisoning | Contamination of agent's knowledge base |





  



 Practice Labs & Applications 




| Project | Type | Features | Purpose |

|:--------|:-----|:---------|:---------|

| [Damn Vulnerable LLM Agent](https://github.com/WithSecureLabs/damn-vulnerable-llm-agent) | Educational Lab | - ReAct agent testing
- Prompt injection scenarios
- SQL injection practice | Learning prompt & ReAct injection techniques |

| [Medusa](https://medusa.detoxio.dev/) | Testing Platform | - Agent vulnerability testing
- Security assessment
- Attack simulation | Practical security testing |



## 📚 Research & Publications

 Academic Papers 




| Title | Authors | Year | Key Findings |

|:------|:--------|:-----|:-------------|

| [Security Concerns with AI Agents](https://www.vpnranks.com/resources/security-concerns-with-ai-agents/) | VPNRanks | 2024 | - 52.5% data leakage predicted by 2025
- Market growth to $7.41B
- PII exposure risks |

| [Key Challenges in AI Agent Security](https://taleliyahu.medium.com/key-challenges-in-ai-agent-security-332d718ec8b4) | Tal Eliyahu | 2024 | - Confidentiality risks
- Integrity concerns
- Availability threats |

| [Beyond RCE: Autonomous Code Execution](https://www.securityrunners.io/post/beyond-rce-autonomous-code-execution-in-agentic-ai) | Security Runners | 2024 | - Code execution risks
- Agent autonomy threats
- Security implications |

| [Exploiting Huggingface's Assistants](https://www.lasso.security/blog/exploiting-huggingfaces-assistants-to-extract-users-data) | Lasso Security | 2023 | - Data extraction vulnerabilities
- Assistant exploitation
- Security measures |



 Security Tools & Frameworks 




| Project | Type | Description | Features |

|:--------|:-----|:------------|:----------|

| [HackSynth](https://github.com/aielte-research/HackSynth) | Framework | AI security testing framework | - Vulnerability assessment
- Attack simulation
- Security validation |

| [OsintAGI](https://github.com/wearetyomsmnv/OsintAGI/) | Tool | OSINT automation framework | - Intelligence gathering
- Data analysis
- Automated research |

| [Agent-Smith](https://github.com/sail-sg/Agent-Smith) | Security Tool | Agent security testing | - Behavior analysis
- Vulnerability detection
- Security assessment |

| [AI-OPS](https://github.com/antoninoLorenzo/AI-OPS) | Platform | Security operations for AI | - Threat detection
- Response automation
- Security monitoring |

| [PentAGI](https://github.com/vxcontrol/pentagi/) | Security Tool | Automated penetration testing | - Autonomous AI agents
- Professional security tools
- Comprehensive monitoring |



 Benchmarks & Evaluations 




| Project | Focus | Metrics | Key Features |

|:--------|:------|:--------|:-------------|

| [Agent-Attack](https://github.com/ChenWu98/agent-attack) | Attack Testing | Security vulnerabilities | - Attack vectors
- Defense evaluation
- Risk assessment |

| [Auto-Pen-Bench](https://github.com/lucagioacchini/auto-pen-bench) | Penetration Testing | Security benchmarks | - Automated testing
- Performance metrics
- Security scoring |

| [ASB](https://github.com/agiresearch/ASB) | Security Benchmark | Agent security | - Security metrics
- Performance analysis
- Vulnerability testing |

| [LLM-Agent-Benchmark](https://github.com/zhangxjohn/LLM-Agent-Benchmark-List) | Comprehensive | Agent evaluation | - Security testing
- Performance metrics
- Benchmark collection |



 Security Projects & Implementations 




| Project | Type | Purpose | Features |

|:--------|:-----|:--------|:----------|

| [Multi-Agent-SecOps](https://github.com/tegridydev/multi-agent-secops-llm) | Security Operations | LLM-based security | - Threat detection
- Response automation
- Security monitoring |

| [Cyber-Security-LLM-Agents](https://github.com/NVISOsecurity/cyber-security-llm-agents) | Security Framework | Agent-based security | - Security automation
- Threat analysis
- Response coordination |

| [Ridge Security](https://ridgesecurity.ai/) | Platform | AI security solution | - Vulnerability assessment
- Security testing
- Risk management |



 Technical Presentations & Whitepapers 




| Title | Organization | Year | Key Topics |

|:------|:-------------|:-----|:-----------|

| [The Double AI Agent](http://i.blackhat.com/EU-24/Presentations/EU-24-Cohen-TheDoubleAIAgent.pdf) | Black Hat EU | 2024 | - Agent manipulation techniques
- Double agent scenarios
- Defense strategies |

| [Mind the Data Gap](http://i.blackhat.com/EU-24/Presentations/EU-24-Pappu-Mind-the-Data-Gap.pdf) | Black Hat EU | 2024 | - Privacy in AI agents
- Multi-agent systems
- Security controls |

| [Agentic AI: New Frontier](https://aisuf.org/blogs/f/agentic-ai-a-new-frontier-in-security) | AI Security Union | 2024 | - Security implications
- Future trends
- Risk analysis |

| [Agent Security Analysis](https://arxiv.org/abs/2409.10737) | arXiv | 2023 | - Security frameworks
- Vulnerability assessment
- Protection measures |

| [Agent Behavior Study](https://arxiv.org/pdf/2409.03793) | arXiv | 2023 | - Behavioral analysis
- Attack patterns
- Security recommendations |



### Community Resources

- [OWASP AI Agent Security Project](https://github.com/precize/OWASP-Agentic-AI/) - Official repository

- [OWASP Slack #team-llm-autonomus-agents](https://owasp.slack.com/archives/team-llm-autonomus-agents) - Community discussions