Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

Awesome-Agentic-Engineering

A curated map of agentic AI systems — covering architectures, frameworks, memory, evaluation, and safety. 🌟 Star if you like it!
https://github.com/natnew/Awesome-Agentic-Engineering

Last synced: 16 days ago
JSON representation

  • 🛂 Agent Authority, Identity & Delegation

  • 🧪 Evaluation & Safety

    • AI Guardrails

    • Benchmarks

      • SWE-bench - agent benchmark grounded in real GitHub issues and patches; `Verified` subset is the canonical agent workload. | `[official]` · `[benchmark]` |
      • AgentBench - environment LLM agent benchmark covering OS, DB, web, and game tasks. | `[official]` · `[benchmark]` |
      • GAIA - world multi-step tasks and tool use. | `[official]` · `[benchmark]` |
      • WebArena / VisualWebArena - website snapshots; visual variant tests multimodal web agents. | `[official]` · `[benchmark]` |
      • τ-bench - use + user-simulation benchmark measuring agent reliability and consistency across trials. | `[official]` · `[benchmark]` |
      • LLMs Get Lost in Evolving User Intent - turn conversations with argument reveals, revisions, function switches, and combined transitions; supports GSM8K, BIRD-SQL, BrowseComp+, and SWE-bench Verified. | `[official]` · `[benchmark]` [paper](https://arxiv.org/abs/2607.20734) |
      • OSWorld - use benchmark for multimodal agents on real desktop OS tasks across Ubuntu/Windows/macOS; complements web-only benchmarks. | `[official]` · `[benchmark]` [paper](https://arxiv.org/abs/2404.07972) |
      • WebVoyager - agent benchmark on live production websites (not snapshots); tests multimodal browsing under real network and UI drift conditions. | `[official]` · `[benchmark]` [paper](https://arxiv.org/abs/2401.13919) |
      • ClawBench - agent benchmark on live production websites with isolated runs, final-request interception, and five-layer execution evidence. | `[official]` · `[benchmark]` [paper](https://arxiv.org/abs/2604.08523) |
      • DataPrep-Bench - data preparation workflows, adding a data-centric workload for comparing multi-stage preparation capabilities beyond conventional task-execution benchmarks. | `[official]` · `[benchmark]` [paper](https://arxiv.org/search/?query=DataPrep-Bench%3A+Benchmarking+LLMs+as+Training+Data+Preparators&searchtype=title) |

    • Evaluation Frameworks

      • OpenAI Evals
      • DeepEval - source LLM evaluation framework with metrics for hallucination, answer relevance, and task completion. | Application-level evaluation and regression testing. | `[official]` |
      • promptfoo
      • Inspect - graded scoring. | Rigorous research-grade and agent-task evaluation. | `[official]` · `[benchmark]` |
      • ai-evaluation - source LLM evaluation framework with 50+ metrics, LLM-as-Judge, and guardrail scanners (jailbreak, PII, injection). | Application-level evaluation, regression testing, and guardrail scanning. | `[official]` |
      • agent-opt
      • simulate-sdk - and scenario-driven SDK for simulating voice and text AI agents. | Pre-production agent simulation and stress testing. | `[official]` |

    • Safety Tooling & Methodologies

    • Tracing and Monitoring

      • Langfuse
      • Arize Phoenix
      • Helicone - line integration. |
      • traceAI - native tracing for LLM and agent apps with 50+ framework integrations. |
      • agenttrace - first TUI and report generator for auditing AI coding-agent session logs, token and cost usage, latency, tool failures, health, and CI regression checks. |
      • Future AGI - hostable end-to-end agent engineering platform with tracing, evals, guardrails, and gateway. |

  • 🧠 Memory Systems

  • ⚙️ Orchestration Frameworks

    • Deep Dives

    • Frameworks Landscape

      • LangChain
      • LangGraph - based orchestration. Stateful typed-state graphs with checkpointing. | `[official]` |
      • LlamaIndex - centric framework for retrieval-heavy and RAG-oriented agent systems. | `[official]` |
      • Haystack - based framework for search, retrieval, and hybrid agent workflows. | `[official]` |
      • Semantic Kernel
      • Pydantic AI - safe, Pydantic-native; schema-first LLM interactions with dependency injection. | `[official]` |
      • DSPy
      • AutoGen - agent conversations; v0.4 redesigned for async event-driven execution. | `[official]` · `[benchmark]` |
      • CrewAI - based crew members with goals, tools, and sequential/hierarchical processes. | `[official]` |
      • OpenAI Agents SDK - step agents with handoffs, guardrails, sessions, and tracing. | `[official]` |
      • Google ADK - agent orchestration; deploys to Vertex AI Agent Engine. | `[official]` |
      • MetaGPT - oriented. | `[official]` · `[benchmark]` |
      • CAMEL - based simulation and collaborative reasoning research framework. | `[official]` · `[benchmark]` |
      • DeerFlow
      • Smolagents - action agents with sandboxed execution. | `[official]` |
      • Agno - agnostic agent framework with native multi-modal support. | `[official]` |
      • Upsonic - first framework with minimal setup and typed task graphs. | `[official]` |
      • Mastra - first framework with observability, workflows, and memory. | `[official]` |

  • 📡 Protocols and Standards

    • Frameworks Landscape

      • MCP (Model Context Protocol) - authored open standard for exposing tools, resources, prompts, and sampling to LLM hosts; wide multi-vendor adoption in 2025–2026. | `[official]` [spec](https://spec.modelcontextprotocol.io/) |
      • A2A (Agent2Agent) - originated, Linux Foundation–hosted protocol for secure cross-agent communication across vendors and frameworks. | `[official]` [spec](https://a2a-protocol.org/) |
      • OpenAI Function / Tool Calling - schema-typed tool definitions. | `[official]` |
      • OpenAPI - standard HTTP API specification; foundation for typed, discoverable tool surfaces behind MCP or direct function-calling. | `[official]` |

  • 🧭 Reasoning & Planning Models

    • Resources

      • official - 5.4`](https://developers.openai.com/api/docs/models/gpt-5.4) · [`mini`](https://developers.openai.com/api/docs/models/gpt-5.4-mini) · [`nano`](https://developers.openai.com/api/docs/models/gpt-5.4-nano) |
      • official - with-claude/extended-thinking) |
      • official
      • official
      • official
      • official
      • benchmark

  • 📐 Spec-Driven Development

    • Resources

      • GitHub Spec Kit - source toolkit for spec-driven development with agentic coding assistants (Copilot, Claude Code, Cursor, Gemini CLI). Defines the `/specify` → `/plan` → `/tasks` → `/implement` workflow used in this repo's own `specs/` directory. | `[official]` [repo](https://github.com/github/spec-kit) · `[official]` [announce](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/) |
      • OpenAI Model Spec - party example of treating *model behaviour* as a versioned, public spec — objectives, rules, defaults, and conflict resolution. A reference for how to write a spec an agent can actually be aligned to. | `[official]` [spec](https://model-spec.openai.com/) · `[official]` [post](https://openai.com/index/sharing-the-latest-model-spec/) |
      • AGENTS.md - level agent spec | Simple convention for a repository-scoped file that instructs coding agents about build, test, style, and conventions. Widely supported across agent CLIs. | `[official]` [site](https://agents.md/) |
      • Anthropic Claude Skills (SKILL.md) - level spec | Declarative, self-contained skill specs (`SKILL.md`) that package instructions, tools, and examples agents can discover and load on demand. Treats individual capabilities as versioned spec artifacts. | `[official]` [docs](https://docs.claude.com/en/docs/agents-and-tools/agent-skills/overview) |
      • Harness Handbook - centric maps from agent-harness behaviours to source locations, with progressive disclosure for planning edits across evolving harness codebases. | `[benchmark]` [paper](https://arxiv.org/abs/2607.13285) |
      • Pydantic AI - first I/O the default for LLM calls — the practical form of "typed contracts at boundaries" for agent code. | `[official]` [docs](https://ai.pydantic.dev/) |
Programming Languages
Python 33 TypeScript 6 Jupyter Notebook 2 Astro 1 MDX 1 Go 1 Shell 1 Markdown 1 C# 1
Categories
🧪 Evaluation & Safety 33 ⚙️ Orchestration Frameworks 23 🧭 Reasoning & Planning Models 7 📐 Spec-Driven Development 6 📡 Protocols and Standards 4 🧠 Memory Systems 3 🛂 Agent Authority, Identity & Delegation 3 Related Projects 3
Sub Categories
Frameworks Landscape 22 Resources 16 Benchmarks 10 AI Guardrails 7 Evaluation Frameworks 7 Safety Tooling & Methodologies 6 Tracing and Monitoring 6 Deep Dives 5 Frontier Research: Memory Beyond Storage 3
Keywords
llm 18 ai 16 agents 13 ai-agents 9 openai 9 agent 8 python 8 llmops 7 chatgpt 6 framework 6 langchain 6 multi-agent 5 prompt-engineering 5 llm-evaluation 5 large-language-models 5 rag 5 open-source 4 agent-framework 4 agentic-ai 4 agentic 4 generative-ai 4 llms 4 genai 3 mcp 3 awesome-list 3 llm-observability 3 harness 3 evaluation 3 gpt 3 artificial-intelligence 3 evals 3 llm-agent 3 typescript 3 langgraph 3 pydantic 3 sdk 2 workflows 2 llm-framework 2 agentic-agi 2 nodejs 2 orchestration 2 multi-agent-systems 2 natural-language-processing 2 aiagentframework 2 ai-observability 2 llm-security 2 gpt-4 2 security 2 multiagent 2 llamaindex 2