{"id":29246662,"url":"https://github.com/acodercat/agent-performance-evaluator","last_synced_at":"2025-07-03T23:06:03.326Z","repository":{"id":302564842,"uuid":"1012860850","full_name":"acodercat/agent-performance-evaluator","owner":"acodercat","description":null,"archived":false,"fork":false,"pushed_at":"2025-07-03T03:33:23.000Z","size":194,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-07-03T04:32:54.188Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/acodercat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-03T02:13:39.000Z","updated_at":"2025-07-03T03:33:26.000Z","dependencies_parsed_at":"2025-07-03T04:33:01.563Z","dependency_job_id":"dc457cc7-4234-4ae3-a315-73635756392c","html_url":"https://github.com/acodercat/agent-performance-evaluator","commit_stats":null,"previous_names":["acodercat/agent-performance-evaluator"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/acodercat/agent-performance-evaluator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acodercat%2Fagent-performance-evaluator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acodercat%2Fagent-performance-evaluator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acodercat%2Fagent-performance-evaluator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acodercat%2Fagent-performance-evaluator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/acodercat","download_url":"https://codeload.github.com/acodercat/agent-performance-evaluator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/acodercat%2Fagent-performance-evaluator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263416473,"owners_count":23463144,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-07-03T23:05:59.911Z","updated_at":"2025-07-03T23:06:03.295Z","avatar_url":"https://github.com/acodercat.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Agent Performance Evaluator\n\nA comprehensive benchmarking framework for evaluating AI agent implementations and their tool-calling capabilities across multiple scenarios.\n\n## Features\n\n- **Multi-Agent Framework Support**: PyCallingAgent, LangChain, LlamaIndex, OpenAI agents\n- **Diverse Scenarios**: Weather queries, flight booking, event planning, healthcare data processing, investment analysis, product recommendations\n- **Comprehensive Metrics**: Success rates, function call accuracy, argument validation, step efficiency\n- **Multi-Model Testing**: GPT-4o, Claude, DeepSeek, Qwen models, and more\n\n## Development\n\n### Adding New Scenarios\n\n1. Create a new scenario file in `scenarios/`\n2. Define your functions with proper type hints\n3. Add test cases to `ground_truths.json`\n4. Update the evaluation pipeline\n\n## Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone \u003crepository-url\u003e\ncd agent-performance-evaluator\n\n# Install dependencies using uv\nuv sync\n```\n\n### Using PyCallingAgent Framework\n\n```python\nimport asyncio\nfrom adapters.py_calling_agent_adapter import PyCallingAgentFactory\nfrom py_calling_agent.models import LiteLLMModel\nfrom evaluation import evaluate\nimport json\n\n# Load test scenarios\nground_truths = json.load(open(\"./scenarios/ground_truths.json\"))\n\n# Create agent factory\nmodel = LiteLLMModel(\n    model_id=\"gpt-4o\",\n    api_key=\"your-api-key\",\n    base_url=\"your-base-url\",\n    custom_llm_provider='openai'\n)\nagent_factory = PyCallingAgentFactory(model)\n\n# Run evaluation\nasync def main():\n    await evaluate(agent_factory, ground_truths, \"./results/results.json\")\n\nasyncio.run(main())\n```\n\n### Using Other ToolCalling Agent Frameworks\n\n```python\n# LangChain Agent\nfrom adapters.langchain_agent_adapter import LangChainAgentFactory\nfrom langchain_openai import ChatOpenAI\n\nllm = ChatOpenAI(model=\"gpt-4o\")\nagent_factory = LangChainAgentFactory(llm)\nawait evaluate(agent_factory, ground_truths, \"./results/langchain_results.json\")\n\n# LlamaIndex Agent\nfrom adapters.llama_index_agent_adapter import LLamaIndexAgentFactory\nfrom llama_index.llms.openai import OpenAI\n\nllm = OpenAI(model=\"gpt-4o\")\nagent_factory = LLamaIndexAgentFactory(llm)\nawait evaluate(agent_factory, ground_truths, \"./results/llama_index_results.json\")\n```\n\n## Project Structure\n\n```\n├── adapters/                 # Agent implementation adapters\n│   ├── py_calling_agent_adapter.py\n│   ├── langchain_agent_adapter.py\n│   ├── llama_index_agent_adapter.py\n│   └── openai_agent_adapter.py\n├── core/                     # Core evaluation logic\n│   ├── evaluator.py         # Main evaluation engine\n│   ├── agent.py             # Agent interface definitions\n│   └── validation.py        # Function call validation\n├── scenarios/               # Test scenarios and ground truths\n│   ├── weather_query.py\n│   ├── flight_booking.py\n│   ├── event_planning.py\n│   └── ground_truths.json\n├── results/                 # Evaluation results\n└── evaluation.py           # Main evaluation runner\n```\n\n## Evaluation Metrics\n\n- **Success Rate**: Percentage of successful turns\n- **Function Call Accuracy**: Missing calls, wrong arguments, type mismatches\n- **Step Efficiency**: Number of steps per conversation\n- **Argument Validation**: Missing, wrong type, or incorrect values\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facodercat%2Fagent-performance-evaluator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Facodercat%2Fagent-performance-evaluator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Facodercat%2Fagent-performance-evaluator/lists"}