{"id":23016225,"url":"https://github.com/firstbatchxyz/function-calling-eval","last_synced_at":"2025-04-02T18:13:30.463Z","repository":{"id":263440710,"uuid":"889899817","full_name":"firstbatchxyz/function-calling-eval","owner":"firstbatchxyz","description":"The DPAB-α Benchmark","archived":false,"fork":false,"pushed_at":"2025-01-15T18:30:04.000Z","size":1867,"stargazers_count":16,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-08T08:34:55.384Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/firstbatchxyz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-17T14:28:49.000Z","updated_at":"2025-01-22T08:53:40.000Z","dependencies_parsed_at":"2024-12-11T10:22:14.180Z","dependency_job_id":"f69c3f3f-ad1f-4000-a348-f8d0cf176983","html_url":"https://github.com/firstbatchxyz/function-calling-eval","commit_stats":null,"previous_names":["firstbatchxyz/function-calling-eval"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Ffunction-calling-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Ffunction-calling-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Ffunction-calling-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/firstbatchxyz%2Ffunction-calling-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/firstbatchxyz","download_url":"https://codeload.github.com/firstbatchxyz/function-calling-eval/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246866099,"owners_count":20846496,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-15T11:14:43.677Z","updated_at":"2025-04-02T18:13:30.443Z","avatar_url":"https://github.com/firstbatchxyz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DPAB-α: Dria Pythonic Agent Benchmark\n\nDPAB-α is a comprehensive benchmark designed to evaluate LLMs function calling capabilities through both Pythonic and JSON-based approaches. \nThis benchmark contains 100 synthetically generated and validated problems across different difficulty levels.\n\nEach task has both functions defined in Python and JSON schemas. \nThe benchmark evaluates the model's ability to generate correct function calls based on the given problem description.\n\nPythonic function calling performance often outstrips JSON-based function calling in scenarios that require creative or multi-step solutions, reinforcing the premise that Pythonic function calling can be more natural and powerful.\n\n### Installation\n\n```bash\ngit clone https://github.com/firstbatchxyz/function-calling-eval.git\ncd function-calling-eval\npip install -r requirements.txt\n```\n\n### Usage\n\nBasic usage:\n\n```bash\npython run.py --model anthropic/claude-3.5-sonnet --provider openrouter\n```\n\n### Command Line Arguments\n\n- `--model`: Model identifier (default: \"anthropic/claude-3.5-sonnet\")\n- `--provider`: API provider (default: \"openrouter\")\n- `--strict`: Enable strict evaluation mode (optional)\n- `--show_completion`: Show model completions (default: False)\n- `--mode`: Evaluation mode, either \"json\" or \"pythonic\" (default: \"pythonic\")\n\n#### Providers\n\nDBAP-a supports the following providers:\n\n- `openrouter`: OpenRouter API\n- `lm_studio`: LM Studio\n- `vllm`: Local models via vLLM\n- `ollama`: Local models via Ollama\n\n### Example\n\n```bash\n# Evaluate Claude 3.5 in pythonic mode\npython run.py --model anthropic/claude-3.5-sonnet --provider openrouter --mode pythonic --strict \n\n# Evaluate with JSON mode and show completions\npython run.py --model qwen/qwen-2.5-7b-instruct --provider openrouter --mode json --strict  --show_completion\n```\n\n### Benchmark Structure\n\nEach test case in the benchmark contains:\n- `difficulty`: Easy or hard\n- `function_schema_python`: Python function definitions\n- `function_schema_json`: JSON function schemas\n- `mock_functions`: Implementation with return values\n- `user_query`: Natural language question\n- `checklist`: Validation criteria\n\n## Results\n\nCurrent benchmark results for various models **(strict)**:\n\n| Model Name                      | Pythonic | JSON |\n|---------------------------------|----------|------|\n| **Closed Models**               |          |      |\n| Claude 3.5 Sonnet              | 87       | 45   |\n| o1-preview-2024-09-12           | 55       | 39   |\n| o1-mini-2024-09-12              | 59       | 35   |\n| gpt-4o-2024-11-20              | 60       | 30   |\n| **Open Models**                 |          |      |\n| **\u003e 100B Parameters**           |          |      |\n| DeepSeek V3 (685B)             | 63       | 33   |\n| MiniMax-01                     | 62       | 40   |\n| Llama-3.1-405B-Instruct        | 60       | 38   |\n| **\u003e 30B Parameters**            |          |      |\n| Qwen-2.5-Coder-32b-Instruct    | 68       | 32   |\n| Qwen-2.5-72b-instruct          | 65       | 39   |\n| Llama-3.3-70b-Instruct         | 59       | 40   |\n| QwQ-32b-Preview                | 47       | 21   |\n| **\u003c 20B Parameters**           |          |      |\n| Dria-Agent-a-7B               | 70       | 38   |\n| Qwen2.5-Coder-7B-Instruct      | 44       | 39   |\n| Dria-Agent-a-3B               | 72       | 31   |\n| Qwen2.5-Coder-3B-Instruct      | 26       | 37   |\n| Qwen-2.5-7B-Instruct           | 47       | 34   |\n| Phi-4 (14B)                    | 55       | 35   |\n\nFor details, please refer to our [blog post](https://huggingface.co/blog/andthattoo/dpab-a).\n\n#### Citation\n\nIf you use this benchmark in your research, please cite:\n\n```bibtex\n@misc{Dria-Agent-a,\n      url={https://huggingface.co/blog/andthattoo/dria-agent-a},\n      title={Dria-Agent-a},\n      author={\"andthattoo\", \"Atakan Tekparmak\"}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Ffunction-calling-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffirstbatchxyz%2Ffunction-calling-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffirstbatchxyz%2Ffunction-calling-eval/lists"}