{"id":50315800,"url":"https://github.com/microsoft/state-bench","last_synced_at":"2026-05-29T00:01:21.809Z","repository":{"id":359194514,"uuid":"1235887692","full_name":"microsoft/STATE-Bench","owner":"microsoft","description":"Benchmark AI Agents on Enterprise Workflows","archived":false,"fork":false,"pushed_at":"2026-05-28T19:59:01.000Z","size":2668,"stargazers_count":33,"open_issues_count":7,"forks_count":4,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-28T20:25:53.391Z","etag":null,"topics":["ai","ai-agents","benchmark","benchmark-framework","microsoft"],"latest_commit_sha":null,"homepage":"https://microsoft.github.io/STATE-Bench/leaderboard/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":"SUPPORT.md","governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-11T18:48:50.000Z","updated_at":"2026-05-28T19:52:36.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/microsoft/STATE-Bench","commit_stats":null,"previous_names":["microsoft/state-bench"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/microsoft/STATE-Bench","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSTATE-Bench","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSTATE-Bench/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSTATE-Bench/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSTATE-Bench/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/STATE-Bench/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2FSTATE-Bench/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33630999,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-05-28T02:00:06.440Z","response_time":99,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","ai-agents","benchmark","benchmark-framework","microsoft"],"created_at":"2026-05-29T00:01:00.632Z","updated_at":"2026-05-29T00:01:21.789Z","avatar_url":"https://github.com/microsoft.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# STATE-Bench\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/🏆%20Leaderboard-Coming%20Soon-crimson?style=for-the-badge\" alt=\"Leaderboard\" /\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\" alt=\"License\" /\u003e\n  \u003ca href=\"https://opensource.microsoft.com/blog/2026/05/19/introducing-state-bench-a-benchmark-for-ai-agent-memory/\"\u003e\u003cimg src=\"https://img.shields.io/badge/Blog-Read-blue\" alt=\"Blog\" /\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"RUN_BENCHMARK.md\"\u003eMain Track\u003c/a\u003e \u0026nbsp;·\u0026nbsp; \u003ca href=\"MEMORY_TRACK.md\"\u003eMemory Track\u003c/a\u003e\n\u003c/p\u003e\n\nSTATE-Bench evaluates AI agents on realistic, multi-turn enterprise tasks across **travel**, **customer support**, and **shopping assistant** domains.\n\nEach task gives the agent a task-local sandbox database, domain-specific tools, and a simulated user. To pass a task, the agent must do multi-step reasoning by gathering the right information with domain tools, applying the correct policy, taking actions to update the database to the right final state when needed, and following the required procedure in conversation.\n\n## Overview\n\nSTATE-Bench includes 450 challenging enterprise tasks across three domains.\n\n| Domain | Tasks | Description |\n| --- | ---: | --- |\n| **Travel** | 150 | Flight, hotel, and car rental bookings; cancellations, updates, fee and policy reasoning, cross-product trip planning |\n| **Customer Support** | 150 | Returns, refunds, exchanges, warranty claims, cancellations, shipping issues, and order changes |\n| **Shopping Assistant** | 150 | Product search, cart updates, applying promos, loyalty redemption, shipping options, and compatibility checks |\n\n## Choose Your Benchmark Track\n\nStart with the track that matches what you want to evaluate. Each track guide links to the setup and reference docs only when you need them.\n\n| Goal | Start here |\n| --- | --- |\n| Evaluate an agent or model directly on the provided enterprise benchmark tasks | **[Main Track](RUN_BENCHMARK.md)** |\n| Evaluate agentic memory | **[Memory Track](MEMORY_TRACK.md)** |\n\nThe **Main Track** is the default benchmark path. The **Memory Track** uses the same simulator, domain tools, judges, and metrics, but adds train trajectories and a retrieval hook for procedural learnings.\n\n\u003cbr/\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/chat_bubble_2.svg\" alt=\"Sample task trajectory from the Travel domain\" width=\"55%\" /\u003e\n  \u003cbr/\u003e\n  \u003cem\u003eSample task trajectory from the Travel domain.\u003c/em\u003e\n\u003c/p\u003e\n\n## Metrics\n\nSTATE-Bench reports four headline metrics:\n\n| Metric | What it measures |\n| --- | --- |\n| **Task Completion pass@1** | Average task completion rate across five runs per task. |\n| **Task Completion pass^5** | Percentage of tasks completed successfully on all five runs. |\n| **UX Score** | LLM-judged conversation quality on a 1-5 scale. |\n| **Cost Per Task** | Average agent cost from user-reported token usage and pricing. |\n\n## License\n\nSTATE-Bench is released under the MIT License. See [LICENSE](LICENSE).\n\n## Trademarks\n\nThis project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark \u0026 Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.\n\n## Disclosures\n\nDatasets provided in this benchmark were synthetically generated using large language models. The benchmark is intended for research purposes and users should exercise caution and consider the limitations of synthetic data when interpreting results.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fstate-bench","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fstate-bench","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fstate-bench/lists"}