{"id":18602975,"url":"https://github.com/devlooped/ai.benchmarks","last_synced_at":"2025-11-02T10:30:26.899Z","repository":{"id":261327424,"uuid":"883949479","full_name":"devlooped/AI.Benchmarks","owner":"devlooped","description":"Benchmark different AI services that can be accessed using Microsoft.Extensions.AI","archived":false,"fork":false,"pushed_at":"2025-02-10T22:40:09.000Z","size":60,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-02-13T17:57:21.964Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/devlooped.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"devlooped"}},"created_at":"2024-11-05T21:34:02.000Z","updated_at":"2025-02-10T22:40:11.000Z","dependencies_parsed_at":"2024-11-06T00:36:22.488Z","dependency_job_id":null,"html_url":"https://github.com/devlooped/AI.Benchmarks","commit_stats":null,"previous_names":["devlooped/ai.benchmarks"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlooped%2FAI.Benchmarks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlooped%2FAI.Benchmarks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlooped%2FAI.Benchmarks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/devlooped%2FAI.Benchmarks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/devlooped","download_url":"https://codeload.github.com/devlooped/AI.Benchmarks/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239394671,"owners_count":19631117,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T02:13:07.957Z","updated_at":"2025-11-02T10:30:26.854Z","avatar_url":"https://github.com/devlooped.png","language":"C#","funding_links":["https://github.com/sponsors/devlooped"],"categories":[],"sub_categories":[],"readme":"# Microsoft.Extensions.AI Benchmarks\n\nUses the new [Microsoft.Extensions.AI](https://devblogs.microsoft.com/dotnet/introducing-microsoft-extensions-ai-preview/) preview \nto compare the relative performance of OpenAI, AzureAI and xAI, using the same client code.\n\nCurrently, comparison includes:\n\n* OpenAI: gpt-4o, gpt-4o-mini\n* AzureAI: gpt-4o, gpt-4o-mini\n* xAI: grok-beta, grok-2-latest\n\n![xAI grok2](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fgithub.com%2Fdevlooped%2FAI.Benchmarks%2Fraw%2Frefs%2Fheads%2Fmain%2Fartifacts%2Fsummary.json\u0026query=%24.xai.mean\u0026logo=x\u0026label=grok2)\n![Azure gpt-4o](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fgithub.com%2Fdevlooped%2FAI.Benchmarks%2Fraw%2Frefs%2Fheads%2Fmain%2Fartifacts%2Fsummary.json\u0026query=%24..aai.mean\u0026label=gpt-4o\u0026logo=data:image/svg+xml;base64,PHN2ZyBpZD0idXVpZC1hZGJkYWU4ZS01YTQxLTQ2ZDEtOGMxOC1hYTczY2RiZmVlMzIiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgd2lkdGg9IjE4IiBoZWlnaHQ9IjE4IiB2aWV3Qm94PSIwIDAgMTggMTgiPjxkZWZzPjxyYWRpYWxHcmFkaWVudCBpZD0idXVpZC0yYTc0MDdhYS1iNzg3LTQ4ZGQtYTk2YS0wZDgxYWI2ZTkzYmIiIGN4PSItNjcuOTgxIiBjeT0iNzkzLjE5OSIgcj0iLjQ1IiBncmFkaWVudFRyYW5zZm9ybT0idHJhbnNsYXRlKC0xNzkzOS4wMyAyMDM2OC4wMjkpIHJvdGF0ZSg0NSkgc2NhbGUoMjUuMDkxIC0zNC4xNDkpIiBncmFkaWVudFVuaXRzPSJ1c2VyU3BhY2VPblVzZSI+PHN0b3Agb2Zmc2V0PSIwIiBzdG9wLWNvbG9yPSIjODNiOWY5IiAvPjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iIzAwNzhkNCIgLz48L3JhZGlhbEdyYWRpZW50PjwvZGVmcz48cGF0aCBkPSJtMCwyLjd2MTIuNmMwLDEuNDkxLDEuMjA5LDIuNywyLjcsMi43aDEyLjZjMS40OTEsMCwyLjctMS4yMDksMi43LTIuN1YyLjdjMC0xLjQ5MS0xLjIwOS0yLjctMi43LTIuN0gyLjdDMS4yMDksMCwwLDEuMjA5LDAsMi43Wk0xMC44LDB2My42YzAsMy45NzYsMy4yMjQsNy4yLDcuMiw3LjJoLTMuNmMtMy45NzYsMC03LjE5OSwzLjIyMi03LjIsNy4xOTh2LTMuNTk4YzAtMy45NzYtMy4yMjQtNy4yLTcuMi03LjJoMy42YzMuOTc2LDAsNy4yLTMuMjI0LDcuMi03LjJaIiBmaWxsPSJ1cmwoI3V1aWQtMmE3NDA3YWEtYjc4Ny00OGRkLWE5NmEtMGQ4MWFiNmU5M2JiKSIgc3Ryb2tlLXdpZHRoPSIwIiAvPjwvc3ZnPg==)\n![OpenAI gpt-4o](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fgithub.com%2Fdevlooped%2FAI.Benchmarks%2Fraw%2Frefs%2Fheads%2Fmain%2Fartifacts%2Fsummary.json\u0026query=%24.oai.mean\u0026logo=openai\u0026label=gpt-4o)\n\nPrompt:\n\n```csharp\nIList\u003cChatMessage\u003e prompt =\n[\n    new ChatMessage(ChatRole.System, \"You are a chatbot inspired by the Hitchhiker's Guide to the Galaxy.\"),\n    new ChatMessage(ChatRole.User, \"What is the meaning of life, the universe, and everything?\"),\n];\n```\n\nResults:\n\n\u003c!-- include artifacts/results/AI.Benchmarks.ModelPerformance-report-github.md --\u003e\n```\n\nBenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.4890/23H2/2023Update/SunValley3)\nIntel Core i9-10900T CPU 1.90GHz, 1 CPU, 20 logical and 10 physical cores\n.NET SDK 9.0.200-preview.0.25057.12\n  [Host]     : .NET 8.0.13 (8.0.1325.6609), X64 RyuJIT AVX2\n  DefaultJob : .NET 8.0.13 (8.0.1325.6609), X64 RyuJIT AVX2\n\n\n```\n| Method | Client            | Provider | Model         | Mean    | Error    | StdDev   | Median  |\n|------- |------------------ |--------- |-------------- |--------:|---------:|---------:|--------:|\n| **Chat**   | **aai-gpt-4o**        | **Azure AI** | **gpt-4o**        | **1.724 s** | **0.2150 s** | **0.6029 s** | **1.493 s** |\n| **Chat**   | **aai-gpt-4o-mini**   | **Azure AI** | **gpt-4o-mini**   | **1.897 s** | **0.2355 s** | **0.6407 s** | **1.730 s** |\n| **Chat**   | **oai-gpt-4o**        | **OpenAI**   | **gpt-4o**        | **1.678 s** | **0.1188 s** | **0.3408 s** | **1.660 s** |\n| **Chat**   | **oai-gpt-4o-mini**   | **OpenAI**   | **gpt-4o-mini**   | **2.835 s** | **0.2741 s** | **0.7995 s** | **2.629 s** |\n| **Chat**   | **xai-grok-2-latest** | **xAI**      | **grok-2-latest** | **1.629 s** | **0.1254 s** | **0.3676 s** | **1.588 s** |\n| **Chat**   | **xai-grok-beta**     | **xAI**      | **grok-beta**     | **1.678 s** | **0.1107 s** | **0.3194 s** | **1.655 s** |\n\n\u003c!-- artifacts/results/AI.Benchmarks.ModelPerformance-report-github.md --\u003e\n\n## Run locally\n\nYou can trivially update (and optionally send a PR) the results by running the benchmarks locally.\nSimply fork the repository, clone it locally and run the `.\\update.ps1` script from a `pwsh` prompt.\n\nThe only requirement is to have `jq` installed and available in your `PATH`.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevlooped%2Fai.benchmarks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdevlooped%2Fai.benchmarks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdevlooped%2Fai.benchmarks/lists"}