{"id":30438398,"url":"https://github.com/ash-project/evals","last_synced_at":"2025-08-23T04:05:11.439Z","repository":{"id":300859809,"uuid":"1007394108","full_name":"ash-project/evals","owner":"ash-project","description":"Tools for evaluating models against Elixir code, helping us find what works and what doesn't","archived":false,"fork":false,"pushed_at":"2025-08-21T01:43:24.000Z","size":128,"stargazers_count":37,"open_issues_count":6,"forks_count":3,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-08-22T19:50:31.935Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ash-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["zachdaniel","jimsynz"]}},"created_at":"2025-06-23T23:42:08.000Z","updated_at":"2025-08-18T20:57:51.000Z","dependencies_parsed_at":"2025-06-24T01:25:56.351Z","dependency_job_id":"380ef1e6-d117-460e-b0d7-f91256f6c9d9","html_url":"https://github.com/ash-project/evals","commit_stats":null,"previous_names":["ash-project/evals"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ash-project/evals","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ash-project%2Fevals","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ash-project%2Fevals/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ash-project%2Fevals/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ash-project%2Fevals/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ash-project","download_url":"https://codeload.github.com/ash-project/evals/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ash-project%2Fevals/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271740714,"owners_count":24812642,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-08-23T04:02:31.842Z","updated_at":"2025-08-23T04:05:11.429Z","avatar_url":"https://github.com/ash-project.png","language":"Elixir","funding_links":["https://github.com/sponsors/zachdaniel","https://github.com/sponsors/jimsynz"],"categories":["Observability, Evaluation \u0026 Guardrails","Generative AI"],"sub_categories":["How to Join","Development Tools"],"readme":"# Evals\n\nA evaluation tool for testing and comparing AI language models on various coding tasks. This allows you to run structured evaluations, compare model performance with and without usage rules, and generate detailed reports.\n\n## Features\n\n- **Multiple Model Support**: Evaluate and compare different language models side-by-side\n- **Usage Rules Integration**: Test how well models follow specific package usage rules and guidelines\n- **Code Generation \u0026 Validation**: Evaluate models on code writing tasks with automated assertion testing\n- **Flexible Evaluation Options**: Control iterations, debug output, and evaluation scope\n- **Rich Reporting**: Generate summary or detailed reports with performance breakdowns\n- **YAML-Based Test Definitions**: Define evaluations in simple YAML files organized by category\n\n## Roadmap\n\n- For `write_code_and_assert` type, more complex setup tasks where the LLM only needs to generate a subset of a response, not all the code.\n- Different types of evals, like `response_contains`,  `response_doesnt_contain`, and also `llm_judge` where you ask a separate judge LLM if a certain property is attained by the output.\n- The ability to experiment with different system prompts, i.e does \"you are an expert Elixir developer\" matter?\n- The ability to benchmark fully agentic flows like multi-turn working with hex docs search, plan files, custom context etc.\n\n## Report\n\nWe only have a few evals here, but eventually this will be expensive for me to\noperate, so its not running in CI etc. I will run it when I feel like its worth\nrunning again, when the are more evals etc. Others are encouraged to run this\nlocally with their own keys if they want to throw a few coins in the machine to\nhelp out.\n\nSee the [reports folder](reports/) for more.\n\nFor example:\n\n[reports/flagship](reports/flagship.md?plain=1)\n\n## Quick Start\n\n```elixir\n# Define your models\nmodels = [\n  {\"gpt-4\", %LangChain.ChatModels.ChatOpenAI{model: \"gpt-4\"}},\n  {\"claude-3-sonnet\", %LangChain.ChatModels.ChatAnthropic{model: \"claude-3-sonnet-20240229\"}}\n]\n\n# Run evaluations and get a report\n{results, report} = Evals.report(models,\n  usage_rules: :compare,\n  title: \"Model Comparison\",\n  format: \"summary\"\n)\n\nIO.puts(report)\n```\n\n## Common Model Comparisons\n\nThe `Evals.Common` module provides convenient functions for testing common model combinations:\n\n### Flagship Models\n\nCompare the latest flagship models from OpenAI and Anthropic:\n\n```elixir\n# Quick flagship comparison\nreport = Evals.Common.flagship(usage_rules: :compare, format: \"summary\")\nIO.puts(report)\n\n# Full detailed report\nreport = Evals.Common.flagship(usage_rules: :compare, format: :full)\nIO.puts(report)\n```\n\nThis compares:\n- GPT-4.1\n- GPT-4o\n- Claude Sonnet 4\n- Claude Sonnet 3.7\n\n### GPT Models Only\n\nCompare different GPT model variants:\n\n```elixir\nreport = Evals.Common.gpt(usage_rules: :compare)\nIO.puts(report)\n```\n\nThis compares:\n- GPT-4.1\n- GPT-4o\n\nAll `Evals.Common` functions accept the same options as `Evals.report/2` and return the formatted report string directly.\n\n## Contributing Evaluations\n\nWe welcome contributions of new evaluation cases! Here's how to add your own:\n\n### Creating a New Evaluation\n\n1. **Choose a category** or create a new one in the `evals/` directory\n2. **Create a YAML file** with a descriptive name (e.g., `async_genserver.yml`)\n3. **Follow the evaluation format** shown below\n\n### Evaluation Guidelines\n\n- **Be specific**: Test one clear concept or skill per evaluation\n- **Include context**: Provide enough background in the user message\n- **Write clear assertions**: Make sure your test validates the intended behavior\n- **Test edge cases**: Consider boundary conditions and common mistakes\n- **Add realistic scenarios**: Use examples that mirror real-world usage\n\n### Example Contribution\n\n```yaml\n# evals/genserver/async_operations.yml\ntype: write_code_and_assert\nmessages:\n  - type: user\n    text: |\n      Write a function called `add` that adds two numbers. Return just the function, not wrapped in a module\neval:\n  assert:\n    # wrap the answer in a module\n    wrap_in_module: true\n    assertion: \"\u003c%= @module_name %\u003e.add(2, 3) == 5\"\n```\n\n### Testing Your Evaluation\n\nBefore submitting, test your evaluation locally:\n\n```elixir\n# Test only your new evaluation\n{results, report} = Evals.report(models, only: \"evals/your_category/your_eval.yml\")\nIO.puts(report)\n```\n\n## Evaluation Structure\n\nEvaluations are organized in the `evals/` directory by category:\n\n```\nevals/\n├── basic_elixir/\n│   ├── pattern_matching.yml\n│   └── list_operations.yml\n├── ash_framework/\n│   ├── resource_definition.yml\n│   └── changeset_usage.yml\n└── phoenix/\n    ├── controller_actions.yml\n    └── live_view_basics.yml\n```\n\nEach YAML file defines a test case with:\n- **Type**: Currently supports `write_code_and_assert`\n- **Messages**: Conversation history leading to the code generation request\n- **Code**: Optional existing code context\n- **Install**: Package dependencies to install\n- **Eval**: Assertion criteria for validating the generated code\n\n### Example Evaluation File\n\n```yaml\ntype: write_code_and_assert\ninstall:\n  - package: ash\n    version: \"~\u003e 3.0\"\nmessages:\n  - type: user\n    text: \"Create a basic Ash resource for a User with name and email fields\"\neval:\n  assert:\n    wrap_in_module: true\n    assertion: |\n      Code.ensure_loaded(\u003c%= assigns.module_name %\u003e)\n      function_exported?(\u003c%= assigns.module_name %\u003e, :__resource__, 0)\n```\n\n## API Reference\n\n### Core Functions\n\n#### `Evals.evaluate(models, opts \\\\ [])`\n\nRuns evaluations and returns raw results.\n\n**Options:**\n- `:iterations` - Number of runs per test (default: 1). Higher iterations will cause much longer evaluation times due to rate limits\n- `:usage_rules` - `:compare`, `true`, or `false` (default: `false`)\n- `:only` - Limit to specific file pattern\n- `:debug` - Enable debug output\n- `:system_prompt` - Override system prompt\n\n#### `Evals.report(models, opts \\\\ [])`\n\nRuns evaluations and returns formatted report.\n\n**Additional Report Options:**\n- `:title` - Custom report title\n- `:format` - `:summary` or `:full` (default: `:full`)\n\n### Usage Rules\n\nWhen `:usage_rules` is enabled, the framework automatically:\n1. Installs specified packages via `Mix.install`\n2. Locates `usage-rules.md` files in package dependencies\n3. Includes these rules in the system prompt\n4. Compares model performance with and without rules (when `:compare`)\n\n### Example Results\n\n```elixir\nresults = %{\n  {\"gpt-4\", \"ash_framework\", \"resource_definition\", true} =\u003e 0.85,\n  {\"gpt-4\", \"ash_framework\", \"resource_definition\", false} =\u003e 0.72,\n  {\"claude-3-sonnet\", \"ash_framework\", \"resource_definition\", true} =\u003e 0.78,\n  {\"claude-3-sonnet\", \"ash_framework\", \"resource_definition\", false} =\u003e 0.65\n}\n```\n\n## Report Formats\n\n### Summary Format\nShows only model averages, optionally broken down by usage rules:\n\n```\n================================================================================\nModel Performance Comparison\nIterations: 1\n================================================================================\n\nOVERALL SUMMARY:\n----------------------------------------\n\nWith usage rules:\n  gpt-4              | 85.2%\n  claude-3-sonnet    | 82.1%\n\nWithout usage rules:\n  gpt-4              | 72.4%\n  claude-3-sonnet    | 69.8%\n================================================================================\n```\n\n### Full Format\nIncludes detailed breakdown by category and individual tests.\n\n## Setup\n\n1. **Clone the repository:**\n   ```bash\n   git clone \u003crepository-url\u003e\n   cd evals\n   ```\n\n2. **Install dependencies:**\n   ```bash\n   mix deps.get\n   ```\n\n3. **Set up your API keys:**\n   ```bash\n   export OPENAI_API_KEY=\"your-openai-key\"\n   export ANTHROPIC_API_KEY=\"your-anthropic-key\"\n   ```\n\n4. **Run evaluations:**\n   ```bash\n   iex -S mix\n   ```\n\n   Then in the IEx console:\n   ```elixir\n   models = [\n     {\"gpt-4\", %LangChain.ChatModels.ChatOpenAI{model: \"gpt-4\"}},\n     {\"claude-3-sonnet\", %LangChain.ChatModels.ChatAnthropic{model: \"claude-3-sonnet-20240229\"}}\n   ]\n\n   {results, report} = Evals.report(models, usage_rules: :compare)\n   IO.puts(report)\n   ```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fash-project%2Fevals","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fash-project%2Fevals","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fash-project%2Fevals/lists"}