{"id":45974483,"url":"https://github.com/fswair/vowel","last_synced_at":"2026-02-28T16:03:21.293Z","repository":{"id":324258546,"uuid":"1095197777","full_name":"fswair/vowel","owner":"fswair","description":"YAML Based Eval Specification Language for LLMs and Developers.","archived":false,"fork":false,"pushed_at":"2026-02-23T19:35:48.000Z","size":253,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-24T02:03:54.873Z","etag":null,"topics":["evals","llms","pydantic-evals","specification","yaml"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/vowel/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fswair.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"patreon":"venus_ai"}},"created_at":"2025-11-12T18:11:42.000Z","updated_at":"2026-02-23T19:34:13.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/fswair/vowel","commit_stats":null,"previous_names":["fswair/vowel"],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/fswair/vowel","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fswair%2Fvowel","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fswair%2Fvowel/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fswair%2Fvowel/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fswair%2Fvowel/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fswair","download_url":"https://codeload.github.com/fswair/vowel/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fswair%2Fvowel/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29941802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-28T13:49:17.081Z","status":"ssl_error","status_checked_at":"2026-02-28T13:48:50.396Z","response_time":90,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["evals","llms","pydantic-evals","specification","yaml"],"created_at":"2026-02-28T16:03:20.700Z","updated_at":"2026-02-28T16:03:21.278Z","avatar_url":"https://github.com/fswair.png","language":"Python","funding_links":["https://patreon.com/venus_ai"],"categories":[],"sub_categories":[],"readme":"# VOWEL\n\n**YAML-based evaluation framework for testing Python functions with AI-powered test generation, function healing and TDD approach.**\n\nvowel makes it easy to define test cases in YAML and run them against your Python functions. It also provides AI-powered generators that can automatically create test specs, generate implementations, and fix buggy functions.\n\n[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/fswair/vowel) [![zread](https://img.shields.io/badge/Ask_Zread-_.svg?style=flat\u0026color=00b0aa\u0026labelColor=000000\u0026logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB3aWR0aD0iMTYiIGhlaWdodD0iMTYiIHZpZXdCb3g9IjAgMCAxNiAxNiIgZmlsbD0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPHBhdGggZD0iTTQuOTYxNTYgMS42MDAxSDIuMjQxNTZDMS44ODgxIDEuNjAwMSAxLjYwMTU2IDEuODg2NjQgMS42MDE1NiAyLjI0MDFWNC45NjAxQzEuNjAxNTYgNS4zMTM1NiAxLjg4ODEgNS42MDAxIDIuMjQxNTYgNS42MDAxSDQuOTYxNTZDNS4zMTUwMiA1LjYwMDEgNS42MDE1NiA1LjMxMzU2IDUuNjAxNTYgNC45NjAxVjIuMjQwMUM1LjYwMTU2IDEuODg2NjQgNS4zMTUwMiAxLjYwMDEgNC45NjE1NiAxLjYwMDFaIiBmaWxsPSIjZmZmIi8%2BCjxwYXRoIGQ9Ik00Ljk2MTU2IDEwLjM5OTlIMi4yNDE1NkMxLjg4ODEgMTAuMzk5OSAxLjYwMTU2IDEwLjY4NjQgMS42MDE1NiAxMS4wMzk5VjEzLjc1OTlDMS42MDE1NiAxNC4xMTM0IDEuODg4MSAxNC4zOTk5IDIuMjQxNTYgMTQuMzk5OUg0Ljk2MTU2QzUuMzE1MDIgMTQuMzk5OSA1LjYwMTU2IDE0LjExMzQgNS42MDE1NiAxMy43NTk5VjExLjAzOTlDNS42MDE1NiAxMC42ODY0IDUuMzE1MDIgMTAuMzk5OSA0Ljk2MTU2IDEwLjM5OTlaIiBmaWxsPSIjZmZmIi8%2BCjxwYXRoIGQ9Ik0xMy43NTg0IDEuNjAwMUgxMS4wMzg0QzEwLjY4NSAxLjYwMDEgMTAuMzk4NCAxLjg4NjY0IDEwLjM5ODQgMi4yNDAxVjQuOTYwMUMxMC4zOTg0IDUuMzEzNTYgMTAuNjg1IDUuNjAwMSAxMS4wMzg0IDUuNjAwMUgxMy43NTg0QzE0LjExMTkgNS42MDAxIDE0LjM5ODQgNS4zMTM1NiAxNC4zOTg0IDQuOTYwMVYyLjI0MDFDMTQuMzk4NCAxLjg4NjY0IDE0LjExMTkgMS42MDAxIDEzLjc1ODQgMS42MDAxWiIgZmlsbD0iI2ZmZiIvPgo8cGF0aCBkPSJNNCAxMkwxMiA0TDQgMTJaIiBmaWxsPSIjZmZmIi8%2BCjxwYXRoIGQ9Ik00IDEyTDEyIDQiIHN0cm9rZT0iI2ZmZiIgc3Ryb2tlLXdpZHRoPSIxLjUiIHN0cm9rZS1saW5lY2FwPSJyb3VuZCIvPgo8L3N2Zz4K\u0026logoColor=ffffff)](https://zread.ai/fswair/vowel) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/vowel?period=total\u0026units=INTERNATIONAL_SYSTEM\u0026left_color=BLACK\u0026right_color=GREEN\u0026left_text=downloads)](https://pepy.tech/projects/vowel)\n\n## Installation\n\n```bash\npip install vowel\n\n# Or with uv\nuv add vowel\n```\n\n## Optional Dependencies\n\nVowel supports several optional dependency groups for enhanced functionality:\n\n| Group         | Install Command                        | Purpose / Extras                |\n|---------------|---------------------------------------|---------------------------------|\n| **all**       | `pip install vowel[all]`              | All optional features           |\n| **dev**       | `pip install vowel[dev]`              | Development \u0026 testing tools     |\n| **mcp**       | `pip install vowel[mcp]`              | MCP server   |\n| **optimization** | `pip install vowel[optimize]`      | Performance optimizations       |\n| **monty**     | `pip install vowel[monty]`            | Monty runtime support     |\n| **logfire**   | `pip install vowel[logfire]`          | Logfire integration             |\n\n\u003e **Tip:**  \n\u003e You can install multiple extras at once, e.g.  \n\u003e `pip install vowel[dev,mcp]`\n\u003e Recommended: `pip install vowel[all]`\n\n### Development\n\n```bash\ngit clone https://github.com/fswair/vowel.git\ncd vowel\npip install -e \".[all]\"\n```\n\n---\n\n## Quick Start\n\n\u003e **Note:**  \n\u003e For a deeper understanding of how vowel handles fixtures, see the examples in [`db_fixture.yml`](./db_fixture.yml) and [`db.py`](./db.py). These files demonstrate the underlying mechanics of fixture setup and usage.\n\n\u003e **Tip:**  \n\u003e To enable YAML schema validation in your editor, place `vowel-schema.json` in your project directory.  \n\u003e Then, add the following directive at the top of your YAML file to activate schema support and instructions:\n\u003e\n\u003e ```yaml\n\u003e # yaml-language-server: $schema=\u003cpath/to/vowel-schema.json\u003e\n\u003e ```\n\u003e\n\u003e Replace `\u003cpath/to/vowel-schema.json\u003e` with the actual path to your schema file.\n\n### 1. Create a YAML spec\n\n```yaml\n# evals.yml\nadd:\n  dataset:\n    - case:\n        inputs: { x: 1, y: 2 }\n        expected: 3\n    - case:\n        inputs: { x: -5, y: 5 }\n        expected: 0\n\ndivide:\n  evals:\n    Type:\n      type: \"float\"\n  dataset:\n    - case:\n        inputs: { a: 10, b: 2 }\n        expected: 5.0\n    - case:\n        inputs: { a: 1, b: 0 }\n        raises: ZeroDivisionError\n```\n\n### 2. Run from CLI\n\n```bash\nvowel evals.yml\n```\n\n### 3. Or programmatically\n\n```python\nfrom vowel import run_evals\n\ndef add(x: int, y: int) -\u003e int:\n    return x + y\n\ndef divide(a: float, b: float) -\u003e float:\n    return a / b\n\nsummary = run_evals(\"evals.yml\", functions={\"add\": add, \"divide\": divide})\nprint(f\"All passed: {summary.all_passed}\")\nprint(f\"Coverage: {summary.coverage * 100:.1f}%\")\n```\n\n### 4. Or use the fluent API\n\n```python\nfrom vowel import RunEvals\n\nsummary = (\n    RunEvals.from_file(\"evals.yml\")\n    .with_functions({\"add\": add, \"divide\": divide})\n    .filter([\"add\"])\n    .debug()\n    .run()\n)\n\nsummary.print()\n```\n\n---\n\n## Features\n\n### Evaluators\n\n8 built-in evaluators for flexible testing:\n\n| Evaluator | Purpose |\n|-----------|---------|\n| **Expected** | Exact value matching |\n| **Type** | Return type checking (strict/lenient) |\n| **Assertion** | Custom Python expressions (`output \u003e 0`, `output == input * 2`) |\n| **Duration** | Performance constraints (function-level \u0026 case-level) |\n| **Pattern** | Regex validation on output |\n| **ContainsInput** | Verify output contains the input |\n| **Raises** | Exception class + optional message matching |\n| **LLMJudge** | AI-powered rubric evaluation |\n\n```yaml\nfactorial:\n  evals:\n    Assertion:\n      assertion: \"output \u003e 0\"\n    Type:\n      type: \"int\"\n    Duration:\n      duration: 1.0\n  dataset:\n    - case: { input: 0, expected: 1 }\n    - case: { input: 5, expected: 120 }\n```\n\n\u003e **Full reference:** [docs/EVALUATORS.md](https://github.com/fswair/vowel/blob/main/docs/EVALUATORS.md)\n\n### Fixtures (Dependency Injection)\n\nInject databases, temp files, caches into functions under test. Three patterns: generator (yield), tuple (setup/teardown), simple (setup only).\n\n```yaml\nfixtures:\n  db:\n    setup: myapp.setup_db\n    teardown: myapp.close_db\n    scope: module\n\nquery_user:\n  fixture: [db]\n  dataset:\n    - case:\n        inputs: { user_id: 1 }\n        expected: { name: \"Alice\" }\n```\n\n```python\ndef query_user(user_id: int, *, db: dict) -\u003e dict | None:\n    return db[\"users\"].get(user_id)\n```\n\n\u003e **Full reference:** [docs/FIXTURES.md](https://github.com/fswair/vowel/blob/main/docs/FIXTURES.md)\n\n### Input Serializers\n\nTransform YAML inputs into Pydantic models, dates, or custom types:\n\n```python\nsummary = (\n    RunEvals.from_file(\"evals.yml\")\n    .with_functions({\"get_user\": get_user})\n    .with_serializer({\"get_user\": User})      # Schema mode\n    .run()\n)\n```\n\n\u003e **Full reference:** [docs/SERIALIZERS.md](https://github.com/fswair/vowel/blob/main/docs/SERIALIZERS.md)\n\n### AI-Powered Generation\n\n#### EvalGenerator — test existing functions\n\n```python\nfrom vowel import EvalGenerator, Function\n\ngenerator = EvalGenerator(model=\"openai:gpt-4o\", load_env=True)\nfunc = Function.from_callable(my_function)\n\nresult = generator.generate_and_run(func, auto_retry=True, heal_function=True)\nprint(f\"Coverage: {result.summary.coverage * 100:.1f}%\")\n```\n\n#### TDDGenerator — generate everything from a description\n\n```python\nfrom vowel.tdd import TDDGenerator\n\ngenerator = TDDGenerator(model=\"gemini-3-flash-preview\", load_env=True)\n\nresult = generator.generate_all(\n    description=\"Binary search for target in sorted list. Returns index or -1.\",\n    name=\"binary_search\"\n)\n\nresult.print()  # Shows: signature → tests → code → results\n```\n\nStep-by-step control:\n\n```python\ndescription = \"Calculate factorial of a non-negative integer\"\nsignature = generator.generate_signature(description=description, name=\"factorial\")\nrunner, yaml_spec = generator.generate_evals_from_signature(signature, description=description)\nfunc = generator.generate_implementation(signature, yaml_spec, description=description)\nsummary = runner.with_functions({\"factorial\": func.impl}).run()\n```\n\n\u003e **Full reference:** [docs/AI_GENERATION.md](https://github.com/fswair/vowel/blob/main/docs/AI_GENERATION.md)\n\n### MCP Server\n\nExpose vowel's capabilities to AI assistants like Claude Desktop via Model Context Protocol.\n\n\u003e **Setup guide:** [docs/MCP.md](https://github.com/fswair/vowel/blob/main/docs/MCP.md)\n\n---\n\n## CLI\n\n```bash\nvowel evals.yml                          # Run single file\nvowel -d ./tests                         # Run directory\nvowel evals.yml -f add,divide            # Filter functions\nvowel evals.yml --ci --cov 90         # CI mode\nvowel evals.yml --watch                  # Watch mode\nvowel evals.yml --dry-run                # Show plan without running\nvowel evals.yml --export-json out.json   # Export results\nvowel evals.yml -v                       # Verbose summary\nvowel evals.yml -v --hide-report         # Verbose, hide pydantic_evals report\n```\n\n\u003e **Full reference:** [docs/CLI.md](https://github.com/fswair/vowel/blob/main/docs/CLI.md)\n\n---\n\n## EvalSummary\n\n```python\nsummary = run_evals(\"evals.yml\", functions={...})\n\nsummary.all_passed       # bool\nsummary.success_count    # int\nsummary.failed_count     # int\nsummary.total_count      # int\nsummary.coverage         # float (0.0-1.0)\nsummary.failed_results   # list[EvalResult]\n\nsummary.meets_coverage(0.9)    # Check threshold\nsummary.print()                # Rich formatted output\nsummary.to_json()              # Export as dict\nsummary.xml()                  # Export as XML\n```\n\n---\n\n## Documentation\n\n| Document | Description |\n|----------|-------------|\n| [YAML Spec](https://github.com/fswair/vowel/blob/main/docs/YAML_SPEC.md) | Complete YAML format reference |\n| [Evaluators](https://github.com/fswair/vowel/blob/main/docs/EVALUATORS.md) | All 8 evaluator types |\n| [Fixtures](https://github.com/fswair/vowel/blob/main/docs/FIXTURES.md) | Dependency injection guide |\n| [Serializers](https://github.com/fswair/vowel/blob/main/docs/SERIALIZERS.md) | Input serializer patterns |\n| [AI Generation](https://github.com/fswair/vowel/blob/main/docs/AI_GENERATION.md) | EvalGenerator \u0026 TDDGenerator |\n| [CLI](https://github.com/fswair/vowel/blob/main/docs/CLI.md) | Command-line reference |\n| [MCP Server](https://github.com/fswair/vowel/blob/main/docs/MCP.md) | AI assistant integration |\n| [Troubleshooting](https://github.com/fswair/vowel/blob/main/docs/TROUBLESHOOTING.md) | Common errors \u0026 solutions |\n\n---\n\n## License\n\nApache License 2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffswair%2Fvowel","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffswair%2Fvowel","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffswair%2Fvowel/lists"}