{"id":14360998,"url":"https://github.com/poyro/poyro","last_synced_at":"2025-08-22T03:30:32.395Z","repository":{"id":246388202,"uuid":"812796837","full_name":"poyro/poyro","owner":"poyro","description":"Test your web app LLM integrations using existing testing frameworks. Confidently launch AI-driven webapps to production.","archived":false,"fork":false,"pushed_at":"2025-01-13T20:42:50.000Z","size":1940,"stargazers_count":31,"open_issues_count":13,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-13T21:33:38.496Z","etag":null,"topics":["ai","evaluation","llm","llmops","nodejs","prompt","prompt-engineering","prompts","testing","vitest"],"latest_commit_sha":null,"homepage":"https://docs.poyro.dev","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/poyro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-09T22:37:32.000Z","updated_at":"2024-12-09T16:20:52.000Z","dependencies_parsed_at":"2024-09-09T19:13:15.006Z","dependency_job_id":"09ea2e78-9493-405d-b8aa-912c1203b567","html_url":"https://github.com/poyro/poyro","commit_stats":null,"previous_names":["poyro/poyro"],"tags_count":66,"template":false,"template_full_name":null,"purl":"pkg:github/poyro/poyro","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poyro%2Fpoyro","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poyro%2Fpoyro/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poyro%2Fpoyro/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poyro%2Fpoyro/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/poyro","download_url":"https://codeload.github.com/poyro/poyro/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poyro%2Fpoyro/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271579344,"owners_count":24784231,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","evaluation","llm","llmops","nodejs","prompt","prompt-engineering","prompts","testing","vitest"],"created_at":"2024-08-27T16:00:59.207Z","updated_at":"2025-08-22T03:30:31.945Z","avatar_url":"https://github.com/poyro.png","language":"TypeScript","funding_links":[],"categories":["Evaluation","ai","Plugins"],"sub_categories":["Framework-agnostic Plugins"],"readme":"# Poyro\n\n[![npm version](https://badge.fury.io/js/%40poyro%2Fvitest.svg)](https://badge.fury.io/js/%40poyro%2Fvitest) | [![docs](https://img.shields.io/badge/poyro-docs-blue)](https://docs.poyro.dev) | [![Discord](https://img.shields.io/discord/1250274948233039883?logo=discord\u0026label=Discord)](https://discord.gg/gmCjjJ5jSf)\n\n## What is Poyro?\n\nPoyro is a lightweight extension of [Vitest](https://vitest.dev/) (a modern testing framework with a Jest-like API) for testing LLM applications. Familiarizing yourself with Vitest can help you get more out of Poyro.\n\nPoyro follows the best practice of using LLMs to evaluate the outputs of your application LLM. Poyro converts natural language conditions into binary unit tests. Unit testing should be free and open source by default -- that includes AI apps -- so we [use a locally run small LLM](https://docs.poyro.dev/how-does-it-work) to evaluate your application's outputs.\n\nFor more context on why to use Poyro and examples on how to use it along with Vitest to unit test LLM apps, read our essays:\n\n- [AI Testing as part of Full-Stack Engineering](https://docs.poyro.dev/essays/ai-testing-as-part-of-fullstack-engineering)\n- [How To Write Unit Tests for Your AI Web App](https://docs.poyro.dev/essays/how-to-write-unit-tests-for-ai-web-app)\n- [Case Study - Unit Testing a Legal AI App](https://docs.poyro.dev/essays/unit-testing-a-legal-ai-app)\n  - [Runnable code examples for this article](https://github.com/poyro/unit-test-legal-ai-demo)\n\n## Prerequisites\n\n- Node.js 20 or later\n- Vitest 1.6.0 or later\n- Your project must have \"type\": \"module\" in its package.json (this library is ESM only)\n- Recommended: CUDA-compatible GPU (Nvidia) or Metal-compatible GPU (Apple Silicon) for best performance, but not required\n\n---\n\n## Usage\n\n### Install\n\nTo get started quickly, simply run:\n\n```bash\nnpx poyro init\n```\n\n### Create your first test\n\nTo use the matchers, create a file with the extension `.test.ts` anywhere within your project. Here is an example:\n\n```javascript\n// MyFirstTest.test.ts\nimport { describe, expect, it } from \"vitest\";\n\ndescribe(\"MyFirstTest\", () =\u003e {\n  it(\"should be true\", async () =\u003e {\n    // Replace this with your LLM app's output\n    const llmOutput = \"Hello, world!\";\n\n    // Criterion in natural language: True or False\n    await expect(llmOutput).toFulfillCriterion(\"Says hello\");\n  }, 10000); // Increase the timeout as needed with the third argument\n});\n```\n\n`toFulfillCriterion` can determine whether an LLM output meets or does not meet a natural language criterion.\n\nThe first time you run a test like the one above, the model file for our locally run LLM will be downloaded. It should take a couple of minutes.\n\n### Run your test\n\nRun vitest with your package manager, for example with `npm`:\n\n```bash\nnpm test\n```\n\nSimilar commands work with `yarn` and `pnpm`.\n\n---\n\n## Help\n\n### Troubleshooting\n\nWe provide solutions to common issues in our [Troubleshooting](https://docs.poyro.dev/troubleshooting) page. Take a look at the solutions there, and if you continue to run into problems please don't hesitate to reach out to us.\n\n### Submit an issue\n\nYou can submit an issue by going to our GitHub repository and [creating a new issue](https://github.com/poyro/poyro/issues/new). Prior to submitting an issue, please check if a similar issue has already been submitted.\n\n### Join the community\n\nIf you want to get help, brainstorm on good evals, or if you just want to chat with other Poyro developers, join our community on [Discord](https://discord.gg/gmCjjJ5jSf)!\n\n\n## Contributing\n\nIf you would like to contribute to Poyro, please read our [contributing guide](CONTRIBUTING.md).\n\nBegin by cloning the repository:\n\n```bash\ngit clone https://github.com/poyro/poyro.git\n```\n\nNext, install all the dependencies:\n\n```bash\npnpm install\n```\n\nWhen developing, you can either run `pnpm dev` at the root, which will automatically build all relevant packages when changes are made, or you can run `pnpm dev` within the package you are working on.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoyro%2Fpoyro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpoyro%2Fpoyro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoyro%2Fpoyro/lists"}