Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/poyro/poyro
Test your web app LLM integrations using existing testing frameworks. Confidently launch AI-driven webapps to production.
https://github.com/poyro/poyro
ai evaluation llm llmops nodejs prompt prompt-engineering prompts testing vitest
Last synced: 20 days ago
JSON representation
Test your web app LLM integrations using existing testing frameworks. Confidently launch AI-driven webapps to production.
- Host: GitHub
- URL: https://github.com/poyro/poyro
- Owner: poyro
- Created: 2024-06-09T22:37:32.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-08-26T18:43:37.000Z (21 days ago)
- Last Synced: 2024-08-26T21:59:13.564Z (21 days ago)
- Topics: ai, evaluation, llm, llmops, nodejs, prompt, prompt-engineering, prompts, testing, vitest
- Language: TypeScript
- Homepage: https://docs.poyro.dev
- Size: 1.85 MB
- Stars: 27
- Watchers: 2
- Forks: 0
- Open Issues: 13
-
Metadata Files:
- Readme: README.md
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
- fucking-awesome-vite - @poyro/vitest - Test LLM integrations using Vitest. (Plugins / Framework-agnostic Plugins)
- awesome-vite - @poyro/vitest - Test LLM integrations using Vitest. (Plugins / Framework-agnostic Plugins)
README
# Poyro
[](https://badge.fury.io/js/%40poyro%2Fvitest) | [](https://docs.poyro.dev) | [](https://discord.gg/gmCjjJ5jSf)
## What is Poyro?
Poyro is a lightweight extension of [Vitest](https://vitest.dev/) (a modern testing framework with a Jest-like API) for testing LLM applications. Familiarizing yourself with Vitest can help you get more out of Poyro.
Poyro follows the best practice of using LLMs to evaluate the outputs of your application LLM. Poyro converts natural language conditions into binary unit tests. Unit testing should be free and open source by default -- that includes AI apps -- so we [use a locally run small LLM](https://docs.poyro.dev/how-does-it-work) to evaluate your application's outputs.
For more context on why to use Poyro and examples on how to use it along with Vitest to unit test LLM apps, read our essays:
- [AI Testing as part of Full-Stack Engineering](https://docs.poyro.dev/essays/ai-testing-as-part-of-fullstack-engineering)
- [How To Write Unit Tests for Your AI Web App](https://docs.poyro.dev/essays/how-to-write-unit-tests-for-ai-web-app)
- [Case Study - Unit Testing a Legal AI App](https://docs.poyro.dev/essays/unit-testing-a-legal-ai-app)
- [Runnable code examples for this article](https://github.com/poyro/unit-test-legal-ai-demo)
## Prerequisites
- Node.js 20 or later
- Vitest 1.6.0 or later
- Your project must have "type": "module" in its package.json (this library is ESM only)
- Recommended: CUDA-compatible GPU (Nvidia) or Metal-compatible GPU (Apple Silicon) for best performance, but not required
---
## Usage
### Install
To get started quickly, simply run:
```bash
npx poyro init
```
### Create your first test
To use the matchers, create a file with the extension `.test.ts` anywhere within your project. Here is an example:
```javascript
// MyFirstTest.test.ts
import { describe, expect, it } from "vitest";
describe("MyFirstTest", () => {
it("should be true", async () => {
// Replace this with your LLM app's output
const llmOutput = "Hello, world!";
// Criterion in natural language: True or False
await expect(llmOutput).toFulfillCriterion("Says hello");
}, 10000); // Increase the timeout as needed with the third argument
});
```
`toFulfillCriterion` can determine whether an LLM output meets or does not meet a natural language criterion.
The first time you run a test like the one above, the model file for our locally run LLM will be downloaded. It should take a couple of minutes.
### Run your test
Run vitest with your package manager, for example with `npm`:
```bash
npm test
```
Similar commands work with `yarn` and `pnpm`.
---
## Help
### Troubleshooting
We provide solutions to common issues in our [Troubleshooting](https://docs.poyro.dev/troubleshooting) page. Take a look at the solutions there, and if you continue to run into problems please don't hesitate to reach out to us.
### Submit an issue
You can submit an issue by going to our GitHub repository and [creating a new issue](https://github.com/poyro/poyro/issues/new). Prior to submitting an issue, please check if a similar issue has already been submitted.
### Join the community
If you want to get help, brainstorm on good evals, or if you just want to chat with other Poyro developers, join our community on [Discord](https://discord.gg/gmCjjJ5jSf)!
## Contributing
If you would like to contribute to Poyro, please read our [contributing guide](CONTRIBUTING.md).
Begin by cloning the repository:
```bash
git clone https://github.com/poyro/poyro.git
```
Next, install all the dependencies:
```bash
pnpm install
```
When developing, you can either run `pnpm dev` at the root, which will automatically build all relevant packages when changes are made, or you can run `pnpm dev` within the package you are working on.