https://github.com/yukinagae/genkitx-promptfoo
Community Plugin for Genkit to use Promptfoo
https://github.com/yukinagae/genkitx-promptfoo
ai evaluation evaluation-framework firebase genkit genkit-plugin genkitx llm llm-eval llm-evaluation llm-evaluation-framework llmops plugin prompt prompt-testing promptfoo prompts testing
Last synced: 3 months ago
JSON representation
Community Plugin for Genkit to use Promptfoo
- Host: GitHub
- URL: https://github.com/yukinagae/genkitx-promptfoo
- Owner: yukinagae
- License: apache-2.0
- Created: 2024-09-10T14:52:43.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-03T15:03:39.000Z (9 months ago)
- Last Synced: 2025-03-21T06:41:25.363Z (7 months ago)
- Topics: ai, evaluation, evaluation-framework, firebase, genkit, genkit-plugin, genkitx, llm, llm-eval, llm-evaluation, llm-evaluation-framework, llmops, plugin, prompt, prompt-testing, promptfoo, prompts, testing
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/genkitx-promptfoo
- Size: 553 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 7
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
- Security: SECURITY.md
Awesome Lists containing this project
- awesome-genkit - `genkitx-promptfoo` - Plugin for Promptfoo Evaluations. (Plugins / JavaScript - Community)
README
Firebase Genkit <> Promptfoo Plugin
Promptfoo Community Plugin for Google Firebase Genkit
**`genkitx-promptfoo`** is a community plugin for using [Promptfoo](https://github.com/promptfoo/promptfoo) with [Firebase Genkit](https://github.com/firebase/genkit).
Built by [**Yuki Nagae**](https://github.com/yukinagae).
This Genkit plugin allows to use [Promptfoo](https://github.com/promptfoo/promptfoo).
> [!WARNING]
> **This version is experimental and may have API changes and critical bugs. Use it for prototypes or hobby projects, not in production**.
## Installation
Install the plugin in your project with your favorite package manager:
- `npm install genkitx-promptfoo`
- `yarn add genkitx-promptfoo`
- `pnpm add genkitx-promptfoo`
## Usage
### Configuration
```typescript
import { promptfooEval } from 'genkitx-promptfoo';
configureGenkit({
plugins: [
promptfooEval({
metrics: [
{
type: 'contains',
value: 'Hello, World!',
},
{
type: 'regex',
value: '^Hello, World!$',
},
{
type: 'javascript',
value: "output.includes('Hello, World!')",
},
{
type: 'similar',
value: 'Aloha, World!',
threshold: 0.8,
provider: 'vertex:embedding:text-embedding-004',
},
{
type: 'llm-rubric',
value: 'It is a friendly greeting.',
provider: 'vertex:gemini-1.5-flash',
},
],
}),
],
});
```
### Basic examples
You can specify the Promptfoo metrics below:
- [contains / icontains](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains)
```typescript
{
type: 'contains',
value: 'The expected substring',
},
```
- [contains-all](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-all)
```typescript
{
type: 'contains-all',
value: [
'Value 1',
'Value 2',
'Value 3',
],
},
```
- [contains-any](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-any)
```typescript
{
type: 'contains-any',
value: [
'Value 1',
'Value 2',
'Value 3',
],
},
```
- [regex](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#regex)
```typescript
{
type: 'regex',
value: '\\d{4}', // Matches a 4-digit number
},
```
- [contains-json](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-json) / [is-json](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-json)
```typescript
{
type: 'contains-json',
value:
{
'required': ['latitude', 'longitude'],
'type': 'object',
'properties':
{
'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
},
}
},
```
- [contains-sql](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-sql) / [is-sql](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-sql)
To use this assertion, you need to install the node-sql-parser package. You can install it using npm: `npm install node-sql-parser`.
```typescript
{
type: 'contains-sql',
},
```
- [equals](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#equality)
```typescript
{
type: 'equals',
value: 'The expected output',
},
```
- [contains-xml](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-xml) / [is-xml](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-xml)
```typescript
{
type: 'contains-xml',
},
```
- [javascript](https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/)
```typescript
{
type: 'javascript',
value: "output.includes('Hello, World!')",
},
```
- [starts-with](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#starts-with)
```typescript
{
type: 'starts-with',
value: 'Yes',
},
```
- [similar](https://www.promptfoo.dev/docs/configuration/expected-outputs/similar/)
```typescript
{
type: 'similar',
value: 'The expected output',
threshold: 0.8,
provider: 'vertex:embedding:text-embedding-004',
},
```
- [llm-rubric](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/)
```typescript
{
type: 'llm-rubric',
value: 'Is not apologetic and provides a clear, concise answer',
provider: 'vertex:gemini-1.5-flash',
},
```
## Contributing
Want to contribute to the project? That's awesome! Head over to our [Contribution Guidelines](https://github.com/yukinagae/genkitx-promptfoo/blob/main/CONTRIBUTING.md).
## Need support?
> [!NOTE]
> This repository depends on Google's Firebase Genkit. For issues and questions related to Genkit, please refer to instructions available in [Genkit's repository](https://github.com/firebase/genkit).
Reach out by opening a discussion on [Github Discussions](https://github.com/yukinagae/genkitx-promptfoo/discussions).
## Credits
This plugin is proudly maintained by Yuki Nagae [**Yuki Nagae**](https://github.com/yukinagae).
I got the inspiration, structure and patterns to create this plugin from the [Genkit Community Plugins](https://github.com/TheFireCo/genkit-plugins) repository built by the [Fire Compnay](https://github.com/TheFireCo) as well as the [github plugin](https://github.com/xavidop/genkitx-github).
## License
This project is licensed under the [Apache 2.0 License](https://github.com/yukinagae/genkitx-promptfoo/blob/main/LICENSE).
[](https://github.com/yukinagae/genkitx-promptfoo/blob/main/LICENSE)