{"id":15114280,"url":"https://github.com/yukinagae/genkitx-promptfoo","last_synced_at":"2025-07-27T16:44:12.237Z","repository":{"id":257779567,"uuid":"855240215","full_name":"yukinagae/genkitx-promptfoo","owner":"yukinagae","description":"Community Plugin for Genkit to use Promptfoo","archived":false,"fork":false,"pushed_at":"2025-01-03T15:03:39.000Z","size":566,"stargazers_count":3,"open_issues_count":7,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-21T06:41:25.363Z","etag":null,"topics":["ai","evaluation","evaluation-framework","firebase","genkit","genkit-plugin","genkitx","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","plugin","prompt","prompt-testing","promptfoo","prompts","testing"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/genkitx-promptfoo","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yukinagae.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-10T14:52:43.000Z","updated_at":"2024-10-10T00:12:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"4ea2d6bb-9a91-450b-9288-57777e143b76","html_url":"https://github.com/yukinagae/genkitx-promptfoo","commit_stats":null,"previous_names":["yukinagae/genkitx-promptfoo"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukinagae%2Fgenkitx-promptfoo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukinagae%2Fgenkitx-promptfoo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukinagae%2Fgenkitx-promptfoo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yukinagae%2Fgenkitx-promptfoo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yukinagae","download_url":"https://codeload.github.com/yukinagae/genkitx-promptfoo/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247358684,"owners_count":20926266,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","evaluation","evaluation-framework","firebase","genkit","genkit-plugin","genkitx","llm","llm-eval","llm-evaluation","llm-evaluation-framework","llmops","plugin","prompt","prompt-testing","promptfoo","prompts","testing"],"created_at":"2024-09-26T01:41:09.280Z","updated_at":"2025-04-05T15:31:22.896Z","avatar_url":"https://github.com/yukinagae.png","language":"TypeScript","funding_links":[],"categories":["Plugins"],"sub_categories":["JavaScript - Community"],"readme":"![Firebase Genkit + Promptfoo](https://github.com/yukinagae/genkitx-promptfoo/blob/main/assets/genkit-promptfoo.png?raw=true)\n\n\u003ch1\u003e\n   Firebase Genkit \u003c\u003e Promptfoo Plugin\n\u003c/h1\u003e\n\n\u003ch4\u003ePromptfoo Community Plugin for Google Firebase Genkit\u003c/h4\u003e\n\n\u003cdiv\u003e\n   \u003cimg alt=\"Github version\" src=\"https://img.shields.io/github/v/release/yukinagae/genkitx-promptfoo\"\u003e\n   \u003cimg alt=\"NPM Downloads\" src=\"https://img.shields.io/npm/dw/genkitx-promptfoo\"\u003e\n   \u003cimg alt=\"GitHub License\" src=\"https://img.shields.io/github/license/yukinagae/genkitx-promptfoo\"\u003e\n   \u003cimg alt=\"Static Badge\" src=\"https://img.shields.io/badge/yes-a?label=maintained\"\u003e\n\u003c/div\u003e\n\n\u003cdiv\u003e\n   \u003cimg alt=\"GitHub Issues or Pull Requests\" src=\"https://img.shields.io/github/issues/yukinagae/genkitx-promptfoo?color=blue\"\u003e\n   \u003cimg alt=\"GitHub Issues or Pull Requests\" src=\"https://img.shields.io/github/issues-pr/yukinagae/genkitx-promptfoo?color=blue\"\u003e\n   \u003cimg alt=\"GitHub commit activity\" src=\"https://img.shields.io/github/commit-activity/m/yukinagae/genkitx-promptfoo\"\u003e\n\u003c/div\u003e\n\n\u003c/br\u003e\n\n**`genkitx-promptfoo`** is a community plugin for using [Promptfoo](https://github.com/promptfoo/promptfoo) with [Firebase Genkit](https://github.com/firebase/genkit).\nBuilt by [**Yuki Nagae**](https://github.com/yukinagae).\n\nThis Genkit plugin allows to use [Promptfoo](https://github.com/promptfoo/promptfoo).\n\n\u003e [!WARNING]\n\u003e **This version is experimental and may have API changes and critical bugs. Use it for prototypes or hobby projects, not in production**.\n\n## Installation\n\nInstall the plugin in your project with your favorite package manager:\n\n- `npm install genkitx-promptfoo`\n- `yarn add genkitx-promptfoo`\n- `pnpm add genkitx-promptfoo`\n\n## Usage\n\n### Configuration\n\n```typescript\nimport { promptfooEval } from 'genkitx-promptfoo';\n\nconfigureGenkit({\n  plugins: [\n    promptfooEval({\n      metrics: [\n        {\n          type: 'contains',\n          value: 'Hello, World!',\n        },\n        {\n          type: 'regex',\n          value: '^Hello, World!$',\n        },\n        {\n          type: 'javascript',\n          value: \"output.includes('Hello, World!')\",\n        },\n        {\n          type: 'similar',\n          value: 'Aloha, World!',\n          threshold: 0.8,\n          provider: 'vertex:embedding:text-embedding-004',\n        },\n        {\n          type: 'llm-rubric',\n          value: 'It is a friendly greeting.',\n          provider: 'vertex:gemini-1.5-flash',\n        },\n      ],\n    }),\n  ],\n});\n```\n\n### Basic examples\n\nYou can specify the Promptfoo metrics below:\n\n- [contains / icontains](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains)\n\n```typescript\n{\n  type: 'contains',\n  value: 'The expected substring',\n},\n```\n\n- [contains-all](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-all)\n\n```typescript\n{\n  type: 'contains-all',\n  value: [\n   'Value 1',\n   'Value 2',\n   'Value 3',\n  ],\n},\n```\n\n- [contains-any](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-any)\n\n```typescript\n{\n  type: 'contains-any',\n  value: [\n   'Value 1',\n   'Value 2',\n   'Value 3',\n  ],\n},\n```\n\n- [regex](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#regex)\n\n```typescript\n{\n  type: 'regex',\n  value: '\\\\d{4}', // Matches a 4-digit number\n},\n```\n\n- [contains-json](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-json) / [is-json](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-json)\n\n```typescript\n{\n  type: 'contains-json',\n  value:\n  {\n    'required': ['latitude', 'longitude'],\n    'type': 'object',\n    'properties':\n      {\n        'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },\n        'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },\n      },\n  }\n},\n```\n\n- [contains-sql](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-sql) / [is-sql](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-sql)\n\nTo use this assertion, you need to install the node-sql-parser package. You can install it using npm: `npm install node-sql-parser`.\n\n```typescript\n{\n  type: 'contains-sql',\n},\n```\n\n- [equals](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#equality)\n\n```typescript\n{\n  type: 'equals',\n  value: 'The expected output',\n},\n```\n\n- [contains-xml](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#contains-xml) / [is-xml](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-xml)\n\n```typescript\n{\n  type: 'contains-xml',\n},\n```\n\n- [javascript](https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/)\n\n```typescript\n{\n  type: 'javascript',\n  value: \"output.includes('Hello, World!')\",\n},\n```\n\n- [starts-with](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#starts-with)\n\n```typescript\n{\n  type: 'starts-with',\n  value: 'Yes',\n},\n```\n\n- [similar](https://www.promptfoo.dev/docs/configuration/expected-outputs/similar/)\n\n```typescript\n{\n  type: 'similar',\n  value: 'The expected output',\n  threshold: 0.8,\n  provider: 'vertex:embedding:text-embedding-004',\n},\n```\n\n- [llm-rubric](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/)\n\n```typescript\n{\n  type: 'llm-rubric',\n  value: 'Is not apologetic and provides a clear, concise answer',\n  provider: 'vertex:gemini-1.5-flash',\n},\n```\n\n## Contributing\n\nWant to contribute to the project? That's awesome! Head over to our [Contribution Guidelines](https://github.com/yukinagae/genkitx-promptfoo/blob/main/CONTRIBUTING.md).\n\n## Need support?\n\n\u003e [!NOTE]  \n\u003e This repository depends on Google's Firebase Genkit. For issues and questions related to Genkit, please refer to instructions available in [Genkit's repository](https://github.com/firebase/genkit).\n\nReach out by opening a discussion on [Github Discussions](https://github.com/yukinagae/genkitx-promptfoo/discussions).\n\n## Credits\n\nThis plugin is proudly maintained by Yuki Nagae [**Yuki Nagae**](https://github.com/yukinagae).\n\nI got the inspiration, structure and patterns to create this plugin from the [Genkit Community Plugins](https://github.com/TheFireCo/genkit-plugins) repository built by the [Fire Compnay](https://github.com/TheFireCo) as well as the [github plugin](https://github.com/xavidop/genkitx-github).\n\n## License\n\nThis project is licensed under the [Apache 2.0 License](https://github.com/yukinagae/genkitx-promptfoo/blob/main/LICENSE).\n\n[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202%2E0-lightgrey.svg)](https://github.com/yukinagae/genkitx-promptfoo/blob/main/LICENSE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyukinagae%2Fgenkitx-promptfoo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyukinagae%2Fgenkitx-promptfoo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyukinagae%2Fgenkitx-promptfoo/lists"}