https://github.com/moeki0/promptest
The Prompt testing library for LLM that allows comparing patterns of prompts.
https://github.com/moeki0/promptest
ai llm npm prompt prompt-engineering test
Last synced: about 1 month ago
JSON representation
The Prompt testing library for LLM that allows comparing patterns of prompts.
- Host: GitHub
- URL: https://github.com/moeki0/promptest
- Owner: moeki0
- License: mit
- Created: 2023-06-28T14:59:39.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-06-28T23:30:17.000Z (almost 3 years ago)
- Last Synced: 2025-07-29T08:45:34.242Z (11 months ago)
- Topics: ai, llm, npm, prompt, prompt-engineering, test
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/promptest
- Size: 230 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# PrompTest
The Prompt testing library for LLM that allows comparing patterns of prompts.
## Installation
$ npm i --save-dev promptest
## Usage
```ts
// prompt.test.js
const { Configuration, OpenAIApi } = require("openai")
const { variable, promptest } = require('promptest')
const configuration = new Configuration({
apiKey: process.env.OPENAI_API_KEY,
})
const openai = new OpenAIApi(configuration)
variable("name", ["John", "Jane"])
variable("age", ["20", "30"])
promptest(
"Your name is {{name}}, and you are {{age}} years old.",
"Hello!",
async (subject, input) => {
const chatCompletion = await openai.createChatCompletion({
model: "gpt-3.5-turbo",
messages: [
{ role: "system", content: subject },
{ role: "user", content: input },
],
})
return chatCompletion.data.choices[0].message.content
}
).then(console.log).catch(console.error)
```
```bash
npx node prompt.test.js
```
### Variable
```ts
variable("name", ["John", "Jane"])
```
`variable` is called with variable name and patterns.
### Subject
The subject prompt template is defined with `{{name}}`.
```ts
promptest(
"Your name is {{name}}, and you are {{age}} years old.",
// ...
)
```
### Input
Input is ordinary user prompt for testing.
```ts
promptest(
// ..
"Hello!"
// ..
)
```
### Callback
The callback is test function. The arguments are subject prompt and input prompt.
```ts
promptest(
// ...
async (subject: string, input: string) => {
const chatCompletion = await openai.createChatCompletion({
model: "gpt-3.5-turbo",
messages: [
{ role: "system", content: subject },
{ role: "user", content: input },
],
})
return chatCompletion.data.choices[0].message.content
}
)
```
### Result
The result is JSON.
```json
{
"input": "Hello!",
"results": [
{
"subject": "Your name is John, and you are 20 years old",
"output": "Hello! How can I assist you today?"
},
{
"subject": "Your name is Jane, and you are 20 years old",
"output": "Hello! How can I assist you today?"
},
{
"subject": "Your name is John, and you are 30 years old",
"output": "Hello! How can I help you today?"
},
{
"subject": "Your name is Jane, and you are 30 years old",
"output": "Hello! How can I assist you today?"
},
]
}
```
## Development
After checking out the repo, run `npm i` to install dependencies. Then, run `npm run test` to run the tests.
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/moekidev/promptest. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/moekidev/promptest/blob/main/CODE_OF_CONDUCT.md).
## License
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
## Code of Conduct
Everyone interacting in the Promptest project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/moekidev/promptest/blob/main/CODE_OF_CONDUCT.md).