{"id":15103574,"url":"https://github.com/explosion/prodigy-openai-recipes","last_synced_at":"2025-09-27T02:31:41.285Z","repository":{"id":65322084,"uuid":"575424467","full_name":"explosion/prodigy-openai-recipes","owner":"explosion","description":"✨ Bootstrap annotation with zero- \u0026 few-shot learning via OpenAI GPT-3","archived":true,"fork":false,"pushed_at":"2023-08-09T08:29:40.000Z","size":7611,"stargazers_count":318,"open_issues_count":0,"forks_count":27,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-09-26T19:40:32.648Z","etag":null,"topics":["annotation-tool","few-shot-learning","gpt-3","nlp","openai","openai-api","prodigy","zero-shot-learning"],"latest_commit_sha":null,"homepage":"https://prodi.gy","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/explosion.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-12-07T13:37:38.000Z","updated_at":"2024-07-07T10:43:05.000Z","dependencies_parsed_at":"2024-09-20T15:32:26.957Z","dependency_job_id":null,"html_url":"https://github.com/explosion/prodigy-openai-recipes","commit_stats":{"total_commits":156,"total_committers":9,"mean_commits":"17.333333333333332","dds":0.5448717948717949,"last_synced_commit":"a0866f0d05f0a744cab15ad202201c5b691d48b9"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/explosion%2Fprodigy-openai-recipes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/explosion%2Fprodigy-openai-recipes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/explosion%2Fprodigy-openai-recipes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/explosion%2Fprodigy-openai-recipes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/explosion","download_url":"https://codeload.github.com/explosion/prodigy-openai-recipes/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":234376915,"owners_count":18822416,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["annotation-tool","few-shot-learning","gpt-3","nlp","openai","openai-api","prodigy","zero-shot-learning"],"created_at":"2024-09-25T19:40:33.626Z","updated_at":"2025-09-27T02:31:35.836Z","avatar_url":"https://github.com/explosion.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"https://explosion.ai\"\u003e\u003cimg src=\"https://explosion.ai/assets/img/logo.svg\" width=\"125\" height=\"125\" align=\"right\" /\u003e\u003c/a\u003e\n\n## Archival notice \n\nThe recipes in this repository have since moved to [Prodigy](https://prodi.gy/) and are being maintained there. They will soon even get an upgrade with the advent of [spacy-llm](https://github.com/explosion/spacy-llm) support, which features better prompts and multiple LLM providers. That is why we've opted to archive this repo, so that we may focus on maintaining these recipes as part of spaCy and Prodigy directly.\n\nYou can learn more by checking out the [large language models section](https://prodi.gy/docs/large-language-models) on the docs.\n\n# Prodigy OpenAI recipes\n\nThis repository contains example code on how to combine **zero- and few-shot learning\nwith a small annotation effort** to obtain a **high-quality dataset with maximum efficiency**. Specifically, we use large language models available from [OpenAI](https://openai.com) to provide us with an initial set of predictions,\nthen spin up a [Prodigy](https://prodi.gy) instance on our local machine\nto go through these predictions and curate them. This allows us to obtain a\ngold-standard dataset pretty quickly, and train a smaller, supervised model that fits\nour exact needs and use-case.\n\n![](https://user-images.githubusercontent.com/13643239/208497043-178beb47-f7c6-4b3e-a253-9e12e2f0c807.png)\n\nhttps://user-images.githubusercontent.com/13643239/208504034-0ab6bcbe-6d2b-415d-8257-233f2074ba31.mp4\n\n## ⏳ Setup and Install\n\nMake sure to [install Prodigy](https://prodi.gy/docs/install) as well as a few additional Python dependencies:\n\n```bash\npython -m pip install prodigy -f https://XXXX-XXXX-XXXX-XXXX@download.prodi.gy\npython -m pip install -r requirements.txt\n```\n\nWith `XXXX-XXXX-XXXX-XXXX` being your personal Prodigy license key.\n\nThen, create a new API key from [openai.com](https://beta.openai.com/account/api-keys) or fetch an existing\none. Record the secret key as well as the [organization key](https://beta.openai.com/account/org-settings)\nand make sure these are available as environmental variables. For instance, set them in a `.env` file in the\nroot directory:\n\n```\nOPENAI_ORG = \"org-...\"\nOPENAI_KEY = \"sk-...\"\n```\n\n## 📋 Named-entity recognition (NER)\n\n### `ner.openai.correct`: NER annotation with zero- or few-shot learning\n\nThis recipe marks entity predictions obtained from a large language model and allows you to flag them as correct, or to\nmanually curate them. This allows you to quickly gather a gold-standard dataset through zero-shot or few-shot learning.\nIt's very much like using the standard [`ner.correct`](https://prodi.gy/docs/recipes#ner-correct) recipe in Prodi.gy,\nbut we're using GPT-3 as a backend model to make predictions.\n\n```bash\npython -m prodigy ner.openai.correct dataset filepath labels [--options] -F ./recipes/openai_ner.py\n```\n\n| Argument                | Type | Description                                                                                                                                     | Default                         |\n| ----------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |\n| `dataset`               | str  | Prodigy dataset to save annotations to.                                                                                                         |                                 |\n| `filepath`              | Path | Path to `.jsonl` data to annotate. The data should at least contain a `\"text\"` field.                                                           |                                 |\n| `labels`                | str  | Comma-separated list defining the NER labels the model should predict.                                                                          |                                 |\n| `--lang`, `-l`          | str  | Language of the input data - will be used to obtain a relevant tokenizer.                                                                       | `\"en\"`                          |\n| `--segment`, `-S`       | bool | Flag to set when examples should be split into sentences. By default, the full input article is shown.                                          | `False`                         |\n| `--model`, `-m`         | str  | GPT-3 model to use for initial predictions.                                                                                                     | `\"text-davinci-003\"`            |\n| `--prompt_path`, `-p`   | Path | Path to the `.jinja2` [prompt template](templates).                                                                                             | `./templates/ner_prompt.jinja2` |\n| `--examples-path`, `-e` | Path | Path to examples to help define the task. The file can be a .yml, .yaml or .json. If set to `None`, zero-shot learning is applied.              | `None`                          |\n| `--max-examples`, `-n`  | int  | Max number of examples to include in the prompt to OpenAI. If set to 0, zero-shot learning is always applied, even when examples are available. | 2                               |\n| `--batch-size`, `-b`    | int  | Batch size of queries to send to the OpenAI API.                                                                                                | 10                              |\n| `--verbose`, `-v`       | bool | Flag to print extra information to the terminal.                                                                                                | `False`                         |\n\n#### Example usage\n\nLet's say we want to recognize dishes, ingredients and cooking equipment from some text we obtained from a cooking subreddit.\nWe'll send the text to GPT-3, hosted by OpenAI, and provide an annotation prompt to explain\nto the language model the type of predictions we want. Something like:\n\n```\nFrom the text below, extract the following entities in the following format:\ndish: \u003ccomma delimited list of strings\u003e\ningredient: \u003ccomma delimited list of strings\u003e\nequipment: \u003ccomma delimited list of strings\u003e\n\nText:\n...\n```\n\nWe define the definition of this prompt in a .jinja2 file which also describes how to append examples for few-shot learning.\nYou can create your own [template](templates) and provide it to the recipe with the `--prompt-path` or `-p` option.\nAdditionally, with `--examples-path` or `-e` you can set the file path of a .y(a)ml or .json file that contains additional examples:\n\n```bash\npython -m prodigy ner.openai.correct my_ner_data ./data/reddit_r_cooking_sample.jsonl \"dish,ingredient,equipment\" -p ./templates/ner_prompt.jinja2 -e ./examples/ner.yaml -n 2 -F ./recipes/openai_ner.py\n```\n\nAfter receiving the results from the OpenAI API, the Prodigy recipe converts the predictions into an annotation task\nthat can be rendered with Prodigy. The task even shows the original prompt as well as the raw answer we obtained\nfrom the language model.\n\n\u003cimg src=\"https://user-images.githubusercontent.com/8796347/208484904-72fd79e4-9f14-4c40-9993-97a5776aafb3.png\" width=\"600\" /\u003e\n\nHere, we see that the model is able to correctly recognize dishes, ingredients and cooking equipment right from the start!\n\nThe recipe also offers a `--verbose` or `-v` option that includes the exact prompt and response on the terminal as traffic is received.\nNote that because the requests to the API are batched, you might have to scroll back a bit to find the current prompt.\n\n### Interactively tune the prompt examples\n\nAt some point, you might notice a mistake in the predictions of the OpenAI language model. For instance, we noticed an error\nin the recognition of cooking equipment in this example:\n\n\u003cimg src=\"https://user-images.githubusercontent.com/8796347/208485149-a32fa2da-db8a-42a5-a2a7-4708f127b592.png\" width=\"600\" /\u003e\n\nIf you see these kind of systematic errors, you can steer the predictions in the right direction by correcting the example and then selecting the small \"flag\" icon\nin the top right of the Prodigy UI:\n\n\u003cimg src=\"https://user-images.githubusercontent.com/8796347/208485453-861fdecf-6283-4a8f-b802-58314f3e496d.png\" width=\"600\" /\u003e\n\nOnce you hit \u003ckbd\u003eaccept\u003c/kbd\u003e on the Prodigy interface, the flagged example will be automatically picked up and added to the examples\nthat are sent to the OpenAI API as part of the prompt.\n\n\u003e **Note**  \n\u003e Because Prodigy batches these requests, the prompt will be updated with a slight\n\u003e delay, after the next batch of prompts is sent to OpenAI. You can experiment\n\u003e with making the batch size (`--batch-size` or `-b`) smaller to have the change\n\u003e come into effect sooner, but this might negatively impact the speed of the\n\u003e annotation workflow.\n\n### `ner.openai.fetch`: Fetch examples up-front\n\nThe `ner.openai.correct` recipe fetches examples from OpenAI while annotating, but we've also included a recipe that can fetch a large batch of examples upfront.\n\n```bash\npython -m prodigy ner.openai.fetch input_data.jsonl predictions.jsonl \"dish,ingredient,equipment\" -F ./recipes/ner.py\n```\n\nThis will create a `predictions.jsonl` file that can be loaded with the [`ner.manual`](https://prodi.gy/docs/recipes#ner-manual) recipe.\n\nNote that the OpenAI API might return \"429 Too Many Request\" errors when requesting too much data at once - in this case it's best to ensure you only request\n100 or so examples at a time.\n\n### Exporting the annotations and training an NER model\n\nAfter you've curated a set of predictions, you can export the results with [`db-out`](https://prodi.gy/docs/recipes#db-out):\n\n```bash\npython -m prodigy db-out my_ner_data  \u003e ner_data.jsonl\n```\n\nThe format of the exported annotations contains all the data you need to train a smaller model downstream. Each example\nin the dataset contains the original text, the tokens, span annotations denoting the entities, etc.\n\nYou can also export the data to spaCy's [binary format](https://spacy.io/api/data-formats#training), using [`data-to-spacy`](https://prodi.gy/docs/recipes#data-to-spacy). This format lets you load in the annotations as spaCy `Doc` objects, which can be convenient for further conversion. The `data-to-spacy` command also makes it easy to train an NER model with spaCy. First you export the data, specifying the train data as 20% of the total:\n\n```bash\npython -m prodigy data-to-spacy ./data/annotations/ --ner my_ner_data -es 0.2\n```\n\nThen you can train a model with spaCy or [Prodigy](https://prodi.gy/docs/recipes/#training):\n\n```bash\npython -m spacy train ./data/annotations/config.cfg --paths.train ./data/annotations/train.spacy --paths.dev ./data/annotations/dev.spacy -o ner-model\n```\n\nThis will save a model to the `ner-model/` directory.\n\nWe've also included an experimental script to load in the `.spacy` binary format and train a model with the HuggingFace `transformers` library. You can use the same data you just exported and run the script like this:\n\n```bash\n# First you need to install the HuggingFace library and requirements\npip install -r requirements_train.txt\npython ./scripts/train_hf_ner.py ./data/annotations/train.spacy ./data/annotations/dev.spacy -o hf-ner-model\n```\n\nThe resulting model will be saved to the `hf-ner-model/` directory.\n\n## 📋 Text categorization (Textcat)\n\n### `textcat.openai.correct`: Textcat annotation with zero- or few-shot learning\n\nThis recipe enables us to classify texts faster with the help of a large\nlanguage model. It also provides a \"reason\" to explain why a particular label\nwas chosen. \n\n```bash\npython -m prodigy textcat.openai.correct dataset filepath labels [--options] -F ./recipes/openai_textcat.py\n```\n\n| Argument                    | Type | Description                                                                                                                                     | Default                             |\n| --------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |\n| `dataset`                   | str  | Prodigy dataset to save annotations to.                                                                                                         |                                     |\n| `filepath`                  | Path | Path to `.jsonl` data to annotate. The data should at least contain a `\"text\"` field.                                                           |                                     |\n| `labels`                    | str  | Comma-separated list defining the text categorization labels the model should predict.                                                          |                                     |\n| `--lang`, `-l`              | str  | Language of the input data - will be used to obtain a relevant tokenizer.                                                                       | `\"en\"`                              |\n| `--segment`, `-S`           | bool | Flag to set when examples should be split into sentences. By default, the full input article is shown.                                          | `False`                             |\n| `--model`, `-m`             | str  | GPT-3 model to use for initial predictions.                                                                                                     | `\"text-davinci-003\"`                |\n| `--prompt-path`, `-p`       | Path | Path to the `.jinja2` [prompt template](templates).                                                                                             | `./templates/textcat_prompt.jinja2` |\n| `--examples-path`, `-e`     | Path | Path to examples to help define the task. The file can be a .yml, .yaml or .json. If set to `None`, zero-shot learning is applied.              | `None`                              |\n| `--max-examples`, `-n`      | int  | Max number of examples to include in the prompt to OpenAI. If set to 0, zero-shot learning is always applied, even when examples are available. | 2                                   |\n| `--batch-size`, `-b`        | int  | Batch size of queries to send to the OpenAI API.                                                                                                | 10                                  |\n| `--exclusive-classes`, `-E` | bool | Flag to make the classification task exclusive.                                                                                                 | `False`                             |\n| `--verbose`, `-v`           | bool | Flag to print extra information to the terminal.                                                                                                | `False`                             |\n\n\n#### Example usage\n\nThe `textcat` recipes can be used for binary, multiclass, and multilabel text\ncategorization. You can set this by passing the appropriate number of labels in\nthe `--labels` parameter; for example, passing a single label turns it into\nbinary classification and so on. We will talk about each one in the proceeding\nsections.\n\n##### Binary text categorization\n\nSuppose we want to know if a particular Reddit comment talks about a food\nrecipe.  We'll send the text to GPT-3 and provide a prompt that instructs the\npredictions we want. \n\n```\nFrom the text below, determine wheter or not it contains a recipe. If it is a \nrecipe, answer \"accept.\" If it is not a recipe, answer \"reject.\"\n\nYour answer should only be in the following format:\nanswer: \u003cstring\u003e\nreason: \u003cstring\u003e\n\nText:\n```\n\nFor binary classification, we want GPT-3 to return \"accept\" if a given text is a\nfood recipe and \"reject\" otherwise. GPT-3's suggestion is then displayed\nprominently in the UI. We can press the \u003ckbd\u003eACCEPT\u003c/kbd\u003e (check mark) button to\ninclude the text as a positive example or press the \u003ckbd\u003eREJECT\u003c/kbd\u003e (cross\nmark) button if it is a negative example.\n\n\n```sh\npython -m prodigy textcat.openai.correct my_binary_textcat_data data/reddit_r_cooking_sample.jsonl --labels recipe -F recipes/openai_textcat.py\n```\n\n\u003cimg src=\"https://user-images.githubusercontent.com/12949683/214230166-ee492fe5-04da-4b93-9590-b5ef23ce488d.png\" width=\"600\"/\u003e\n\n\n##### Multilabel and multiclass text categorization\n\nNow, suppose we want to classify Reddit comments as a recipe, a feedback, or a\nquestion. We can write the following prompt: \n\n```\nClassify the text below to any of the following labels: recipe, feedback, question.\nThe task is exclusive, so only choose one label from what I provided.\n\nYour answer should only be in the following format:\nanswer: \u003cstring\u003e\nreason: \u003cstring\u003e\n\nText:\n```\n\nThen, we can use this recipe to handle multilabel and multiclass cases by\npassing the three labels to the `--labels` parameter. We should also set the\n`--exclusive-classes` flag to render a single-choice UI:\n\n```sh\npython -m prodigy textcat.openai.correct my_multi_textcat_data data/reddit_r_cooking_sample.jsonl \\\n    --labels recipe,feedback,question \\\n    --exclusive-classes \\\n    -F recipes/openai_textcat.py\n```\n\n\u003cimg src=\"https://user-images.githubusercontent.com/12949683/214230269-bdacbbc8-8edc-4be5-8334-0b7eaf4712e0.png\" width=\"600\" /\u003e\n\n### Writing templates\n\nWe write these prompts as a .jinja2 template that can also take in examples for\nfew-shot learning. You can create your own [template](templates) and provide it\nto the recipe with the `--prompt-path` or `-p` option.  Additionally, with\n`--examples-path` or `-e` you can set the file path of a .y(a)ml or .json file\nthat contains additional examples. You can also add context in these examples as\nwe observed it to improve the output:\n\n```bash\npython -m prodigy textcat.openai.correct my_binary_textcat_data \\\n    ./data/reddit_r_cooking_sample.jsonl \\\n    --labels recipe \\\n    --prompt-path ./templates/textcat_prompt.jinja2 \\\n    --examples-path ./examples/textcat_binary.yaml -n 2 \\\n    -F ./recipes/openai_textcat.py\n```\n\nSimilar to the NER recipe, this recipe also converts the predictions into an \nannotation task that can be rendered with Prodigy. For binary classification, we\nuse the [`classification`](https://prodi.gy/docs/api-interfaces#classification)\ninterface with custom HTML elements, while for multilabel or multiclass text\ncategorization, we use the\n[`choice`](https://prodi.gy/docs/api-interfaces#choice) annotation interface.\nNotice that we include the original prompt and the OpenAI response in the UI. \n\n\nLastly, you can use the `--verbose` or `-v` flag to show the exact prompt and\nresponse on the terminal. Note that because the requests to the API are batched,\nyou might have to scroll back a bit to find the current prompt.\n\n\n### Interactively tune the prompt examples\n\nSimilar to the NER recipes, you can also steer the predictions in the right\ndirection by correcting the example and then selecting the small \"flag\" icon in\nthe top right of the Prodigy UI:\n\n\u003cimg src=\"https://user-images.githubusercontent.com/12949683/214780178-96be66e4-8a02-4820-a51e-3e6040bcddc1.png\" width=\"600\"/\u003e\n\nOnce you hit the \u003ckbd\u003eaccept\u003c/kbd\u003e button on the Prodigy interface, the flagged\nexample will be picked up and added to the few-shot examples sent to the OpenAI\nAPI as part of the prompt.\n\n\u003e **Note**  \n\u003e Because Prodigy batches these requests, the prompt will be updated with a slight\n\u003e delay, after the next batch of prompts is sent to OpenAI. You can experiment\n\u003e with making the batch size (`--batch-size` or `-b`) smaller to have the change\n\u003e come into effect sooner, but this might negatively impact the speed of the\n\u003e annotation workflow.\n\n### `textcat.openai.fetch`: Fetch text categorization examples up-front\n\nThe `textcat.openai.fetch` recipe allows us to fetch a large batch of examples\nupfront. This is helpful when you are with a highly-imbalanced data and interested\nonly in rare examples.\n\n```bash\npython -m prodigy textcat.openai.fetch input_data.jsonl predictions.jsonl --labels Recipe -F ./recipes/openai_textcat.py\n```\n\nThis will create a `predictions.jsonl` file that can be loaded with the\n[`textcat.manual`](https://prodi.gy/docs/recipes#textcat-manual) recipe.\n\nNote that the OpenAI API might return \"429 Too Many Request\" errors when\nrequesting too much data at once - in this case it's best to ensure you only\nrequest 100 or so examples at a time and take a look at the [API's rate\nlimits](https://help.openai.com/en/articles/5955598-is-api-usage-subject-to-any-rate-limits).\n\n#### Working with imbalanced data\n\nThe `textcat.openai.fetch` recipe is suitable for working with datasets where\nthere is severe class imbalance. Usually, you'd want to find examples of the\nrare class rather than annotating a random sample. From there, you want to\nupsample them to train a decent model and so on.\n\nThis is where large language models like OpenAI might help.\n\nUsing the [Reddit r/cooking dataset](data), we prompted OpenAI to\nlook for comments that resemble a food recipe. Instead of annotating 10,000\nexamples, we ran `textcat.openai.fetch` and obtained 145 positive classes. Out\nof those 145 examples, 114 turned out to be true positives (79% precision). We\nthen checked 1,000 negative examples and found 12 false negative cases (98%\nrecall). \n\nIdeally, once we fully annotated the dataset, we can train a supervised model\nthat is better to use than relying on zero-shot predictions for production. The\nrunning cost is low and it's easier to manage.\n\n### Exporting the annotations and training a text categorization model\n\nAfter you've curated a set of predictions, you can export the results with\n[`db-out`](https://prodi.gy/docs/recipes#db-out):\n\n```bash\npython -m prodigy db-out my_textcat_data  \u003e textcat_data.jsonl\n```\n\nThe format of the exported annotations contains all the data you need to train a\nsmaller model downstream. Each example in the dataset contains the original\ntext, the tokens, span annotations denoting the entities, etc.\n\nYou can also export the data to spaCy's [binary\nformat](https://spacy.io/api/data-formats#training), using\n[`data-to-spacy`](https://prodi.gy/docs/recipes#data-to-spacy). This format lets\nyou load in the annotations as spaCy `Doc` objects, which can be convenient for\nfurther conversion. The `data-to-spacy` command also makes it easy to train a\ntext categorization model with spaCy. First you export the data, specifying the\ntrain data as 20% of the total:\n\n```bash\n# For binary textcat\npython -m prodigy data-to-spacy ./data/annotations/ --textcat my_textcat_data -es 0.2\n# For multilabel textcat\npython -m prodigy data-to-spacy ./data/annotations/ --textcat-multilabel my_textcat_data -es 0.2\n```\nThen you can train a model with spaCy or [Prodigy](https://prodi.gy/docs/recipes/#training):\n\n```bash\npython -m spacy train ./data/annotations/config.cfg --paths.train ./data/annotations/train.spacy --paths.dev ./data/annotations/dev.spacy -o textcat-model\n```\n\nThis will save a model to the `textcat-model/` directory.\n\n## 📋 Terms \n\n### `terms.openai.fetch`: Fetch phrases and terms based on a query\n\nThis recipe generates terms and phrases obtained from a large language model. These\nterms can be curated and turned into patterns files, which can help with downstream annotation tasks. \n\n```bash\npython -m prodigy terms.openai.fetch query filepath [--options] -F ./recipes/openai_terms.py\n```\n\n|        Argument       | Type  | Description                                         | Default                         |\n|:---------------------:|-------|-----------------------------------------------------|---------------------------------|\n| `query`               | str   | Query to send to OpenAI                             |                                 |\n| `output_path`         | Path  | Path to save the output                             |                                 |\n| `--seeds`,`-s`        | str   | One or more comma-separated seed phrases.           | `\"\"`                            |\n| `--n`,`-n`            | int   | Minimum number of items to generate                 | `100`                           |\n| `--model`, `-m`       | str   | GPT-3 model to use for completion                   | `\"text-davinci-003\"`            |\n| `--prompt-path`, `-p` | Path  | Path to jinja2 prompt template                      | `templates/terms_prompt.jinja2` |\n| `--verbose`,`-v`      | bool  | Print extra information to terminal                 | `False`                         |\n| `--resume`, `-r`      | bool  | Resume by loading in text examples from output file | `False`                         |\n| `--progress`,`-pb`    | bool  | Print progress of the recipe.                       | `False`                         |\n| `--temperature`,`-t`  | float | OpenAI temperature param                            | `1.0`                           |\n| `--top-p`, `--tp`     | float | OpenAI top_p param                                  | `1.0`                           |\n| `--best-of`, `-bo`    | int   | OpenAI best_of param\"                               | `10`                            |\n| `--n-batch`,`-nb`     | int   | OpenAI batch size param                             | `10`                            |\n| `--max-tokens`, `-mt` | int   | Max tokens to generate per call                     | `100`                           |\n\n#### Example usage\n\nSuppose you're interested in detecting skateboard tricks in text, then you might want to start\nwith a term list of known tricks. You might want to start with the following query:\n\n```bash\n# Base behavior, fetch at least 100 terms/phrases\npython -m prodigy terms.openai.fetch \"skateboard tricks\" tricks.jsonl --n 100 --prompt-path templates/terms_prompt.jinja2 -F recipes/openai_terms.py\n```\n\nThis will generate a prompt to OpenAI that asks to try and generate at least 100 examples of \"skateboard tricks\".\nThere's an upper limit to the amount of tokens that can be generated by OpenAI, but this recipe will try and keep\ncollecting terms until it reached the amount specified. \n\nYou can choose to make the query more elaborate if you want to try to be more precise, but you can alternatively\nalso choose to add some seed terms via `--seeds`. These will act as starting examples to help steer OpenAI \nin the right direction.\n\n```bash\n# Base behavior but with seeds\npython -m prodigy terms.openai.fetch \"skateboard tricks\" tricks.jsonl --n 100 --seeds \"kickflip,ollie\" --prompt-path templates/terms_prompt.jinja2 -F recipes/openai_terms.py\n```\n\nCollecting many examples can take a while, so it can be helpful to show the progress, via `--progress` \nas requests are sent. \n\n```bash\n# Adding progress output as we wait for 500 examples\npython -m prodigy terms.openai.fetch \"skateboard tricks\" tricks.jsonl --n 500 --progress --seeds \"kickflip,ollie\" --prompt-path templates/terms_prompt.jinja2 -F recipes/openai_terms.py\n```\n\nAfter collecting a few examples, you might want to generate more. You can choose to continue from a\nprevious output file. This will effectively re-use those examples as seeds for the prompt to OpenAI.\n\n```bash\n# Use the `--resume` flag to re-use previous examples\npython -m prodigy terms.openai.fetch \"skateboard tricks\" tricks.jsonl --n 50 --resume --prompt-path templates/terms_prompt.jinja2 -F recipes/openai_terms.py\n```\n\nWhen the recipe is done, you'll have a `tricks.jsonl` file that has contents that look like this: \n\n```json\n{\"text\":\"pop shove it\",\"meta\":{\"openai_query\":\"skateboard tricks\"}}\n{\"text\":\"switch flip\",\"meta\":{\"openai_query\":\"skateboard tricks\"}}\n{\"text\":\"nose slides\",\"meta\":{\"openai_query\":\"skateboard tricks\"}}\n{\"text\":\"lazerflip\",\"meta\":{\"openai_query\":\"skateboard tricks\"}}\n{\"text\":\"lipslide\",\"meta\":{\"openai_query\":\"skateboard tricks\"}}\n...\n```\n\n### Towards Patterns \n\nYou now have a `tricks.jsonl` file on disk that contains skateboard tricks, but you cannot\nassume that all of these will be accurate. The next step would be to review the terms and you\ncan use the [`textcat.manual`](https://prodi.gy/docs/recipes/#textcat-manual) recipe that comes\nwith Prodigy for that. \n\n```bash\n# The tricks.jsonl was fetched from OpenAI beforehand\npython -m prodigy textcat.manual skateboard-tricks-list tricks.jsonl --label skateboard-tricks\n```\n\nThis generates an interface that looks like this: \n\n\u003cimg src=\"https://user-images.githubusercontent.com/1019791/212869305-58f1d087-e036-4eab-a818-df80aab68ce8.png\" width=\"600\" /\u003e\n\nYou can manually accept or reject each example and when you're done annotating you can export\nthe annotated text into a patterns file via the [`terms.to-patterns`](https://prodi.gy/docs/recipes/#terms-to-patterns) recipe.\n\n```bash\n# Generate a `patterns.jsonl` file.\npython -m prodigy terms.to-patterns skateboard-tricks-list patterns.jsonl --label skateboard-tricks --spacy-model blank:en\n```\n\nWhen the recipe is done, you'll have a `patterns.jsonl` file that has contents that look like this: \n\n```json\n{\"label\":\"skateboard-tricks\",\"pattern\":[{\"lower\":\"pop\"},{\"lower\":\"shove\"},{\"lower\":\"it\"}]}\n{\"label\":\"skateboard-tricks\",\"pattern\":[{\"lower\":\"switch\"},{\"lower\":\"flip\"}]}\n{\"label\":\"skateboard-tricks\",\"pattern\":[{\"lower\":\"nose\"},{\"lower\":\"slides\"}]}\n{\"label\":\"skateboard-tricks\",\"pattern\":[{\"lower\":\"lazerflip\"}]}\n{\"label\":\"skateboard-tricks\",\"pattern\":[{\"lower\":\"lipslide\"}]} \n...\n```\n\n### Known Limitations \n\nOpenAI has a hard limit on the prompt size. You cannot have a prompt larger than 4079 tokens. Unfortunately\nthat means that there is a limit to the size of term lists that you can generate. The recipe will report\nan error when this happens, but it's good to be aware of this limitation.\n\n## 📋 Prompt A/B evaluation\n\n### `ab.openai.prompts`: A/B evaluation of prompts\n\nThe goal of this recipe is to quickly allow someone to compare the quality of outputs from two prompts \nin a quantifiable and blind way. \n\n```bash\npython -m prodigy ab.openai.prompts dataset inputs_path display_template_path prompt1_template_path prompt2_template_path [--options] -F ./recipes/openai_ab.py\n```\n\n|        Argument        | Type  | Description                                          | Default              |\n|:----------------------:|-------|------------------------------------------------------|----------------------|\n| `dataset`              | str   | Prodigy dataset to save answers into                 |                      |\n| `inputs_path`          | Path  | Path to jsonl inputs                                 |                      |\n| `display_template_path`| Path  | Template for summarizing the arguments               |                      |\n| `prompt1_template_path`| Path  | Path to the first jinja2 prompt template             |                      |\n| `prompt2_template_path`| Path  | Path to the second jinja2 prompt template            |                      |\n| `--model`, `-m`        | str   | GPT-3 model to use for completion                    | `\"text-davinci-003\"` |\n| `--batch-size`, `-b`   | int   | Batch size to send to OpenAI API                     | `10`                 |\n| `--verbose`,`-v`       | bool  | Print extra information to terminal                  | `False`              |\n| `--no-random`,`-NR`    | bool  | Don't randomize which annotation is shown as correct | `False`              |\n| `--repeat`, `-r`       | int   | How often to send the same prompt to OpenAI          | `1`                  |\n\n#### Example usage\n\nAs an example, let's try to generate humorous haikus. To do that we first need to \nconstruct two jinja files that represent the prompt to send to OpenAI. \n\n##### `templates/ab/prompt1.jinja2`\n\n```\nWrite a haiku about {{topic}}.\n```\n\n##### `templates/ab/prompt2.jinja2`\n\n```\nWrite an incredibly hilarious haiku about {{topic}}. So funny!\n```\n\nYou can provide variables for these prompts by constructing a .jsonl file with the required\nparameters. In this case we need to make sure that `{{topic}}` is accounted for. \n\nHere's an example `.jsonl` file that could work. \n\n##### `data/ab_example.jsonl`\n\n```json\n{\"id\": 0, \"prompt_args\": {\"topic\": \"star wars\"}}\n{\"id\": 0, \"prompt_args\": {\"topic\": \"kittens\"}}\n{\"id\": 0, \"prompt_args\": {\"topic\": \"the python programming language\"}}\n{\"id\": 0, \"prompt_args\": {\"topic\": \"maths\"}}\n```\n\n\u003e **Note**\n\u003e\n\u003e All the arguments under `prompt_args` will be passed to render the jinja templates. \n\u003e The `id` is mandatory and can be used to identify groups in later analysis.\n\nWe're nearly ready to evaluate, but this recipe requires one final jinja2 template. \nThis one won't be used to generate a prompt, but it will generate a useful title \nthat reminds the annotator of the current task. Here's an example of such a template. \n\n##### `templates/ab/input.jinja2`\n\n```\nA haiku about {{topic}}.\n```\n\nWhen you put all of these templates together you can start annotating. The command below \nstarts the annotation interface and also uses the `--repeat 4` option. This will ensure\nthat each topic will be used to generate a prompt at least 4 times.\n\n```\npython -m prodigy ab.openai.prompts haiku data/ab_example.jsonl templates/ab/input.jinja2 templates/ab/prompt1.jinja2 templates/ab/prompt2.jinja2 --repeat 5 -F recipes/openai_ab.py\n```\n\nThis is what the annotation interface looks like:\n\n![](https://user-images.githubusercontent.com/1019791/216607308-97a0b82d-03ea-4d09-ab79-0ec6b26cc033.png)\n\nWhen you look at this interface you'll notice that the title template is rendered and that\nyou're able to pick from two options. Both options are responses from OpenAI that were\ngenerated by the two prompt templates. You can also see the `prompt_args` rendered in \nthe lower right corner of the choice menu.\n\nFrom here you can annotate your favorite examples and gather data that might help you\ndecide on which prompt is best.\n\n#### Results \n\nOnce you're done annotating you'll be presented with an overview of the results. \n\n```\n=========================== ✨  Evaluation results ===========================\n✔ You preferred prompt1.jinja2\n\nprompt1.jinja2   11\nprompt2.jinja2    5\n```\n\nBut you can also fetch the raw annotations from the database for further analysis. \n\n```\npython -m prodigy db-out haiku\n```\n\n## ❓ What's next?\n\nThere’s lots of interesting follow-up experiments to this, and lots of ways to adapt the basic idea to different tasks or data sets. We’re also interested to try out different prompts. It’s unclear how much the format the annotations are requested in might change the model’s predictions, or whether there’s a shorter prompt that might perform just as well. We also want to run some end-to-end experiments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexplosion%2Fprodigy-openai-recipes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fexplosion%2Fprodigy-openai-recipes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fexplosion%2Fprodigy-openai-recipes/lists"}