{"id":19401066,"url":"https://github.com/google-research/prompt-tuning","last_synced_at":"2025-05-15T08:05:54.318Z","repository":{"id":37072514,"uuid":"403986458","full_name":"google-research/prompt-tuning","owner":"google-research","description":"Original Implementation of Prompt Tuning from Lester, et al, 2021","archived":false,"fork":false,"pushed_at":"2025-03-06T04:47:38.000Z","size":3571,"stargazers_count":673,"open_issues_count":9,"forks_count":60,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-03T04:08:51.745Z","etag":null,"topics":["flax","jax","language-model","machine-learning","nlp","prompt-tuning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/google-research.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-07T13:21:08.000Z","updated_at":"2025-03-10T11:14:35.000Z","dependencies_parsed_at":"2024-05-22T03:30:22.975Z","dependency_job_id":"3a61ef90-e1f5-43b8-bb93-910fc129299c","html_url":"https://github.com/google-research/prompt-tuning","commit_stats":{"total_commits":89,"total_committers":12,"mean_commits":7.416666666666667,"dds":0.4719101123595506,"last_synced_commit":"77a7aecf1f0b26ffe813c0450cc6b33b4a8aaf3d"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fprompt-tuning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fprompt-tuning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fprompt-tuning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/google-research%2Fprompt-tuning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/google-research","download_url":"https://codeload.github.com/google-research/prompt-tuning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248166865,"owners_count":21058481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["flax","jax","language-model","machine-learning","nlp","prompt-tuning"],"created_at":"2024-11-10T11:16:59.977Z","updated_at":"2025-04-10T06:19:53.558Z","avatar_url":"https://github.com/google-research.png","language":"Python","funding_links":[],"categories":["Papers","Python"],"sub_categories":["Language Models"],"readme":"# Prompt Tuning\n\nThis is the code to reproduce the experiments from the EMNLP 2021 paper \"The\nPower of Scale for Parameter-Efficient Prompt Tuning\"\n[(Lester et al., 2021)](https://aclanthology.org/2021.emnlp-main.243/).\n\nThese models are built on [T5X](https://github.com/google-research/t5x), which\ndefines the model and training loop;\n[Flaxformer](https://github.com/google/flaxformer), which defines the actual\nmodel computation; [Flax](https://github.com/google/flax), which defines the low\nlevel model layers; and [Jax](https://github.com/jax-ml/jax), which provides the\nactual execution. Details of our implementation can be found\n[here](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/implementation.md).\n\n## Table of Contents\n\n*   [Installation](#installation)\n*   [Training a Prompt](#training-a-prompt)\n    *   [Training a Prompt on a Pod Slice](#training-a-prompt-on-a-pod-slice)\n    *   [Custom Dependencies](#custom-dependencies)\n*   [Inference with a Prompt](#inference-with-a-prompt)\n*   [Model Configuration](#model-configuration)\n*   [Prompt Initialization](#prompt-initialization)\n*   [Released Model Checkpoints](#released-model-checkpoints)\n*   [Released Prompts](#released-prompts)\n*   [Extra Resources](#extra-resources)\n*   [ How to Cite](#how-to-cite)\n\n## Installation\n\n1.  Follow the first 3 steps in the\n    [T5X installation instructions](https://github.com/google-research/t5x#installation)\n    to create a cloud TPU VM. Also follow step 5 and create a Google Cloud\n    Storage (GCS) bucket. We will read and write data to this bucket using a URI\n    formatted like `gs://{bucket-name}/path/to/item/in/bucket`. This is where we\n    will store cached datasets as well as model checkpoints and results. For\n    ease of reference, some of the most common cloud commands for interacting\n    with the TPU VMs are\n\n```sh\n# Create a Cloud TPU VM\n$ gcloud alpha compute tpus tpu-vm create ${TPU_NAME} \\\n    --zone ${ZONE} \\\n    --accelerator-type v3-8 \\\n    --version v2-alpha\n\n# SSH into a Cloud TPU VM\n$ gcloud alpha compute tpus tpu-vm ssh ${TPU_NAME} --zone ${ZONE}\n\n# Delete a Cloud TPU VM\n$ gcloud alpha compute tpus tpu-vm delete ${TPU_NAME} --zone ${ZONE}\n```\n\n2.  You should now be at the command-line of the TPU VM instance. Clone the\n    Prompt Tuning repository.\n\n```sh\ngit clone --branch=main https://github.com/google-research/prompt-tuning\ncd prompt-tuning\n```\n\n3.  Install the Prompt Tuning library.\n\n```sh\npython3 -m pip install .[tpu] -f https://storage.googleapis.com/jax-releases/libtpu_releases.html\n```\n\nIf you run into an error where pip tries to install earlier and earliler\nversions of dependencies (TensorFlow for example) until it tries to install\nversion `0.0.0` and then fails try adding `--use-deprecated=legacy-resolver` to\nthe install command. This error is related to required versions betweens\ndependencies and the behavior is often called backtracking. If you use the\nflag, it is possible that incompatible versions of libraries may be installed\nand you should look out for warnings about mismatches in the output of the\ninstall command.\n\n*Note:* If you plan to hack on the internals of prompt tuning and need an\neditable install (so changes in the cloned code are used when you run training)\nrun `pip` with the `-e` flag and you may need to delete the `pyproject.toml`\nfile if you are getting errors during installation.\n\nTo run the tests, install the package with the `[test]` (`python3 -m pip install\n.[test] ...`) option and then run `python3 -m pytest` from the root of the\ncloned repository.\n\n## Training a Prompt\n\nTraining a prompt is similar to\n[fine-tuning a model with T5X](https://github.com/google-research/t5x/blob/main/README.md#fine-tuning);\nthe main difference is that we have our own set of Prompt Tuning configuration\nfiles to use.\n\nWe provide a demo script (`prompt_tuning/scripts/sst2-demo.sh`) that has all the\nrequired parts for training a prompt. You can use this as a starting point, or\nset `MODEL_DIR` and `TFDS_DATA_DIR` environment variables with paths to your\nGoogle Cloud Storage bucket to run this script directly.\n\n```sh\n./prompt-tuning/prompt_tuning/scripts/sst2-demo.sh\n```\n\nTo help with iteration speed, we tend to specify a lot more options the command\nline rather than bundling all of the configuration into a single gin file. A few\noptions of note:\n\n*   `--gin_search_paths` :: a comma separated list of directories to use as path\n    prefixes for gin files. We can use `prompt_tuning.scripts.find_module\n    ${module}` to find the install location of libraries that bundle\n    configurations with them.\n*   `--gin_file` :: The gin file to load. We tend to use paths relative starting\n    with the library they are installed with, i.e.\n    `prompt_tuning/configs/models/t5_1_1_base_prompt.gin` over\n    `models/t5_1_1_base_prompt.gin` to avoid any confusion. Using the flag\n    multiple time can be used to specify multiple gin files that will get merged\n    together. Any configurations options set in multiple files will use the\n    value from the last file in the list.\n*   `--gin.{PARAM}={VALUE}` :: This general override flag will set `PARAM` to\n    `VALUE`. This can be used to easily set configuration options without\n    requiring them to be actual command line arguments. For example.\n    `--gin.utils.SaveCheckpointConfig.keep=20` will save the last 20\n    checkpoints.\n\n### Training a Prompt on a Pod Slice\n\nAs models get larger, xl and xxl for example, they do not fit on the 8 TPUs that\ncome with a single TPU VM. In these cases we will need a slice of a TPU pod\n(more information about TPU architecture and available configurations can be\nfound [here](https://cloud.google.com/tpu/docs/system-architecture-tpu-vm)). The\nmain difference between training a prompt on a single TPU VM and on a Pod slice\nis that we now have multiple TPU VMs and will run the same SPMD JAX each VM,\nthis page has more information on\n[multi-host](https://jax.readthedocs.io/en/latest/multi_process.html) JAX\nprograms. [This guide](https://cloud.google.com/tpu/docs/jax-pods) gives a quick\nintroduction to running JAX programs on a TPU Pod slice, but we will hit main\npoints here.\n\n1.  Create a TPU Pod slice.\n    [This page](https://cloud.google.com/tpu/docs/types-zones#us) lists which\n    accelerator types are available in which zones. This is the same as\n    creating a TPU VM above, except that we are requesting 32 TPUs instead of\n    8.\n\n```sh\n$ gcloud alpha compute tpus tpu-vm create ${TPU_NAME} \\\n    --zone ${ZONE} \\\n    --accelerator-type v3-32 \\\n    --version v2-alpha\n```\n\n2.  Install the Prompt Tuning library. Given that we now have 4 TPU VM, each one\n    has 8 of out TPUs, we want to forgo ssh'ing directly into the VM, as we\n    would need to do that for each host. Instead, the Google Cloud SSH command\n    allows use to specify a command to run with the `--command=` flag and that\n    it should be run on all our VMs (called workers) with `--worker=all`.\n\n```sh\n$ gcloud alpha compute tpus tpu-vm ssh ${TPU_NAME} \\\n  --zone ${ZONE} \\\n  --worker=all \\\n  --command=\"git clone --branch=main https://github.com/google-research/prompt-tuning \u0026\u0026 cd prompt-tuning \u0026\u0026 \"\npython3 -m pip install . -f https://storage.googleapis.com/jax-releases/libtpu_releases.html\n```\n\n3.  Write the script to train your prompt. We included a demo script\n    (`/prompt_tuning/scripts/sst2-xxl-demo.sh`) the trains an prompt to solve\n    the [SST2](https://nlp.stanford.edu/sentiment/) dataset using T5 1.1 lm100k\n    XXL. You can use this as a starting point or just fill in the paths to your\n    Google Cloud Storage bucket to specify where you want to save your results\n    (`MODEL_DIR`) and where to cache TFDS data (`TFDS_DATA_DIR`), or set them as\n    environment variables.\n\n4.  Copy your training script each worker. If this is your first time running\n    `scp` you may get error, run the `ssh-add /.../.ssh/google_compute_engine`\n    command from the error message and try again.\n\n```sh\n$ gcloud alpha compute tpus tpu-vm scp sst2-xxl-demo.sh ${TPU_NAME}: \\\n  --zone=${ZONE}\n  --worker=all\n```\n\n5.  Execute your training script.\n\n```sh\n$ gcloud alpha compute tpus tpu-vm ssh ${TPU_NAME} \\\n  --zone ${ZONE} \\\n  --worker=all \\\n  --command=\"./sst2-xxl-demo.sh\"\n```\n\nIf one of the workers has an error during training, you will be left with\nprocesses that are using the TPUs on the other workers. This will stop you from\nrestarting your job until those processes a terminated and release the TPU. The\nfollowing command should end all these processes. You may see the `kill` command\nman page come back from the worker who had the initial error.\n\n```sh\n$ gcloud alpha compute tpus tpu-vm ssh ${TPU_NAME} \\\n  --zone ${ZONE} \\\n  --worker=all \\\n  --command=\"sudo lsof -t /dev/accel0 | xargs kill -9\"\n```\n\n### Custom Dependencies\n\nTo train prompts using custom parts, like your own dataset, follow the\n[T5X Instructions on Custom Components](https://github.com/google-research/t5x#custom-components)\n\nIf you package your code as a pip-installable python package, you won't be bound\nto a single directory, and you can use `python3 -m\nprompt_tuning.scripts.find_module {your_module}` to help set the\n`gin_search_paths` so that gin configs bundled in your library are findable.\n*Note:* If you do plan to bundle gin configs in an installable package, make\nsure that the directories that contain the config files have an `__init__.py` as\ngin requires files to be in a python package.\n\nIf parts of your custom components are gin configurable, they need to be\nexplicitly imported in your gin files; if they end up getting imported after the\ngin files are parsed, they will cause an error. If none of your dependencies\ncontain gin configurables, you can avoid writing a gin file by passing\n`--gin.MIXTURE_OR_TASK_MODULE=\"'path.to.your.module'`. This will automatically\nimport your module and is convenient for when all you are doing is swapping out\ndatasets.\n\n\n## Inference with a Prompt\n\nOur suggested way to do inference with a prompt is to load the original\ncheckpoint used to initialize the model, and the prompt from a file. As\nexplained in this section about\n[partial loading](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/implementation.md#partial-loading)\nT5X supports loading some model parameters while initializing others from\nscratch. We use this in conjunction with the `from_array` prompt initializer to\nreload the frozen parameters from the original checkpoint and the prompt file a\nfile. The\n[`configs/runs/prompt_eval.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/runs/prompt_eval.gin)\nsets up this configuration for you; you just have to supply a `PROMPT_FILE`. If\nyour model was trained with any of the `prompts/` config files, you can remove\nthem from the arguments to the evaluation script.\n\nThe included `sst2-demo-eval.sh` script shows an example of doing evaluation\nthis way. All that is needed is to set `EVAL_DIR` and `TFDS_DATA_DIR`\nenvironment variables to the paths to store the output of evaluation and the\ntensorflow datasets cache respectivly.\n\nIn T5X, the evaluation script assumes that your dataset has labels and outputs\nthe final results from your dataset's metric functions. The inference script\ndoes not require labels and instead outputs your model's prediction. We\ninclude an analogous\n[`prompt_infer.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/runs/prompt_infer.gin)\nfile to use with the inference script.\n\nIf you want to do inference or evaluation with the t5x checkpoint that is\nproduced from a prompt tuning training run, you can use the `(eval|infer).gin`\nconfig from T5X directly. You will need to update the\n`utils.RestoreChekcpointConfig` though. You should set `path` to the new\ncheckpoint, `assignment_map=()` and `fallback_to_scratch=False`.\n\n## Model Configuration\n\nAll model, training, evaluation, saving, restoring, etc. configuration is done\nvia gin. See\n[the gin-config repository](https://github.com/google/gin-config#readme) for a\ngeneral introduction to gin and\n[this primer](https://github.com/google-research/t5x/blob/main/gin-primer.md)\n\nWe follow the T5X configuration layout:\n\n*   `runs/` :: contains configs for the actual training of model. This is where\n    things like dataset and evaluation configuration go.\n*   `architectures/` :: contains configs for how the model works. This is where\n    things like encoder-decoder vs decoder-only and embedding sharing are\n    configured.\n*   `models/` :: contains configs that set model specific parameters like the\n    number of layers or the size of the embedding table. It also configures\n    things like the T5X model wrapper used.\n  *   `models/decoding/` :: contains easy to use configs to swap out how the\n      model generates text during inference, includes configs for beam search\n      and nucleus sampling.\n  *   `models/sizes/` :: contains the various settings to create models of\n      different sizes, these are combined with the default versions to create a\n      sized version, for example,`t5_1_1_prompt.gin` + `sizes/large.gin`\n      creates a T5 1.1 Large model. Some common combinations already available\n      as gin files with the right includes (`t5_1_1_large_prompt.gin` for our\n      example above). _Note:_ These size files need to come __after__ the main\n      model file.\n*   `prompts/` :: Our extra directory contains configs that set the `PROMPT` gin\n    variable, allowing for easy switching of the prompt initialization based\n    which prompt file is added as a `--gin_file` argument (it needs to come\n    after the `models/` gin file),\n\n### Order of gin config files\n\nWhen specifying `--gin_file` arguments in the command line, the order matters.\nThe general order in which the gin files must be specified is:\n\n1.  `models/*.gin`\n1.  `prompts/*.gin`\n1.  `models/sizes/*.gin*`\n1.  `models/decoding/*.gin`\n1.  `runs/*.gin`\n\n\n### Required Fields\n\nT5X has some required fields like `MIXTURE_OR_TASK_NAME` or\n`TASK_FEATURE_LENGTHS`. We add two more:\n\n*   `PROMPT_LENGTH` :: The length of the prompt we are using, this is used in a\n    few different places to we require it as a gin macro we can reference in\n    multiple places and ensure the values are in sync.\n*   `PROMPT` :: This is the configuration of the actual prompt module that will\n    be used in the Flaxformer `PromptX` subclasses.\n\n*Note:* Prompt Tuning does not currently support packing of examples. This means\nthat our max target length only need to be long enough to fit the target for\neach example. This means our `targets` key in the `TASK_FEATURE_LENGTHS` mapping\ncan be much shorter, for example around 4 for many SuperGLUE\n[(Wang et al., 2019)](https://arxiv.org/abs/1905.00537) tasks, compared to 62\nwhich is what the P5X default is.\n\n## Prompt Initialization\n\nThere are several options for the initialization of the prompt parameter. We\nsupport the various methods in section 3.2 our\n[paper](https://aclanthology.org/2021.emnlp-main.243.pdf), as well as\ninitialization from a file. The latter allows one to do things like train on\nBoolQ starting from a prompt learned on MNLI.\n\nAll initializers follow the flax initializer API of being a parameterized\nfunction that returns a closure over the initialization function. The actual\ninitialization function always has the signature of\n\n```python\ndef initializer(rng: Array, shape: Sequence[int]) -\u003e Array:\n  ...\n```\n\nWe provide each initialization scheme as a gin configuration file in the\n`configs/prompts` directory. They can be used by including the gin file with the\n`--gin_file=path/to/configs/prompts/scheme.gin`. This file needs to come\n**after** the main model file, otherwise the default (random uniform) method\nwill overwrite the one you selected. Some of these initialization methods will\nrequire you to set extra gin values either though an override flag of in one of\nyour gin files.\n\n**Random Uniform**\n\nA standard, random initialization similar to what people have used for embedding\ninitialization. This is the default and no gin file is required. The scale of\nthe random values can be adjusted by overridding\n`prompt_init/linen.initializers.uniform.scale=N`.\n\n**Sampled Vocab**\n\nSample a token embedding to use as initialization for each prompt position with\nthe `from_sample_of_embeddings` initializer. You can limit the sampling to the\nfirst `n` embeddings with the\n`prompt_init/prompts.from_samples_of_embeddings.population_size` parameter.\n\nThis can be used with\n[`--gin_file=prompt_tuning/configs/prompts/from_sampled_vocab.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/prompts/from_sampled_vocab.gin).\nThis method uses the embedding table extracted from the initial model\ncheckpoint. You can also provide your own embedding file with\n[`--gin_file=prompt_tuning/configs/prompts/from_sampled_vocab_numpy.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/prompts/from_sampled_vocab_numpy.gin).\nThis method requires that you provide a value for `EMBEDDING_FILE` that is a\nnumpy array of the model's embedding table. This can be extracted from a model\ncheckpoint using\n[prompt_tuning.scripts.extract_variable](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/scripts/extract_variable.py).\n\n**Class Label**\n\nWe support initializing prompt timesteps with the embedding of class labels\n(a.k.a. *verbalizers*) via the `from_embedded_list` initializer. Users providing\na list of words (class labels) to use. Each words is tokenized by a provided\nvocab; embedded with a provided vocab table; aggregated, if need be, across\nsub-tokens; and used to initialize a prompt time-step. If the provided tokens\ndon't cover the full prompt length, the missing tokens are initialized using\nthe provided fall back initializer.\n\nWe can match the paper, where unfilled prompt tokens are filled by sampling from\nthe embedding table, by composing this initialization with the one above. It can\nbe used with\n[`--gin_file=prompt_tuning/configs/prompts/from_class_labels.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/prompts/from_class_labels.gin).\nThis requires setting `CLASS_LABELS`, which is a list of the words that you want\nto embed as prompt initialization. You can also provide your own embedding file\n(which is the same as above) with\n[`--gin_file=prompt_tuning/configs/prompts/from_class_labels_numpy.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/prompts/from_class_labels_numpy.gin).\nThis additionally requires setting `EMBEDDING_FILE`.\n\n**From String**\n\nWe also support initializing a prompt with the embedding of some string, often\nused to start from a discrete prompt or a task description. This uses the\n`from_embedded_string` initializer. The string is tokenized by the provided\nvocabulary, each token is looked up in the provided embedding table, and the\nresulting embedded representation of the string is used as a prompt\ninitialization. If the provided tokens don't cover the full prompt length, the\nmissing tokens are initialized using the provided fall back initializer.\n\n_Note:_ The vocabulary just converts the string into a sequence of ids, you\nwill need to ensure that the string matches the result of any text formatting\n(spaces around punctuation, etc.) that your SeqIO task does.\n\n**From File**\n\nYou can also load a prompt from a file with the `from_array` initializer to\nenable transfer across tasks. This is done with\n[`--gin_file=prompt_tuning/configs/prompts/from_file.gin`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/configs/prompts/from_file.gin).\nThis requires setting `PROMPT_FILE` with a path to the Numpy file with the\nprompt to load. Numpy versions of the prompt are emitted by default when training,\nbut the prompt can also be extracted with the script mentioned above.\n\n## Released Model Checkpoints\n\nWe have released T5X native checkpoints of the T5 1.1 checkpoints that have had\n100K steps of language model adaptation.\n\n*   **t5_1_1_lm100k_small** (~77 million parameters):\n    [gs://t5-data/pretrained_models/t5x/t5_1_1_lm100k_small/checkpoint_1100000](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5x/t5_1_1_lm100k_small/checkpoint_1100000)\n*   **t5_1_1_lm100k_base** (~250 million parameters):\n    [gs://t5-data/pretrained_models/t5x/t5_1_1_lm100k_base/checkpoint_1100000](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5x/t5_1_1_lm100k_base/checkpoint_1100000)\n*   **t5_1_1_lm100k_large** (~800 million parameters):\n    [gs://t5-data/pretrained_models/t5x/t5_1_1_lm100k_large/checkpoint_1100000](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5x/t5_1_1_lm100k_large/checkpoint_1100000)\n*   **t5_1_1_lm100k_xl** (~3 billion parameters):\n    [gs://t5-data/pretrained_models/t5x/t5_1_1_lm100k_xl/checkpoint_1100000](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5x/t5_1_1_lm100k_xl/checkpoint_1100000)\n*   **t5_1_1_lm100k_xxl** (~11 billion parameters):\n    [gs://t5-data/pretrained_models/t5x/t5_1_1_lm100k_xxl/checkpoint_1100000](https://console.cloud.google.com/storage/browser/t5-data/pretrained_models/t5x/t5_1_1_lm100k_xxl/checkpoint_1100000)\n\nThese are converted from the public\n[Mesh TensorFlow checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k).\n\n## Released Prompts\n\nWe have released pretrained prompts on a variety of tasks, and plan to add to\nthem over time.\n\nPrompts can be found in the\n[`pretrained_prompts`](https://github.com/google-research/prompt-tuning/tree/main/prompt_tuning/pretrained_prompts)\ndirectory. From there each sub-directory groups prompts by the model they were\ntrained for. The easiest way to reference these prompts that are bundled with\nthe library is:\n\n```sh\n  --PROMPT_FILE=`python3 -m prompt_tuning.scripts.find_module prompt_tuning`/pretrained_prompts/{MODEL_SIZE}/{PROMPT}.npy\n```\n\nDue to the inherent randomness of parallel computation, there are a few settings\nthat need to match between training and evaluation to get the exact same\nnumbers. Each model sub-directory has a `README.md` the specifies what these\nsettings should be. The most important settings to match are batch size, TPU\ntopology, and model parallelism partitioning. The tables include the scores you\nshould expect to see if you use these prompts in `t5x.eval`\n\n\n## Extra Resources\n\nThis is a collection of additional resources about Prompt Tuning.\n\n*   Presentations:\n    *   EMNLP:\n        [Video](https://blester125.com/presentations/prompt-tuning-emnlp-2021.html)\n        [Slides](https://blester125.com/static/presentations/slides/Prompt-Tuning-Presentation-EMNLP-2021.pdf)\n        [Poster](https://blester125.com/static/presentations/posters/Prompt-Tuning-EMNLP-2021-Poster.pdf)\n\n\n## How to Cite\n\nIf you use this work as a jumping off point, please cite\n\n```bibtex\n@inproceedings{lester-etal-2021-power,\n    title = \"The Power of Scale for Parameter-Efficient Prompt Tuning\",\n    author = \"Lester, Brian  and\n      Al-Rfou, Rami  and\n      Constant, Noah\",\n    booktitle = \"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing\",\n    month = nov,\n    year = \"2021\",\n    address = \"Online and Punta Cana, Dominican Republic\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.emnlp-main.243\",\n    doi = \"10.18653/v1/2021.emnlp-main.243\",\n    pages = \"3045--3059\",\n}\n```\n\n## Note\n\nThis is not an officially supported Google product.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fprompt-tuning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgoogle-research%2Fprompt-tuning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgoogle-research%2Fprompt-tuning/lists"}