{"id":14964820,"url":"https://github.com/kddubey/cappr","last_synced_at":"2025-04-05T02:12:46.792Z","repository":{"id":143511815,"uuid":"605043639","full_name":"kddubey/cappr","owner":"kddubey","description":"Completion After Prompt Probability. Make your LLM make a choice","archived":false,"fork":false,"pushed_at":"2024-11-02T02:45:28.000Z","size":5386,"stargazers_count":75,"open_issues_count":0,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T23:07:21.941Z","etag":null,"topics":["huggingface","kv-cache","llamacpp","llm-inference","probability","prompt-engineering","text-classification","zero-shot"],"latest_commit_sha":null,"homepage":"https://cappr.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kddubey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-22T10:21:05.000Z","updated_at":"2025-03-22T01:52:48.000Z","dependencies_parsed_at":"2023-10-12T08:16:31.626Z","dependency_job_id":"662fc0ee-068a-4362-9c62-fc3340bef925","html_url":"https://github.com/kddubey/cappr","commit_stats":{"total_commits":434,"total_committers":1,"mean_commits":434.0,"dds":0.0,"last_synced_commit":"49fbb01dc1130c3fa7686f362e4b5c677077c50b"},"previous_names":[],"tags_count":39,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kddubey%2Fcappr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kddubey%2Fcappr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kddubey%2Fcappr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kddubey%2Fcappr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kddubey","download_url":"https://codeload.github.com/kddubey/cappr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247276189,"owners_count":20912288,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["huggingface","kv-cache","llamacpp","llm-inference","probability","prompt-engineering","text-classification","zero-shot"],"created_at":"2024-09-24T13:33:49.721Z","updated_at":"2025-04-05T02:12:46.767Z","avatar_url":"https://github.com/kddubey.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CAPPr: Completion After Prompt Probability\n\n[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg?logo=python\u0026style=for-the-badge)](https://www.python.org/downloads/release/python-380/)\n[![tests](https://img.shields.io/github/actions/workflow/status/kddubey/cappr/test.yml?style=for-the-badge\u0026logo=github\u0026label=tests)](https://github.com/kddubey/cappr/actions/workflows/test.yml)\n[![codecov](https://img.shields.io/codecov/c/github/kddubey/cappr?token=NYIL076PSM\u0026style=for-the-badge\u0026logo=codecov\u0026color=%2309BC00)](https://codecov.io/gh/kddubey/cappr)\n[![PyPI - Package Version](https://img.shields.io/pypi/v/cappr?logo=pypi\u0026style=for-the-badge\u0026color=orange)](https://pypi.org/project/cappr/)\n[![License](https://img.shields.io/badge/License-Apache_2.0-purple.svg?logo=apache\u0026style=for-the-badge)](https://opensource.org/licenses/Apache-2.0)\n\n\u003c!-- [![Documentation Status](https://readthedocs.org/projects/cappr/badge/?version=latest\u0026style=for-the-badge)](https://cappr.readthedocs.io/en/latest/?badge=latest) --\u003e\n\nMake your LLM pick from a list of choices. \u003cbr\u003e\nOr compute the probability of a completion given a prompt, which may be\n[useful](https://cappr.readthedocs.io/en/latest/related_work.html). \u003cbr\u003e\nSqueeze [more](https://cappr.readthedocs.io/en/latest/statistical_performance.html) out\nof open source LLMs.\n\n\n## Usage\n\n\u003cdetails\u003e\n\u003csummary\u003eUse a GGUF model\u003c/summary\u003e\n\n```python\nfrom llama_cpp import Llama\nfrom cappr.llama_cpp.classify import predict\n\nmodel = Llama(\"./TinyLLama-v0.Q8_0.gguf\", verbose=False)\n\nprompt = \"\"\"Gary told Spongebob a story:\nThere once was a man from Peru; who dreamed he was eating his shoe. He\nwoke with a fright, in the middle of the night, to find that his dream\nhad come true.\n\nThe moral of the story is to\"\"\"\n\ncompletions = (\n  \"look at the bright side\",\n  \"use your imagination\",\n  \"eat shoes\",\n)\n\npred = predict(prompt, completions, model)\nprint(pred)\n# use your imagination\n```\n\nSee [this page of the\ndocumentation](https://cappr.readthedocs.io/en/latest/select_a_language_model.html#llama-cpp)\nfor more info on using GGUF models.\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eUse a Hugging Face transformers model\u003c/summary\u003e\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom cappr.huggingface.classify import predict\n\nmodel_name = \"gpt2\"\nmodel = AutoModelForCausalLM.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\nprompt = \"Which planet is closer to the Sun: Mercury or Earth?\"\ncompletions = (\"Mercury\", \"Earth\")\n\npred = predict(prompt, completions, model_and_tokenizer=(model, tokenizer))\nprint(pred)\n# Mercury\n```\n\nSee [this page of the\ndocumentation](https://cappr.readthedocs.io/en/latest/select_a_language_model.html#hugging-face)\nfor more info on using ``transformers`` models.\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eCache instructions to save time\u003c/summary\u003e\n\nMany prompts start with the same set of instructions, e.g., a system prompt plus a\nhandful of example input-output pairs. Instead of repeatedly running the model on common\ninstructions, cache them so that future computations are faster.\n\nHere's an\nexample using\n[`cappr.huggingface.classify.cache_model`](https://cappr.readthedocs.io/en/latest/cappr.huggingface.classify.html#cappr.huggingface.classify.cache_model).\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom cappr.huggingface.classify import cache_model, predict\n\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\")\ntokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\nmodel_and_tokenizer = (model, tokenizer)\n\n# Create data\nprompt_prefix = '''Instructions: complete the sequence.\nHere are examples:\nA, B, C =\u003e D\n1, 2, 3 =\u003e 4\n\nComplete this sequence:'''\n\nprompts = [\"X, Y =\u003e\", \"10, 9, 8 =\u003e\"]\ncompletions = [\"7\", \"Z\", \"Hi\"]\n\n# Cache prompt_prefix because it's used for all prompts\ncached_model_and_tokenizer = cache_model(\n    model_and_tokenizer, prompt_prefix\n)\n\n# Compute\npreds = predict(\n    prompts, completions, cached_model_and_tokenizer\n)\nprint(preds)\n# ['Z', '7']\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eCompute token-level log-probabilities\u003c/summary\u003e\n\nHere's an example using\n[`cappr.huggingface.classify.log_probs_conditional`](https://cappr.readthedocs.io/en/latest/cappr.huggingface.classify.html#cappr.huggingface.classify.log_probs_conditional).\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom cappr.huggingface.classify import log_probs_conditional\n\n# Load model and tokenizer\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\")\ntokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n# Create data\nprompts = [\"x y\", \"a b c\"]\ncompletions = [\"z\", \"d e\"]\n\n# Compute\nlog_probs_completions = log_probs_conditional(\n    prompts, completions, model_and_tokenizer=(model, tokenizer)\n)\n\n# Outputs (rounded) next to their symbolic representation\n\nprint(log_probs_completions[0])\n# [[-4.5],        [[log Pr(z | x, y)],\n#  [-5.6, -3.2]]   [log Pr(d | x, y),    log Pr(e | x, y, d)]]\n\nprint(log_probs_completions[1])\n# [[-9.7],        [[log Pr(z | a, b, c)],\n#  [-0.2, -0.03]]  [log Pr(d | a, b, c), log Pr(e | a, b, c, d)]]\n```\n\nEfficiently aggregate these log-probabilities using\n[`cappr.utils.classify.agg_log_probs`](https://cappr.readthedocs.io/en/latest/cappr.utils.classify.html#cappr.utils.classify.agg_log_probs).\n\nFor a slightly more advanced demo, see\n[`./demos/huggingface/dpo.ipynb`](./demos/huggingface/dpo.ipynb).\n\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eExtract the final answer from a step-by-step completion\u003c/summary\u003e\n\nStep-by-step and chain-of-thought prompts are highly effective ways to get an LLM to\n\"reason\" about more complex tasks. But if you need a structured output, a step-by-step\ncompletion is unwieldy. Use CAPPr to extract the final answer from these types of\ncompletions, given a list of possible answers.\n\nSee this idea in action [here in the\ndocumentation](https://cappr.readthedocs.io/en/latest/select_a_prompt_completion_format.html#wrangle-step-by-step-completions).\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eRun in batches, predict probabilities\u003c/summary\u003e\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom cappr.huggingface.classify import predict_proba\n\n# Load a model and its tokenizer\nmodel_name = \"gpt2\"\nmodel = AutoModelForCausalLM.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\nprompts = [\n    \"Stephen Curry is a\",\n    \"Martina Navratilova was a\",\n    \"Dexter, from the TV Series Dexter's Laboratory, is a\",\n    \"LeBron James is a\",\n]\n\n# Each of the prompts could be completed with one of these:\nclass_names = (\"basketball player\", \"tennis player\", \"scientist\")\nprior =       (      1/6,                1/6,            2/3    )\n# Say I expect most of my data to have scientists\n\n# Run CAPPr\npred_probs = predict_proba(\n    prompts=prompts,\n    completions=class_names,\n    model_and_tokenizer=(model, tokenizer),\n    batch_size=2,  # whatever fits on your CPU/GPU\n    prior=prior,\n)\n\n# pred_probs[i,j] = probability that prompts[i] is classified as class_names[j]\nprint(pred_probs.round(1))\n# [[0.5 0.3 0.2]\n#  [0.3 0.6 0.2]\n#  [0.1 0.1 0.8]\n#  [0.8 0.2 0. ]]\n\n# For each prompt, which completion is most likely?\npred_class_idxs = pred_probs.argmax(axis=-1)\npreds = [class_names[pred_class_idx] for pred_class_idx in pred_class_idxs]\nprint(preds)\n# ['basketball player',\n#  'tennis player',\n#  'scientist',\n#  'basketball player']\n```\n\u003c/details\u003e\n\n\n\u003cdetails\u003e\n\u003csummary\u003eRun in batches, where each prompt has a different set of possible completions\n\u003c/summary\u003e\n\nAgain, let's predict probabilities.\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom cappr.huggingface.classify import predict_proba_examples\nfrom cappr import Example\n\n# Load a model and its tokenizer\nmodel_name = \"gpt2\"\nmodel = AutoModelForCausalLM.from_pretrained(model_name)\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Create a sequence of Example objects representing your classification tasks\nexamples = [\n    Example(\n        prompt=\"Jodie Foster played\",\n        completions=(\"Clarice Starling\", \"Trinity in The Matrix\"),\n    ),\n    Example(\n        prompt=\"Batman, from Batman: The Animated Series, was played by\",\n        completions=(\"Pete Holmes\", \"Kevin Conroy\", \"Spongebob!\"),\n        prior=      (     1/3      ,      2/3     ,      0      ),\n    ),\n]\n\n# Run CAPPr\npred_probs = predict_proba_examples(\n    examples, model_and_tokenizer=(model, tokenizer)\n)\n\n# pred_probs[i][j] = probability that examples[i].prompt is classified as\n# examples[i].completions[j]\nprint([example_pred_probs.round(2) for example_pred_probs in pred_probs])\n# [array([0.7, 0.3]),\n#  array([0.03, 0.97, 0.  ])]\n\n# For each example, which completion is most likely?\npred_class_idxs = [\n    example_pred_probs.argmax() for example_pred_probs in pred_probs\n]\npreds = [\n    example.completions[pred_class_idx]\n    for example, pred_class_idx in zip(examples, pred_class_idxs)\n]\nprint(preds)\n# ['Clarice Starling',\n#  'Kevin Conroy']\n```\n\u003c/details\u003e\n\n\nSee the [`demos`](https://github.com/kddubey/cappr/blob/main/demos/) for demonstrations\nof slightly harder classification tasks.\n\nFor CAPPr, GPTQ models are the most computationally performant. These models are\ncompatible with `cappr.huggingface.classify`. See [this page of the\ndocumentation](https://cappr.readthedocs.io/en/latest/select_a_language_model.html#hugging-face)\nfor more info on using these models.\n\n\n## Documentation\n\nhttps://cappr.readthedocs.io\n\n\n## Installation\n\nSee [this page of the\ndocumentation](https://cappr.readthedocs.io/en/latest/installation.html).\n\n\n## Related work\n\nSee [this page of the\ndocumentation](https://cappr.readthedocs.io/en/latest/related_work.html).\n\n\n## Motivation\n\nReduce engineering complexity.\n\nSee [this page of the\ndocumentation](https://cappr.readthedocs.io/en/latest/motivation.html) for more info.\n\n\n## Performance\n\n[Statistical performance](https://cappr.readthedocs.io/en/latest/statistical_performance.html)\n\n[Computational performance](https://cappr.readthedocs.io/en/latest/computational_performance.html)\n\n\n## How it works\n\nYou input a `prompt` string, a `end_of_prompt` string (a whitespace or empty) and a set\nof candidate `completion` strings such that the string—\n\n```python\n{prompt}{end_of_prompt}{completion}\n```\n\n—is a naturally flowing thought. CAPPr picks the `completion` which is mostly likely to\nfollow `prompt` by computing the—\n\n\u003e **C**ompletion\u003cbr\u003e\n  **A**fter\u003cbr\u003e\n  **P**rompt\u003cbr\u003e\n  **Pr**obability\u003cbr\u003e\n\n—as fleshed out in my [question on Cross\nValidated](https://stats.stackexchange.com/q/601159/337906).\n\n\n## Local development\n\nSee [this page of the documentation](https://cappr.readthedocs.io/en/latest/local.html).\n\n\n## Todo\n\nI'm dumping todos here:\n\n[Code changes](https://github.com/users/kddubey/projects/1/views/1)\n\n[Reseach experiments](https://github.com/users/kddubey/projects/2)\n\nFeel free to raise issues ofc\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkddubey%2Fcappr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkddubey%2Fcappr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkddubey%2Fcappr/lists"}