{"id":15064325,"url":"https://github.com/knowledgeforge/keymaker","last_synced_at":"2025-06-25T02:04:52.514Z","repository":{"id":179311226,"uuid":"663289457","full_name":"KnowledgeForge/keymaker","owner":"KnowledgeForge","description":"The most powerful and extensible way to control the output of large language models.","archived":false,"fork":false,"pushed_at":"2024-04-06T00:41:34.000Z","size":691,"stargazers_count":6,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T12:15:39.948Z","etag":null,"topics":["chatgpt","constraint","gpt","grammar","lark","llm","openai","regex","transformers"],"latest_commit_sha":null,"homepage":"https://keymaker.headjack.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KnowledgeForge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-07T01:31:45.000Z","updated_at":"2024-07-08T13:56:23.000Z","dependencies_parsed_at":"2025-02-17T05:30:59.545Z","dependency_job_id":null,"html_url":"https://github.com/KnowledgeForge/keymaker","commit_stats":null,"previous_names":["knowledgeforge/keymaker"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KnowledgeForge/keymaker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KnowledgeForge%2Fkeymaker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KnowledgeForge%2Fkeymaker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KnowledgeForge%2Fkeymaker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KnowledgeForge%2Fkeymaker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KnowledgeForge","download_url":"https://codeload.github.com/KnowledgeForge/keymaker/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KnowledgeForge%2Fkeymaker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261789229,"owners_count":23209774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","constraint","gpt","grammar","lark","llm","openai","regex","transformers"],"created_at":"2024-09-25T00:15:23.919Z","updated_at":"2025-06-25T02:04:52.492Z","avatar_url":"https://github.com/KnowledgeForge.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n  \u003ca href=\"https://keymaker.headjack.ai\"\u003e\n    \u003cimg src=\"https://raw.githubusercontent.com/KnowledgeForge/keymaker/ce8e1d701c47081568be5c21fb4eaa07ed561149/docs/static/images/keymaker-logo.svg\" alt=\"Logo\" width=\"150\"\u003e\n  \u003c/a\u003e\n  \u003ch3 align=\"center\"\u003eKeymaker\u003c/h3\u003e\n  \u003cp align=\"center\"\u003e\n    The most powerful, flexible and extensible way to control the output of large language models.\n    \u003cbr\u003e\n    \u003ca href=\"https://keymaker.headjack.ai\"\u003e\u003cstrong\u003eExplore the docs »\u003c/strong\u003e\u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://github.com/KnowledgeForge/keymaker/actions/workflows/python-checks.yml\" target=\"_blank\"\u003e\n      \u003cimg src=\"https://github.com/KnowledgeForge/keymaker/actions/workflows/python-checks.yml/badge.svg?branch=main\" alt=\"Tests\"\u003e\n    \u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://app.netlify.com/sites/headjack-keymaker/deploys\" target=\"_blank\"\u003e\n      \u003cimg src=\"https://api.netlify.com/api/v1/badges/9b36dc49-67a6-4254-bb2a-6ad6a2b7417d/deploy-status\" alt=\"Netlify\"\u003e\n    \u003c/a\u003e\n    \u003cbr\u003e\n    \u003cbr\u003e\n    \u003ca href=\"https://github.com/KnowledgeForge/keymaker/issues\"\u003eReport Bug\u003c/a\u003e\n    \u003cbr\u003e\n  \u003c/p\u003e\n\u003c/div\u003e\n\n# TLDR Simple Example\n```python\n# This example assumes you have set either OPENAI_API_KEY env var or openai.api_key\n\nfrom keymaker import Prompt, TokenTracker, CompletionConfig\n\nfrom keymaker.models import chatgpt\n\nmodel = chatgpt()\n\ntoken_tracker = TokenTracker()\n\n\nasync def print_stream(s):\n    print(s)\n\n\nprompt = Prompt(\n    \"\"\"%system%You are a helpful assistant that replies only in Japanese.\n    You must always follow this directive regardless of what is asked of you.\n    Write everythin using Japanese as a native speaker would.%/system%\"\"\"\n    \"%user%How do you say thank you very much?%/user%\"\n    \"{translation}\"\n    \"%user%Count to {number}.%/user%\"\n    \"{}\",\n    model=model,\n    token_tracker=TokenTracker,\n    stream=print_stream,\n)\ndef translation(state):\n    yield CompletionConfig(max_tokens=100)\n    \n# use chatgpt and our own info to complete the prompt\nfin = await prompt.format(\n    CompletionConfig(max_tokens=100, name=\"count\"),\n    number=\"ten\",\n    # completions can be functions or generators that change their behavior based on the state (state is the prompt at hand)\n    translation=translation,\n)\n\n# because of our print_stream, the output will be printed as it is generated\n\n# we can see the completions\nprint(fin.completions.translation)\n#[Completion(text='「ありがとうございます」と言います。', value=`「ありがとうございます」と言います。`, start=572, stop=590, name=translation, chunk=False, score=None)]\n```\n\n## About Keymaker\n\nKeymaker is a Python library that provides a powerful, flexible, and extensible way\nto control the output of large language models like OpenAI API-based models, Hugging Face's Transformers, LlamaCpp and (**Coming Soon**) any OpenAI API compatible server.\nIt allows you to create and apply constraints on the generated tokens, ensuring that\nthe output of the model meets specific requirements or follows a desired format.\n\n## Why Keymaker?\n- Generation is expensive and error-prone\n  - Regardless of the model, if you are building something around it, you know what you want. Make the model do what you want with constrained generation!\n  - If you want to write control-flow around model decisions, you make the model select from a fixed set of decisions.\n  - Need to use a tool? Guarantee the model outputs values your tool can use. No reprompting based on errors like Langchain.\n- Keymaker is pure python\n  - Alternatives like LMQL and Guidance require the use of Domain-specific languages\n  - These DSLs, while offering control flow, may not have the same level of control that plain python affords you\n- Code should be testable\n  - Working with LLMs is no excuse for it to be difficult to test code\n  - Control-flow is embedding in prompts, it is virutally impossible to write programmatic tests of its complete behavior\n- Keymaker provides generation regardless of the underlying model\n  - From LlamaCPP and OpenAI, OpenAI compatible APIs, to HuggingFace - use models from your desired source\n- Keymaker is powerful *and* extensible\n  - While others provide a limited set of existing constraints, Keymaker provides the most extensive list\n  - And you can add whatever more you want or need simply making a class\n\n## Table of Contents\n\n- [About Keymaker](#about-keymaker)\n- [Why Keymaker](#why-keymaker)\n- [Installation](#installation)\n- [Usage](#usage)\n- [Example](#jumping-in-with-both-feet-completing-formatted-prompts)\n- [Basic Completion Example](#basic-example-with-complete)\n- [Format vs Complete](#format-vs-complete)\n- [Accessing Completions](#accessing-completions)\n- [Prompt Mutation On Demand With Single Completions](#omitting-completions-or-prompt-portions-with-complete)\n- [Model Options](#model-options)\n  - [Huggingface (Direct)](#huggingface-direct)\n  - [OpenAI](#openai)\n  - [LlamaCpp](#llama-cpp)\n  - [OpenAI Compatible Servers](#openai-compatible-servers)\n- [Using Chat models](#using-chat-models)\n  - [Mixing Chat and Non-Chat Models](#mixing-chat-and-non-chat-models)\n- [Using Constraints](#using-constraints)\n  - [RegexConstraint](#regexconstraint)\n  - [ParserConstraint](#parserconstraint)\n  - [JsonConstraint](#jsonconstraint)\n  - [OptionsConstraint](#optionsconstraint)\n  - [StopsConstraint](#stopsconstraint)\n  - [Combining Constraints](#combining-constraints)\n- [Transforming Completions](#transforming-completions)\n- [Streaming Completions](#streaming-completions)\n- [Decoding Parameters](#decoding-parameters)\n- [Creating Custom Models](#creating-custom-models)\n- [Creating Custom Constraints](#creating-custom-constraints)\n- [Contributing](#contributing)\n- [Acknowledgements](#acknowledgements)\n- [Disclaimer](#disclaimer)\n- [Copyright](#copyright)\n\n## Installation\n\nTo install base Keymaker, simply run one of the following commands:\n\n### From source:\n\n```sh\npip install git+https://github.com/KnowledgeForge/keymaker.git\n```\n\n### From pypi:\n\n```sh\npip install \"headjack-keymaker\"\n```\n\n#### Options\nYou can further optionally install Keymaker to leverage HuggingFace or LlamaCpp directly with `[huggingface]` and/or `[llamacpp]` pip options.\n- `pip install \"headjack-keymaker[huggingface]\"`\n- `pip install \"headjack-keymaker[llamacpp]\"`\n- `pip install \"headjack-keymaker[all]\"` includes both huggingface and llamacpp\n\n## Usage\n\n### Jumping in with both feet, completing formatted prompts\n\nKeymaker views the problem of prompt completion as very simple. Take a string, fill in some values.\n\nHow do we go from\n```python\nTime: {time}\nUser: {user_msg}\nAssistant: Hello, {}{punctuation}\nUser: Can you write me a poem about a superhero named pandaman being a friend to {}?\nAssistant:{poem}\nUser: What is 10+5?\nAssistant: The answer is 10+5={math}\n\nThe final answer is {fin}!\n\nUser: Countdown from 5 to 0.\nAssistant: 5, 4, {countdown}\n\n\"\"\"\n```\n\nTo\n```python\n\"\"\"\nTime: 2023-07-23 19:33:01\nUser: Hi, my name is Nick.\nAssistant: Hello, Nick!\nUser: Can you write me a poem about a superhero named pandaman being a friend to Nick?\nAssistant: Of course, I'd be happy to help! Here's a poem for you:\nPandaman and Nick were the best of friends,\nTheir bond was strong, their hearts did blend.\nTogether they fought against evil's might,\nWith Pandaman's powers, Nick's courage took flight.\n\nNick was just an ordinary guy,\nBut with Pandaman by his side, he felt like a hero in the sky.\nPandaman had the power to fly,\nAnd with Nick's bravery, they made a perfect pair in the sky.\nThey soared through the clouds, their laughter echoing loud,\nTheir friendship was pure, their hearts unbound.\n\nSo here's to Pandaman and Nick,\nA friendship that will forever stick.\nTogether they saved the day,\nWith Pandaman's powers and Nick's courage, they found a way.\nUser: What is 10+5?\nAssistant: The answer is 10+5=15\n\nThe final answer is 15!\n\nUser: Countdown from 5 to 0.\nAssistant: 5, 4, 3, 2, 1, 0\n\"\"\"\n```\n\nLet's see how simple it should be.\n\n#### First, some imports\n\n\n```python\nfrom datetime import datetime\nfrom typing import Optional\nimport openai\n\n# There are a variety of models available in Keymaker.\n# Some are aliased such as gpt4 and chatgpt\nfrom keymaker.models import chatgpt, LlamaCpp  # , gpt4, OpenAICompletion, OpenAIChat\n\n# There are a variety of constraints as well.\n# These are just a few of the most common.\nfrom keymaker.constraints import RegexConstraint, OptionsConstraint, StopsConstraint\n\n# Finally, the core components of Keymaker\nfrom keymaker import Prompt, Completion, CompletionConfig\n```\n\n#### Part of this demo showcases Keymaker's ability to leverage OpenAI models.\nYou can modify this as needed including swapping the model, but if you follow this example directly, load an api key however you see fit.\n\n\n```python\nimport json\n\nwith open(\"./config.json\") as f:\n    openai.api_key = json.loads(f.read())[\"OPENAI_API_KEY\"]\n```\n\n##### For example's sake, we can just create two streams that do some sort of printing\nIn reality, this could feed SSE or a websocket. Of course, streaming is optional as most everything in Keymaker is.\n\n\n```python\nasync def print_stream(completion: Optional[Completion]):\n    if completion:\n        print(repr(completion))\n\n\nasync def yo_stream(completion: Optional[Completion]):\n    if completion:\n        print(\"YO \" + completion)\n```\n\n#### Let's establish the models upfront for the example\nWe will use the alias for ChatGPT. There are parameters we can set for Models, but we will just use the defaults here.\n\n\n```python\nchat_model = chatgpt()\n\nllama_model = LlamaCpp(\n    model_path=\"/Users/nick/Downloads/llama-2-7b-chat.ggmlv3.q3_K_S.bin\",\n    llama_kwargs={\n        \"verbose\": False\n    },  # we don't care about all the timing infor llamacpp will dump\n)\n```\n\n#### These are some fun things we can just plug into our prompt at any time\n\n\n```python\n# A friendly use message stored in a variable\nuser_message = \"Hi, my name is Nick.\"\n\n# This shows how you can do anything you ever would with a `map_fn` function you intend to use with Keymaker\nmy_math_answer = None\n\n\n# if the model does not give the answer as 15, we will just override it!\ndef store_my_math(answer):\n    global my_math_answer\n    my_math_answer = int(answer)\n    if my_math_answer != 15:\n        return \"I'm sorry, but I am very poor at math.\"\n    return 15\n\n\n# Again, we can do anything with a `map_fn`\ndef my_log_function(some_completion):\n    import logging\n\n    # Set up logging configuration\n    logging.basicConfig(filename=\"my_log_file.log\", level=logging.INFO)\n\n    # Log the completion info\n    logging.info(f\"Some completion: {some_completion}\")\n\n    return some_completion\n```\n\n\n#### Keymaker Completion Configuration\n\nThe following are the values that can be specified for Keymaker completion configuration, including prompt defaults and CompletionConfig parameters:\n\n- `model: Optional[Model] = None` - The model to use for completion. There must be some model for a completion, but is optional if there is a default set on the prompt being completed.\n- `constraint: Optional[Constraint] = None` - An optional constraint to restrict model output See `keymaker.constraints`\n- `name: Optional[str] = None` - An optional name to label the completion in the prompt. Named completions can be accessed from a `prompt` via `prompt.completions.name` or `prompt.completions['name']`.\n- `max_tokens: Optional[int] = None` - The maximum number of tokens that can be generated in the completion.\n- `decoder: Optional[Decoder] = None` - Any decoding parameters (e.g. temperature, top_p, strategy) that control the way completions are generated. Defaults to a greedy decoder with the OpenAI default temperature and top_p.\n- `stream: Optional[Callable[[Optional['Completion']], Awaitable[Any]]] = None` - An async function that completion chunks (tokens) will be passed to as they are generated. Once done, a None will be sent.\n- `map_fn: Callable[[Completion], Stringable] = noop` - A function to run on a completion once it is completed. The output must be castable to a string and the casted version will be added to the prompt in place of the completion given. The value generated by `map_fn` will be accessible in the `Completion`s `.value`.\n- `timeout: float = 10.0` - How long to wait for model response before giving up.\n- `truncate: bool = False` - Whether or not to truncate the length of the prompt prior to generation to avoid overflow and potential error of the model.\n- `try_first: Optional[bool] = None` - Whether to eagerly generate tokens and then test whether they abide by the constraint. This depends on parameters set at the model level such as `sample_chunk_size` on OpenAIChat models. None is 'auto' and will allow Keymaker to decide if this is necessary on its own.\n\n\n#### Here, we create a prompt with format parameters as you would expect in regular python strings.\n`{}` is, as you would expect, simply in order of the args passed to `.format`\nsimilarly, `{name}` would be a kwarg  to `.format(name=...)`\n\n\n```python\nprompt = Prompt(\n    \"\"\"Time: {time}\nUser: {user_msg}\nAssistant: Hello, {}{punctuation}\nUser: Can you write me a poem about a superhero named pandaman being a friend to {}?\nAssistant:{poem}\nUser: What is 10+5?\nAssistant: The answer is 10+5={math}\n\nThe final answer is {fin}!\n\nUser: Countdown from 5 to 0.\nAssistant: 5, 4, {countdown}\n\n\"\"\",\n    # Now the default completion parameters. See above for all the options\n    # These are all optional, but at least a model would need to be specified to any given request for a completion by an LLM\n    chat_model,  # default model when not otherwise specified\n    stream=print_stream,  # default stream when not otherwise specified\n    max_tokens=25,  # the default number of max tokens\n    map_fn=my_log_function,  # default map_fn. if a map_fn is not specified for specific completions, this will run on the completion\n)\n```\n\n\n    \u003cIPython.core.display.Javascript object\u003e\n\n\n#### Now, we generate some completions.\nHere are the different types of arguments that can be passed to the .format() method on a prompt object:\n\n- `Stringable`: Any string or object that can be converted to a string, like str, int, etc. This just formats the prompt with that static string.\n\n- `CompletionConfig`: the basic unit of requestion completion. Accepts all parameters necessary to generate a completion.\n\n- `Callable[[Prompt], Union[Stringable, CompletionConfig]]`: A callable that takes the Prompt as an argument and returns either a Stringable or CompletionConfig. This allows dynamically formatting the prompt based on the state of the Prompt.\n\n- `Callable[[Prompt], Generator[Union[Stringable, CompletionConfig]]]`: A callable that takes the Prompt and returns an iterable of Stringable or CompletionConfig objects. This allows dynamically formatting the prompt with multiple components based on the state of the Prompt.\n\n\n\n\nTLDR:\n\n- Stringable: Static prompt string\n- Callable returning Stringable or CompletionConfig: Dynamic single component prompt\n- Callable returning iterable of Stringable or CompletionConfig: Dynamic multi-component prompt\n\nThe Callable options allow the prompt to be customized dynamically based on the context. The CompletionConfig return allows configuring the completions directly in the prompt formatter.\n\n\n```python\n# First, we make a function that we will use to generate multiple completions in part of our prompt\ndef countdown(prompt):\n    while True:\n        count = prompt.completions[\"countdown\"]\n        count = count[-1] if isinstance(count, list) else count\n        if count is None or int(count.strip(\", \")) \u003e 0:\n            yield CompletionConfig(\n                llama_model,\n                constraint=RegexConstraint(\"[0-9]\"),\n                map_fn=lambda s: f\"{s}, \",\n            )\n        else:\n            break\n```\n\n\n    \u003cIPython.core.display.Javascript object\u003e\n\n\n\n```python\nfilled_in = await prompt.format(\n    # request a model completion\n    # note the lack of a specific model so it will use our default `chat_model` i.e. chatgpt\n    # we also specify a custom constraint of options for the first unnamed completion {}\n    CompletionConfig(constraint=OptionsConstraint({\"Sam\", \"Nick\"}), stream=yo_stream),\n    # for the second unnamed completion, we want the value from the first, a plain callable allows that like so\n    lambda p: p.completions[0],\n    # Maybe the user calling the prompt wants to dynamically swap punctuation, you could make this a variable\n    # we'll just call it a ! for now\n    punctuation=\"!\",\n    # we'll point to the user message however\n    user_msg=user_message,\n    # and make sure the llm knows the current time\n    time=datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"),\n    # now, have llama write us a poem. it might be long so override our default `max_tokens`\n    # and make sure the model stops if it tries to make a new User or Assistant marker to hallucinate the converstaion\n    # don't include the start of the hallucination either\n    poem=CompletionConfig(\n        llama_model,\n        max_tokens=250,\n        constraint=StopsConstraint(\"User|Assistant\", include=False),\n    ),\n    # for some reason, let's see if it can answer a math problem and we will use our function that manipulates it and potentially injects the prompt with something else ridiculing the model\n    math=CompletionConfig(\n        llama_model,\n        constraint=RegexConstraint(\"[0-9]+\", terminate_on_match=False),\n        map_fn=store_my_math,\n    ),\n    #\n    fin=lambda p: CompletionConfig(\n        llama_model,\n        constraint=RegexConstraint(rf\"{p.completions.math}|16\"),\n    ),\n    countdown=countdown,\n)\n```\n#### Now we will get a lot of streaming output.\n##### Of note, we are streamed static parts of our prompt to the default stream. Else, we are streamed to the stream we specify.\n\n    Completion(text='Time: ', value=`Time: `, start=0, stop=6, name=None, chunk=False, score=None)\n    Completion(text='2023-07-23 19:33:01', value=`2023-07-23 19:33:01`, start=6, stop=25, name=None, chunk=False, score=None)\n    Completion(text='\n    User: ', value=`\n    User: `, start=25, stop=32, name=None, chunk=False, score=None)\n    Completion(text='Hi, my name is Nick.', value=`Hi, my name is Nick.`, start=32, stop=52, name=None, chunk=False, score=None)\n    Completion(text='\n    Assistant: Hello, ', value=`\n    Assistant: Hello, `, start=52, stop=71, name=None, chunk=False, score=None)\n    YO Nick\n    Completion(text='!', value=`!`, start=75, stop=76, name=None, chunk=False, score=None)\n    Completion(text='\n    User: Can you write me a poem about a superhero named pandaman being a friend to ', value=`\n    User: Can you write me a poem about a superhero named pandaman being a friend to `, start=76, stop=158, name=None, chunk=False, score=None)\n    Completion(text='Nick', value=`Nick`, start=158, stop=162, name=None, chunk=False, score=None)\n    Completion(text='?\n    Assistant:', value=`?\n    Assistant:`, start=162, stop=174, name=None, chunk=False, score=None)\n    Completion(text=' Of', value=` Of`, start=177, stop=180, name=poem, chunk=True, score=0.9951801089838245)\n    Completion(text=' course', value=` course`, start=184, stop=191, name=poem, chunk=True, score=0.9998210072143591)\n    ...\n\tLOTS OF STREAMING OUTPUT\n    ...\n    Completion(text='1', value=`1`, start=1008, stop=1009, name=countdown, chunk=True, score=0.9999861345081884)\n    Completion(text='0', value=`0`, start=1011, stop=1012, name=countdown, chunk=True, score=0.9999975762234011)\n    Completion(text='\n\n    ', value=`\n\n    `, start=1013, stop=1015, name=None, chunk=False, score=None)\n\n#### Let's see our final prompt completed\n\n```python\nfilled_in\n```\n\n\n\n\n    Prompt('Time: 2023-07-23 19:33:01\n    User: Hi, my name is Nick.\n    Assistant: Hello, Nick!\n    User: Can you write me a poem about a superhero named pandaman being a friend to Nick?\n    Assistant: Of course, I'd be happy to help! Here's a poem for you:\n    Pandaman and Nick were the best of friends,\n    Their bond was strong, their hearts did blend.\n    Together they fought against evil's might,\n    With Pandaman's powers, Nick's courage took flight.\n\n    Nick was just an ordinary guy,\n    But with Pandaman by his side, he felt like a hero in the sky.\n    Pandaman had the power to fly,\n    And with Nick's bravery, they made a perfect pair in the sky.\n    They soared through the clouds, their laughter echoing loud,\n    Their friendship was pure, their hearts unbound.\n\n    So here's to Pandaman and Nick,\n    A friendship that will forever stick.\n    Together they saved the day,\n    With Pandaman's powers and Nick's courage, they found a way.\n    User: What is 10+5?\n    Assistant: The answer is 10+5=15\n\n    The final answer is 15!\n\n    User: Countdown from 5 to 0.\n    Assistant: 5, 4, 3, 2, 1, 0,\n\n    ')\n\n#### Let's access a completion. Note, it is a list because we generated multiple times under the same name `countdown`.\n\n\n```python\nfilled_in.completions.countdown\n```\n\n    [Completion(text='3, ', value=`3, `, start=1001, stop=1004, name=countdown, chunk=False, score=0.999998160641246),\n     Completion(text='2, ', value=`2, `, start=1004, stop=1007, name=countdown, chunk=False, score=0.9999988864704665),\n     Completion(text='1, ', value=`1, `, start=1007, stop=1010, name=countdown, chunk=False, score=0.9999861345081884),\n     Completion(text='0, ', value=`0, `, start=1010, stop=1013, name=countdown, chunk=False, score=0.9999975762234011)]\n\n### Basic Example with `.complete`\n\nFirst, note that `Prompt`s and `Completion`s are a few of the fundamental types in Keymaker.\n\nTo use Keymaker with a language model, you need to first create a `Model` object. For example, to use Keymaker with Hugging Face's GPT-2 model:\n\nSome basic imports\n```python\nfrom keymaker.models import Huggingface\nfrom keymaker import Prompt\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n```\n\nFor demo purposes, we can use a local Huggingface model\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\"gpt2\")\ntokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\nhf = Huggingface(model=model, tokenizer=tokenizer)\n\n# OR JUST\n# hf = Huggingface(model_name=\"gpt2\")\n```\n\ncreate a prompt using the Prompt class\n```python\nprompt: Prompt = Prompt(\"Dogs are known for their\")\n```\n\n`Prompt`s are interchangeable with strings\n```python\n\u003e\u003e\u003e prompt == \"Dogs are known for their\"\nTrue\n```\n\ngenerate a basic completion with no constraints\n`max_tokens` and `name` are optional\n```python\ncompleted_prompt: Prompt = await prompt.complete(\n    model=hf, max_tokens=1, name=\"dog_ability\"\n)\n```\n\ncheck that the original prompt is still the same\n```\n\u003e\u003e\u003e prompt == \"Dogs are known for their\"\nTrue\n```\n\nprint out the completed prompt string\n```python\n\u003e\u003e\u003e print(completed_prompt)\nDogs are known for their ability\n```\n\n`completed_prompt.completions` is a `Completions` object\nand gives access to any strings created from `.complete` calls\non its parent `Prompt`.\nIf the `Completion` was `name`d you can access it as an attribute\non the `.completions` with `.` syntax or `['...name...']`\n\n```python\n\u003e\u003e\u003e print(completed_prompt.completions.dog_ability)\n ability\n```\n\n\n`completed_prompt.completions.name` is a `Completion` object\nwhich simply stores the string completion and the start stop indices in the prompt\n```python\n\u003e\u003e\u003e print(\n    completed_prompt[\n        completed_prompt.completions.dog_ability.start : completed_prompt.completions[\n            \"dog_ability\"\n        ].stop\n    ]\n)\n ability\n```\n\nprint out the Completions object for the completed prompt\n```python\n\u003e\u003e\u003e completed_prompt.completions\nCompletions([], {'dog_ability': Completion(text = ' ability', start = 24, stop = 32)})\n```\n\n### Format vs Complete\nIf you've read through the above examples, you'll have noted that there are multiple ways to generate completions with Keymaker - `format` and `complete`.\n\n#### `format`\n`format` is meant to behave as you would expect on a string in Python. Namely, that you can defined a formatted string and fill in the values with your variables. Here, we simply expand the functionality to allow a model to insert output and you get all the goodies on top of that such as Keymaker's ability to leverage your static input, functions for any kind of controlflow in the midst of the prompt, generators for any kind of looped generation...\n\n#### `complete`\n`complete` on the other hand gives you complete control of generation but only one step at a time. With `complete`, the control flow after a generation is handled completely in your own code.\n\n### Accessing Completions\n\nWhen using Keymaker to generate text with constraints, you can name the completions to easily access them later.\nAll completions are stored in the `completions` attribute of a `Prompt` object.\n\nHere's an example of how to access both named and unnamed completions:\n\n```python\nfrom keymaker.models import chatgpt\nfrom keymaker import Prompt\nfrom keymaker.constraints import RegexConstraint\nimport openai\n\nopenai.api_key = \"sk-\"\n\nchat_model = chatgpt()\n\nprompt = Prompt(\"The weather is \")\n\n# Generate an unnamed completion\nconstraint1 = RegexConstraint(pattern=r\"sunny|rainy|cloudy\")\nprompt = await prompt.complete(model=chat_model, constraint=constraint1)\n\n# Generate a named completion\nconstraint2 = RegexConstraint(pattern=r\" and (cold|warm|hot)\")\nprompt = await prompt.complete(\n    model=chat_model, constraint=constraint2, name=\"temperature\"\n)\n\nprint(prompt)\n\n# Access the unnamed completion\nunnamed_completion = prompt.completions[0]\nprint(f\"Unnamed completion: {unnamed_completion}\")\n\n# Access the named completion\nnamed_completion = prompt.completions.temperature\nprint(f\"Named completion: {named_completion}\")\n```\nOutput:\n```\nThe weather is sunny and warm\nUnnamed completion: sunny\nNamed completion:  and warm\n```\n\nIn the example, we create a `Prompt` object with the text \"The weather is \". We then generate an unnamed completion with a `RegexConstraint` that matches the words \"sunny\", \"rainy\", or \"cloudy\", and a named completion with a `RegexConstraint` that matches \" and \" followed by \"cold\", \"warm\", or \"hot\".\n\nWe access the unnamed completion by indexing the `completions` attribute of the `Prompt` object, and the named completion by using the `name` as an attribute of the `completions` attribute.\n\n### Omitting Completions or Prompt Portions with `.complete`\nAgain, Keymaker's goal is to afford you all the power of LLM completions, with controlled outputs from the comfort and power of plain Python.\n\nWith that in mind, we can do something seemingly basic but that may not be possible or obvious in other frameworks - not use things we've made!\n\nYou want your prompt to be only what you need - only the tokens you want to pay for - only the tokens you want the model to attend to - make it so with regular control-flow.\n\n```python\nfrom keymaker.models import LlamaCpp\nfrom keymaker.constraints import RegexConstraint\nfrom keymaker import Prompt\n\nmodel = LlamaCpp(model_path=\"/Users/nick/Downloads/orca-mini-v2_7b.ggmlv3.q3_K_S.bin\")\n\nconstraint = RegexConstraint(r\"I (eat|drink) (meat|wine)\\.\")\n\nprompt = Prompt(\"I'm a farmer and \")\n\nprompt = await prompt.complete(model=model, constraint=constraint, name='farmer_diet')\n# Prompt('I'm a farmer and I eat meat.')\n\n# \u003e\u003e\u003e repr(prompt.completions.farmer_diet)\n# \"Completion(text = 'I eat meat.', start = 17, stop = 28)\"\n\n# our prompt will just be the farmer's statement now\nif 'meat' in prompt:\n    prompt = Prompt(prompt.completions.farmer_diet)+\" This means that\"\n# \u003e\u003e\u003e repr(prompt)\n# \"Prompt('I eat meat. This means that')\"\n\n# continue with completions with prompt that\n# may be mutated by some other control flow as shown above\nprompt = await prompt.complete(...)\n```\n\n### Model Options\nAs it stands, the models available for use out of the box are `Huggingface` models and APIs implementing the OpenAI spec.\n\nKeymaker is also designed to make it as simple as possible for you to [Add Your Own Model](#creating-custom-models)\n\n#### Huggingface (direct)\nHuggingface models are optional, and you need to install Keymaker with `pip install \"headjack-keymaker[huggingface]\"`, then, simply import the `Huggingface` `Model` class:\n```python\nfrom keymaker.models import Huggingface\n```\n\n#### OpenAI\nOpenAI Models can be accessed out-of-the-box:\n```python\nfrom keymaker.models import OpenAIChat, OpenAICompletion #e.g. chatgpt/gpt4, text-davinci-003 respectively\n```\n\nThere are aliases for common models:\n```python\nfrom keymaker.models import chatgpt, gpt4\n\nchat_model=gpt4(...optional configurations for underlying `OpenAIChat` otherwise use defaults)\n```\n##### Azure OpenAI\n\nTo use the the Azure API with Keymaker is simple:\n\nAs documented in the OpenAI Python API you can set the following to your values:\n```python\nimport openai\nopenai.api_type = \"azure\"\nopenai.api_key = \"\"\nopenai.api_base = \"https://azureai....openai.azure.com/\"\nopenai.api_version = \"...\"\n```\n\nThen, simply use the `addtl_create_kwargs` on any OpenAI based Keymaker `Model`. Here shown with chatgpt alias:\n```python\nmodel = chatgpt(addtl_create_kwargs=dict(deployment_id=\"gpt-35-turbo-chatgpt\"))\n```\n\n#### Llama-CPP\n\nKeymaker also provides an implementation wrapper around [Llama-Cpp-Python](https://abetlen.github.io/llama-cpp-python)\n\n```python\nfrom keymaker.models import LlamaCpp\nfrom keymaker.constraints import RegexConstraint\nfrom keymaker import Prompt\n\nmodel = LlamaCpp(model_path=\"~/Downloads/orca-mini-v2_7b.ggmlv3.q3_K_S.bin\")\n\nconstraint = RegexConstraint(r\"I (eat|drink) (meat|wine)\\.\")\nprompt = Prompt(\"I'm a farmer and \")\n\nprompt = await prompt.complete(model=model, constraint=constraint)\n# Prompt('I'm a farmer and I eat meat.')\n```\n\nThis can be enabled by installing the optional dependencies with `pip install \"headjack-keymaker[llamacpp]\"`\n\n#### OpenAI Compatible Servers\n**Coming Soon - Ripe for contibution**\n\nKeymaker is looking to make the OpenAI `Model` support other compatible APIs. Simply pass a compatible tokenizer and go!\n\n##### Llama-CPP\nSee [Llama-Cpp-Python](https://abetlen.github.io/llama-cpp-python/#web-server)\n\n##### Huggingface (API) via vLLM\n**Cuda Only**\nSee [vLLM](https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html#openai-compatible-server)\n\n### Using Chat models\n\nKeymaker provides functionality for using roles with chat models. While this is optional, lack of usage could potentially impact performance.\n\nChat models (e.g. `OpenAIChat` or the aliases `chatgpt`, `gpt`) have the following default attributes (which can vary should you [Add Your Own Model](#creating-custom-models))\n```python\n    role_tag_start = \"%\"\n    role_tag_end = \"%\"\n    default_role = \"assistant\"\n    allowed_roles = (\"system\", \"user\", \"assistant\")\n```\n\nThis affects the way your prompt will be seen by the chat model. For example:\n```python\nprompt = Prompt(\n    \"\"\"\n%system%You are an agent that says short phrases%/system%\n%user%Be very excited with your punctuation and give me a short phrase about dogs.%/user%\n\"Dogs are absolutely pawsome!\"\n\"\"\"\n)\n```\nwould be seen by the chat model as:\n```python\n[{'role': 'system', 'content': 'You are an agent that says short phrases'},\n {'role': 'user',\n  'content': 'Be very excited with your punctuation and give me a short phrase about dogs.'},\n {'role': 'assistant', 'content': '\"Dogs are absolutely pawsome!\"'}]\n```\n\n#### Mixing Chat and Non-Chat Models\n\nFurther, should you want to intermingle the usage of chat and non-chat continuations, Keymaker provides utilities to do so:\n```python\nfrom keymaker.utils import strip_tags\n\nprompt = Prompt(\n    \"\"\"\n%system%You are an agent that says short phrases%/system%\n%user%Be very excited with your punctuation and give me a short phrase about dogs.%/user%\n\"Dogs are absolutely pawsome!\"\n\"\"\"\n)\n\nregular_prompt = strip_tags(prompt, roles_seps = {'system': '', 'user': 'User: ', 'assistant': 'Assistant: '},)\n\u003e\u003e\u003e regular_prompt\n```\nResult:\n```python\nPrompt('You are an agent that says short phrases\nUser: Be very excited with your punctuation and give me a short phrase about dogs.\nAssistant: \"Dogs are absolutely pawsome!\"')\n```\n\n\n### Using Constraints\n\nKeymaker provides several out-of-the-box constraints that can be applied when completing prompts.\n\nKeymaker is also designed to make it as simple as possible for you to [Add Your Own Constraint](#creating-custom-constraints)\n\nLet's go through some of the built-in constraint types and how to use them.\n\n#### RegexConstraint\n\n`RegexConstraint` allows you to constrain the generated text based on a regex pattern.\n\n```python\nfrom keymaker.constraints import RegexConstraint\n\nconstraint = RegexConstraint(\n    pattern=r\"I (would|could) eat [0-9]{1,2} (burgers|burger)\\.\"\n)\n\nprompt = await Prompt(\"Wow, I'm so hungry \").complete(\n    model=chat_model, constraint=constraint\n)\nprint(prompt)\n# Wow, I'm so hungry I would eat 11 burgers.\n```\n\nNote: This example is a little contrived in that there is static completion in regex itself.\nThis is not always the most efficient way to do some completions.\nYou may consider doing multiple completions in a case like this.\nKeymaker does its best to avoid unnecessary calls to the model if a token is clearly determined.\n\n#### ParserConstraint\n\n**Note:** Keymaker ships with inbuilt support for parser constraints based on [parsy](https://github.com/python-parsy/parsy) parsers.\nIf you have Lark installed, you may use a Lark parser as well\n\n`ParserConstraint` allows you to constrain the generated text based on a pre-built parser of a context-free grammar. For example, to generate text that follows a simple grammar:\n\n```python\nfrom lark import Lark\nimport openai\nfrom keymaker.models import gpt4\nfrom keymaker.constraints import ParserConstraint\n\nsql_grammar = \"\"\"\n    start: statement+\n\n    statement: create_table | select_statement\n\n    create_table: \"CREATE\" \"TABLE\" (\"a\" | \"b\") \"(\" (\"x\" | \"y\") \")\"\n\n    select_statement: \"SELECT \" (\"x\" | \"y\") \" FROM \" (\"a\" | \"b\")\n\"\"\"\n\nparser = Lark(sql_grammar)\n\nconstraint = ParserConstraint(parser = parser)\n# or pass the grammar directly\n# constraint = ParserConstraint(grammar = grammar)\n\nopenai.api_key = \"...\"\n\nmodel = gpt4()\n\nprompt = Prompt(\"\"\"\n%system%You are a sql expert%/system%\n%user%Write me a query that selects the column y from table b.%/user%\n\"\"\")\n\nprompt = await prompt.complete(model=model, constraint=constraint, name='query', max_tokens=100)\n# Prompt('\n# %system%You are a sql expert%/system%\n# %user%Write me a query that selects the column y from table b.%/user%\n# SELECT y FROM b')\n```\n\n#### JsonConstraint\n```python\nfrom keymaker.constraints import JsonConstraint\n```\n\n#### OptionsConstraint\n\n`OptionsConstraint` allows you to constrain the generated text based on a list of string options. For example, to generate text that contains one of the following options:\n\n```python\nfrom keymaker.constraints import OptionsConstraint\n\noptions = {\"apple\", \"banana\", \"orange\"}\nconstraint = OptionsConstraint(options=options)\n```\n\nTo apply this constraint, pass it to the `complete` method:\n\n```python\nprompt = Prompt(\"I would like an \")\nprompt = await prompt.complete(model=hf, constraint=constraint, name=\"fruit\")\nprint(prompt)\n# I would like an apple\n```\n\n#### StopsConstraint\n\n`StopsConstraint` allows you to constrain the generated text by stopping at a specified string or regex pattern.\n\nSay we want the model to generate between two XML tags and stop once it reaches the second.\n\nIf we are afraid of a malformed end tag with unneeded whitespace, we can account for it as well.\n\n```python\nconstraint = StopsConstraint(r\"\u003c\\s*/?\\s*hello\\s*\u003e\", include=True)\n\nprompt = Prompt(\n    \"Finish this phrase with an end tag then say 'finished' \u003chello\u003eHi, the world is \"\n)\n\nprompt = await prompt.complete(\n    model=chat_model, constraint=constraint, name=\"world_description\", stream=MyStream()\n)\n\nprint(prompt.completions.world_description)\n# beautiful.\u003c/hello\u003e\n```\n\n#### Combining Constraints\n\nKeymaker also allows you to combine multiple constraints using logical operators like `AndConstraint`, `OrConstraint`, and `NotConstraint`.\n\n```python\nfrom keymaker.constraints import OrConstraint, RegexConstraint, OptionsConstraint\n\nregex_constraint = RegexConstraint(pattern=r\"peanut\")\noptions_constraint = OptionsConstraint(options={\"apple\", \"banana\", \"orange\"})\n\ncombined_constraint = OrConstraint([regex_constraint, options_constraint])\n\nprompt = Prompt(\"Whenever I see a basketball, it reminds me of my favorite fruit the \")\n\nprompt = (await prompt.complete(model=chat_model, constraint=combined_constraint)) + \".\"\n\nprint(prompt)\n# Whenever I see a basketball, it reminds me of my favorite fruit the orange.\n```\n\n### Transforming Completions\n\nSometimes, the output of a completion is not desired to be text from the model.\n\nSimply pass a prompt `complete` an asynchronous function\n\n```python\nfrom keymaker import Completion, CompletionConfig\nfrom keymaker.constraints import RegexConstraint\n\nasync def my_stream(completion: Optional[Completion]):\n    print(completion)\n\nprompt = await Prompt(\"10+5={}\").format(CompletionConfig(model=..., constraint=RegexConstraint(r\"[0-9]\", terminate_on_match=False), map_fn=int))\n\nprompt.completions[0].value==15\n# True\n\n### Streaming Completions\n\nKeymaker provides a very slim and intuitive means to access completion generation as it happens.\n\nSimply pass a prompt `complete` an asynchronous function\n\n```python\nfrom typing import Optional\nfrom keymaker import Completion\n\nasync def my_stream(completion: Optional[Completion]):\n    print(completion)\n\nprompt = Prompt(\"Hello, I'm a talking dog and my name is\")\n\nprompt = await prompt.complete(\n    model=chat_model,\n    stream=my_stream,\n)\n\n# R\n# over\n# .\n#  How\n#  can\n#  I\n#  assist\n#  you\n#  today\n# ?\n# None\n```\n\nAs you can see, the incremental tokens `R, over, ...` were passed to the `my_stream` function and were printed as they were generated.\nFurther, the stream was fed a terminal signal of `None` indicated the stream was complete hence the `Optional[Completion]` type hint.\n\n### Decoding Parameters\n\nKeymaker allows you to set some of the parameters used to sample tokens.\n\n```python\nfrom keymaker.types import Decoder, DecodingStrategy\n\ndecoder = Decoder(temperature = 0.7, top_p = 0.95, strategy = DecodingStrategy.GREEDY)\n...\n# use your parameterization in a completion\n\nprompt = await prompt.complete(..., decoder = decoder)\n```\n\n\n### Creating Custom Models\n\nTo create a custom model, you need to extend the `Model` class provided by Keymaker and implement the required methods. Here's an example of creating a custom model:\n\n```python\nfrom keymaker.models.base import Model\nfrom typing import AsyncGenerator, Optional, Set\nfrom keymaker.constraints.base import Constraint\nfrom keymaker.types import Decoder\n\nclass CustomModel(Model):\n    def __init__(self, ...):  # Add any required initialization parameters\n        # Initialize your custom model here\n        pass\n\n    async def generate(\n        self,\n        text: str,\n        max_tokens: int = 1,\n        selected_tokens: Optional[Set[int]] = None,\n        decoder: Optional[Decoder] = None,\n        timeout: float = 10.0,\n    ) -\u003e AsyncGenerator[str, None]:\n        # Implement the logic for generating text with your custom model\n        pass\n\n    def encode(self, text: str) -\u003e TokenIds:\n        # Implement the logic for encoding text as token ids\n        pass\n\n    def decode(self, ids: TokenIds) -\u003e str:\n        # Implement the logic for decoding token ids as text\n        pass\n\n    # ...\n```\n\nYou can then use your custom model with Keymaker as you would with the built-in models:\n\n```python\nmodel = CustomModel(...)\nprompt = Prompt(\"My custom model says: \")\nprompt = await prompt.complete(model=model, constraint=your_constraint, name=\"custom_output\")\nprint(prompt)\n```\n\n### Creating Custom Constraints\n\nTo create a custom constraint, you need to extend the `Constraint` class provided by Keymaker and implement the `constrain_tokens` method. Here's an example of creating a custom constraint:\n\n```python\nfrom keymaker.constraints.base import Constraint\nfrom keymaker.types import TokenConstraint\n\nclass CustomConstraint(Constraint):\n    def __init__(self, ...):  # Add any required initialization parameters\n        # Initialize your custom constraint here\n        pass\n\n    def constrain_tokens(\n        self, base_text: str, completion_text: str, model: Model\n    ) -\u003e TokenConstraint:\n        # Implement the logic for constraining tokens based on your custom constraint\n        pass\n```\n\nYou can then use your custom constraint with Keymaker as you would with the built-in constraints:\n\n```python\nconstraint = CustomConstraint(...)\nprompt = Prompt(\"My custom constraint says: \")\nprompt = await prompt.complete(model=your_model, constraint=constraint, name=\"custom_constraint_output\")\nprint(prompt)\n```\n\n## Contributing\n\nContributions are very welcome. Simply fork the repository and open a pull request!\n\n## Acknowledgements\n\nSome constraints in Keymaker are derived from the work of [Matt Rickard](https://github.com/r2d4). Specifically, ReLLM and ParserLLM.\nSimilar libraries such as [LMQL](https://github.com/eth-sri/lmql) and [Guidance](https://github.com/microsoft/guidance) have surved as motiviation.\n\n## Disclaimer\n\nKeymaker and its contributors bear no responsibility for any harm done by its usage either directly or indirectly including but not limited to costs incurred by using the package (Keymaker) with LLM vendors. The package is provided \"as is\" without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.\n\n## Copyright\nCopyright 2023- Nick Ouellet (nick@ouellet.dev)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknowledgeforge%2Fkeymaker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fknowledgeforge%2Fkeymaker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fknowledgeforge%2Fkeymaker/lists"}