{"id":13457988,"url":"https://github.com/guidance-ai/guidance","last_synced_at":"2025-05-12T20:43:29.474Z","repository":{"id":152151133,"uuid":"564443255","full_name":"guidance-ai/guidance","owner":"guidance-ai","description":"A guidance language for controlling large language models.","archived":false,"fork":false,"pushed_at":"2025-05-04T09:45:45.000Z","size":12928,"stargazers_count":20123,"open_issues_count":295,"forks_count":1098,"subscribers_count":124,"default_branch":"main","last_synced_at":"2025-05-05T16:05:31.946Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guidance-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":"GOVERNANCE.md","roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-11-10T18:21:45.000Z","updated_at":"2025-05-05T03:18:41.000Z","dependencies_parsed_at":"2023-08-12T01:59:33.589Z","dependency_job_id":"ad03e68b-7af8-4dae-8837-9ef3f5862835","html_url":"https://github.com/guidance-ai/guidance","commit_stats":null,"previous_names":["slundberg/guidance","guidance-ai/guidance","microsoft/guidance"],"tags_count":41,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fguidance","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fguidance/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fguidance/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guidance-ai%2Fguidance/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guidance-ai","download_url":"https://codeload.github.com/guidance-ai/guidance/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252601722,"owners_count":21774663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T09:00:41.572Z","updated_at":"2025-05-12T20:43:29.430Z","avatar_url":"https://github.com/guidance-ai.png","language":"Jupyter Notebook","funding_links":[],"categories":["Uncategorized","Jupyter Notebook","LLM Ethics and Governance","Python Libraries","Tools and Frameworks","Structured Generation","structured generation","Context Management","Learning","Autonomous Research \u0026 Content Generation","Repos","\u003ca id=\"tools\"\u003e\u003c/a\u003e🛠️ Tools","安全护栏","Frameworks","Tools \u0026 Frameworks","Schemas \u0026 Patterns","Open Source Projects","Libraries","🛠 Development Frameworks","2. Libraries \u0026 Frameworks","Structured output","🛠️ Developer Infrastructure","Agent SDK / framework","Real-World Applications","Open Source Tools","4. Agentic AI \u0026 Multi-Agent Systems","📊 Structured Output \u0026 Data Extraction","LLM Serving / Inference","Agent SDKs \u0026 Frameworks","LLM Tools"],"sub_categories":["Uncategorized","Reflection and Meta-Cognition","Repositories","Reasoning Frameworks","Bleeding Edge ⚗️","Prompt Programming","Development Tools","AI Products with UI Generation","Frameworks \u0026 Libraries","Low-Level Libraries","Python","LLM Frameworks \u0026 Orchestration","Structured Output \u0026 Multi-Provider"],"readme":"\u003cdiv align=\"right\"\u003e\u003ca href=\"https://guidance.readthedocs.org\"\u003e\u003cimg src=\"https://readthedocs.org/projects/guidance/badge/?version=latest\u0026style=flat\" /\u003e\u003c/a\u003e\u003c/div\u003e\n\u003cdiv align=\"center\"\u003e\u003cpicture\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"docs/figures/guidance_logo_blue_dark.svg\"\u003e\n  \u003cimg alt=\"guidance\" src=\"docs/figures/guidance_logo_blue.svg\" width=300\"\u003e\n\u003c/picture\u003e\u003c/div\u003e\n\u003cbr/\u003e\n\n**Guidance is an efficient programming paradigm for steering language models.** With Guidance, you can control how output is structured and get high-quality output for your use case—*while reducing latency and cost vs. conventional prompting or fine-tuning.* It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditionals, loops, tool use) and generation seamlessly.\n\n   * [Install](#install)\n   * [Features](#features)\n   * [Example notebooks](#example-notebooks)\n   * [Basic generation](#basic-generation)\n   * [Constrained generation](#constrained-generation)\n   * [Stateful control + generation](#stateful-control--generation)\n\n\n## Install\nGuidance is available through PyPI and supports a variety of backends (Transformers, llama.cpp, OpenAI, etc.). To use a specific model see [loading models](#loading-models).\n```bash\npip install guidance\n```\n\n\u003c!-- For a detailed walkthrough of using Guidance on hosted Phi models, check the [Azure AI specific loading instructions.](#azure-ai) and the [Phi-3 + Guidance cookbook](https://github.com/microsoft/Phi-3CookBook/blob/main/code/01.Introduce/guidance.ipynb). --\u003e\n\n\u003c!-- \u003ca href=\"https://www.youtube.com/watch?v=9oXjP5IIMzQ\"  aria-label=\"Watch demo\"\u003e\u003cimg alt=\"Watch demo\" src=\"docs/figures/watch_demo_button.png\" width=\"120\"\u003e\u003c/a\u003e \u003ca href=\"#get-started\" aria-label=\"Get started\"\u003e\u003cimg alt=\"Watch demo\" src=\"docs/figures/get_started_button.png\" width=\"120\"\u003e\u003c/a\u003e --\u003e\n\n## Features\n### Write [pure Python](#basic-generation), with additional LM functionality.\n```python\nfrom guidance import models, gen\n\n# load a model (could be Transformers, LlamaCpp, VertexAI, OpenAI...)\nllama2 = models.LlamaCpp(path) \n\n# append text or generations to the model\nllama2 + f'Do you want a joke or a poem? ' + gen(stop='.')\n```\n\u003cimg alt=\"Do you want a joke or a poem? I'll give you a poem\" src=\"docs/figures/simple_gen_llama2_7b.png\" width=\"354\"\u003e\n\n### [Constrain generation](#constrained-generation) with [selects](#select-basic) (i.e., sets of options), [regular expressions](#regular-expressions), and [context-free grammars](#context-free-grammars), as well as with pre-built components (e.g., substring, json).\n\n```python\nfrom guidance import select\n\n# a simple select between two options\nllama2 + f'Do you want a joke or a poem? A ' + select(['joke', 'poem'])\n```\n\u003cimg alt=\"Do you want a joke or a poem? A poem\" src=\"docs/figures/simple_select_llama2_7b.png\" width=\"277\"\u003e\n\n### Call and deploy tools easily with automatic interleaving of control and generation.\n[Easy tool use](#automatic-interleaving-of-control-and-generation-tool-use), where the model stops generation when a tool is called, calls the tool, then resumes generation. For example, here is a simple version of a calculator, via four separate 'tools':\n```python\n@guidance\ndef add(lm, input1, input2):\n    lm += f' = {int(input1) + int(input2)}'\n    return lm\n@guidance\ndef subtract(lm, input1, input2):\n    lm += f' = {int(input1) - int(input2)}'\n    return lm\n@guidance\ndef multiply(lm, input1, input2):\n    lm += f' = {float(input1) * float(input2)}'\n    return lm\n@guidance\ndef divide(lm, input1, input2):\n    lm += f' = {float(input1) / float(input2)}'\n    return lm\n```\nNow we call `gen` with these tools as options. Notice how generation is stopped and restarted automatically:\n```python\nlm = llama2 + '''\\\n1 + 1 = add(1, 1) = 2\n2 - 3 = subtract(2, 3) = -1\n'''\nlm + gen(max_tokens=15, tools=[add, subtract, multiply, divide])\n```\n\u003cimg width=\"201\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/646e1a7d-0206-419b-8206-1d835c3a0e0a\"\u003e\u003cbr\u003e\n\n### Get high compatibility—execute a single Guidance program on many backends \nWorks with Transformers, llama.cpp, AzureAI, VertexAI, OpenAI and others. Users can write one guidance program and execute it on many backends. (note that the most powerful control features require endpoint integration, and for now work best with Transformers and llama.cpp).\n```python\ngpt = models.OpenAI(\"gpt-3.5-turbo\")\n\nwith user():\n    lm = gpt + \"What is the capital of France?\"\n\nwith assistant():\n    lm += gen(\"capital\")\n\nwith user():\n    lm += \"What is one short surprising fact about it?\"\n\nwith assistant():\n    lm += gen(\"fact\")\n```\n\u003cimg width=\"645\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/f31ed7b8-1868-44d2-b14c-4842b0a40e5c\"\u003e\u003cbr\u003e\n\n### Gain speed with [stateful control + generation functions](#stateful-control--generation)—no need for intermediate parsers. \nIn contrast to chaining, Guidance programs are the equivalent of a single LLM call. More so, whatever non-generated text that gets appended is batched, so that Guidance programs are **faster** than having the LM generate intermediate text when you have a set structure.\n\n### Token healing\nUsers deal with text (or bytes) rather than tokens, and thus don't have to worry about [perverse token boundaries issues](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38) such as 'prompt ending in whitespace'.\n\n### Rich templates with f-strings.\n```python\nllama2 + f'''\\\nDo you want a joke or a poem? A {select(['joke', 'poem'])}.\nOkay, here is a one-liner: \"{gen(stop='\"')}\"\n'''\n```\n\u003cimg width=\"358\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/486ca968-89b1-4c02-b914-3b9714fe5890\"\u003e\u003cbr\u003e\n\n### Abstract chat interface that uses correct special tokens for any chat model.\n```python\n# capture our selection under the name 'answer'\nlm = llama2 + f\"Do you want a joke or a poem? A {select(['joke', 'poem'], name='answer')}.\\n\"\n\n# make a choice based on the model's previous selection\nif lm[\"answer\"] == \"joke\":\n    lm += f\"Here is a one-line joke about cats: \" + gen('output', stop='\\n')\nelse:\n    lm += f\"Here is a one-line poem about dogs: \" + gen('output', stop='\\n')\n```\n\u003cimg width=\"393\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/66d47ce7-1d5a-4dbd-b676-66b9c1094184\"\u003e\u003cbr\u003e\n\n### Easy-to-write reusable components.\n```python\nimport guidance\n\n@guidance\ndef one_line_thing(lm, thing, topic):\n    lm += f'Here is a one-line {thing} about {topic}: ' + gen(stop='\\n')\n    return lm # return our updated model\n\n# pick either a joke or a poem\nlm = llama2 + f\"Do you want a joke or a poem? A {select(['joke', 'poem'], name='thing')}.\\n\"\n\n# call our guidance function\nlm += one_line_thing(lm['thing'], 'cats')\n```\n\u003cimg width=\"386\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/60071680-8bbb-4fa5-a298-613d4fd55fa7\"\u003e\u003cbr\u003e\n\n### A library of pre-built components\nCommon syntax elements are available out of the box, below is an example of `substring` for others (like `json`) checkout the [docs](https://guidance.readthedocs.io/en/latest/api.html#functions).\n```python\nfrom guidance import substring\n\n# define a set of possible statements\ntext = 'guidance is awesome. guidance is so great. guidance is the best thing since sliced bread.'\n\n# force the model to make an exact quote\nllama2 + f'Here is a true statement about the guidance library: \"{substring(text)}\"'\n```\n\u003cimg width=\"589\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/9a7178ad-ed73-4e6b-b418-f9d2a3a76b88\"\u003e\u003cbr\u003e\n\n### Streaming support, also integrated with Jupyter notebooks.\n```python\nlm = llama2 + 'Here is a cute 5-line poem about cats and dogs:\\n'\nfor i in range(5):\n    lm += f\"LINE {i+1}: \" + gen(temperature=0.8, suffix=\"\\n\")\n```\n\u003cimg src=\"docs/figures/simple_streaming_example.gif\" width=\"337\"\u003e\n\nFor environments that don't support guidance's rich IPython/Jupyter/HTML based visualizations (e.g. console applications), all visualizations and console outputs can be supressed by setting `echo=False` in the constructor of any `guidance.models` object:\n\n```python\nllama2 = models.LlamaCpp(path, echo=False)\n```\n### Multi-modal support.\n```python\nfrom guidance import image\n\ngemini = models.VertexAI(\"gemini-pro-vision\")\n\nwith user():\n    lm = gemini + \"What is this a picture of?\" + image(\"longs_peak.jpg\")\n\nwith assistant():\n    lm += gen(\"answer\")\n```\n\u003cimg width=\"673\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/6450d05d-52e9-4ef5-b280-8b57e733d46d\"\u003e\n\n\n## Example notebooks\nWe are working on updating our example notebooks. The following ones have been updated:\n- [Basic tutorial](notebooks/tutorials/intro_to_guidance.ipynb)\n- [Chatbot with search](notebooks/chat_with_search.ipynb)  \n\nMore coming soon\n\n## Basic generation\nAn `lm` object is immutable, so you change it by creating new copies of it. By default, when you append things to `lm`, it creates a copy, e.g.:\n```python\nfrom guidance import models, gen, select\nllama2 = models.LlamaCpp(model)\n\n# llama2 is not modified, `lm` is a copy of `llama2` with 'This is a propmt' appended to its state\nlm = llama2 + 'This is a prompt'\n```\n\u003cimg width=\"124\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/c1e96b2b-8f4a-44ee-a8f4-a694a8d7784b\"\u003e\u003cbr\u003e\n\nYou can append _generation_ calls to model objects, e.g.\n```python\nlm = llama2 + 'This is a prompt' + gen(max_tokens=10)\n```\n\u003cimg width=\"267\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/d2e5ed34-ba9d-4bdd-872d-2b76f8e3cf85\"\u003e\u003cbr\u003e\n\nYou can also interleave generation calls with plain text, or control flows:\n```python\n# Note how we set stop tokens\nlm = llama2 + 'I like to play with my ' + gen(stop=' ') + ' in' + gen(stop=['\\n', '.', '!'])\n```\n\u003cimg width=\"279\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/2d47fd65-1982-4dd8-9ba9-a01e62fba455\"\u003e\u003cbr\u003e\n\n## Constrained generation\n### Select (basic)\n`select` constrains generation to a set of options:\n```python\nlm = llama2 + 'I like the color ' + select(['red', 'blue', 'green'])\n```\n\u003cimg width=\"137\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/f0b97629-78a9-439d-90b2-06af31fdc40e\"\u003e\u003cbr\u003e\n\n### Regular expressions\n`gen` has optional arguments `regex` and `stop_regex`, which allow generation (and stopping, respectively) to be controlled by a regex. \n\n#### Regex to constrain generation\nUnconstrained:\n\n```python\nlm = llama2 + 'Question: Luke has ten balls. He gives three to his brother.\\n'\nlm += 'How many balls does he have left?\\n'\nlm += 'Answer: ' + gen(stop='\\n')\n```\n\u003cimg width=\"405\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/55fb66ea-a717-417a-8a70-14c46eba4c66\"\u003e\u003cbr\u003e\n\nConstrained by regex:\n\n```python\nlm = llama2 + 'Question: Luke has ten balls. He gives three to his brother.\\n'\nlm += 'How many balls does he have left?\\n'\nlm += 'Answer: ' + gen(regex='\\d+')\n```\n\u003cimg width=\"404\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/b45a5a79-55e0-4c15-884a-fba830c0a153\"\u003e\u003cbr\u003e\n\n\n#### Regex as stopping criterion\nUnconstrained:\n```python\nlm = llama2 + '19, 18,' + gen(max_tokens=50)\n```\n\u003cimg width=\"359\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/5dd13454-cc42-4e27-a52c-19a31237891c\"\u003e\u003cbr\u003e\n\nStop with traditional stop text, whenever the model generates the number 7:\n```python\nlm = llama2 + '19, 18,' + gen(max_tokens=50, stop='7')\n```\n\u003cimg width=\"73\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/fc96d7c3-381d-4766-8bee-c930669f518a\"\u003e\u003cbr\u003e\n\n \nStop whenever the model generates the character `7` without any numbers around it: \n```python\nlm = llama2 + '19, 18,' + gen(max_tokens=50, stop_regex='[^\\d]7[^\\d]')\n```\n\u003cimg width=\"293\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/a657e566-b1a4-447a-82a5-b88977b5fedf\"\u003e\u003cbr\u003e\n\n\n### Context-free grammars\nWe expose a variety of operators that make it easy to define CFGs, which in turn can be used to constrain generation.\nFor example, we can use the `select` operator (it accepts CFGs as options), `zero_or_more` and `one_or_more` to define a grammar for mathematical expressions:\n```python\nimport guidance\nfrom guidance import one_or_more, select, zero_or_more\n# stateless=True indicates this function does not depend on LLM generations\n@guidance(stateless=True)\ndef number(lm):\n    n = one_or_more(select(['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']))\n    # Allow for negative or positive numbers\n    return lm + select(['-' + n, n])\n\n@guidance(stateless=True)\ndef operator(lm):\n    return lm + select(['+' , '*', '**', '/', '-'])\n\n@guidance(stateless=True)\ndef expression(lm):\n    # Either\n    # 1. A number (terminal)\n    # 2. two expressions with an operator and optional whitespace\n    # 3. An expression with parentheses around it\n    return lm + select([\n        number(),\n        expression() + zero_or_more(' ') +  operator() + zero_or_more(' ') +  expression(),\n        '(' + expression() + ')'\n    ])\n```\n\nThe `@guidance(stateless=True)` decorator makes it such that a function (e.g. `expression`) lives as a stateless grammar that does not get 'executed' until we call `lm + expression()` or `lm += expression()`. For example, here is an example of _unconstrained_ generation:\n```python\n# Without constraints\nlm = llama2 + 'Problem: Luke has a hundred and six balls. He then loses thirty six.\\n'\nlm += 'Equivalent arithmetic expression: ' + gen(stop='\\n') + '\\n'\n```\n\u003cimg width=\"462\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/54af1909-cad4-4fb1-8987-dfdfc02f8f42\"\u003e\u003cbr\u003e\n\nNotice how the model wrote the right equation but solved it (incorrectly). If we wanted to constrain the model such that it only writes valid expressions (without trying to solve them), we can just append our grammar to it:\n```python\ngrammar = expression()\nlm = llama2 + 'Problem: Luke has a hundred and six balls. He then loses thirty six.\\n'\nlm += 'Equivalent arithmetic expression: ' + grammar + '\\n'\n```\n\u003cimg width=\"460\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/dbda0ff8-8edd-4384-b63d-fc98792e0689\"\u003e\u003cbr\u003e\n\nGrammars are very easy to compose. For example, let's say we want a grammar that generates either a mathematical expression or an expression followed by a solution followed by another expression. Creating this grammar is easy:\n\n```python\nfrom guidance import regex\ngrammar = select([expression(), expression() +  regex(' = \\d+; ') + expression()])\n```\nWe can generate according to it:\n```python\nllama2 + 'Here is a math expression for two plus two: ' + grammar\n```\n\u003cimg width=\"346\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/283e6973-0b8d-4153-a82b-9f5db1460da9\"\u003e\u003cbr\u003e\n\n```python\nllama2 + '2 + 2 = 4; 3+3\\n' + grammar\n```\n\u003cimg width=\"109\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/d584a93c-bf24-43d5-8f8d-501e7eb88422\"\u003e\u003cbr\u003e\n\nEven if you don't like thinking in terms of recursive grammars, this formalism makes it easy to constrain generation. For example, let's say we have the following one-shot prompt:\n```python\n@guidance(stateless=True)\ndef ner_instruction(lm, input):\n    lm += f'''\\\n    Please tag each word in the input with PER, ORG, LOC, or nothing\n    ---\n    Input: John worked at Apple.\n    Output:\n    John: PER\n    worked: \n    at: \n    Apple: ORG\n    .: \n    ---\n    Input: {input}\n    Output:\n    '''\n    return lm\ninput = 'Julia never went to Morocco in her life!!'\nllama2 + ner_instruction(input) + gen(stop='---')\n```\n\u003cimg width=\"465\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/8ecf5ad4-68b8-4e7a-b107-b1a5613e4c68\"\u003e\u003cbr\u003e\n\nNotice that the model did not spell the word 'Morocco' correctly. Sometimes the model might also hallucinate a tag that doesn't exist. We can improve this by adding more few-shot examples, etc, but we can also constrain generation to the exact format we want:\n```python\nimport re\n\n@guidance(stateless=True)\ndef constrained_ner(lm, input):\n    # Split into words\n    words = [x for x in re.split('([^a-zA-Z0-9])', input) if x and not re.match('\\s', x)]\n    ret = ''\n    for x in words:\n        ret += x + ': ' + select(['PER', 'ORG', 'LOC', '']) + '\\n'\n    return lm + ret\nllama2 + ner_instruction(input) + constrained_ner(input)\n```\n\u003cimg width=\"462\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/72545093-ef16-479a-b666-bd97c54a5dc7\"\u003e\n\nWhile `constrained_ner(input)` **is** a grammar that constrains the model generation, it _feels_ like you're just writing normal imperative python code with `+=` and `selects`.\n\n### Capture a generation\nThe string generated by a stateless function can be saved to the `lm` object by using the `capture` function. `capture` takes two arguments: the stateless function and the name to store the captured variable.\n\n```python\nfrom guidance import capture, one_or_more\n\nans = lm + \"To close the open bracket sequence [[ the corresponding closing brackets are \" + capture(one_or_more(\"]\"), \"brackets\")\nans[\"brackets\"]\n```\n\u003cimg alt=\"]]]\" src=\"docs/figures/capture_example.png\" width=\"500\"\u003e\n\n## Stateful control + generation\n### State in immutable objects\nWhenever you do `lm + grammar` or `lm + gen`, `lm + select`, etc, you return a new lm object with additional state. For example:\n\n```python\nlm = llama2 + 'This is a prompt' + gen(name='test', max_tokens=10)\nlm += select(['this', 'that'], name='test2')\nlm['test'], lm['test2']\n```\n\u003cimg width=\"296\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/f0f9d180-6209-40df-9401-40da35d46e1a\"\u003e\u003cbr\u003e\n\n### Stateful `{guidance}` functions\nThe guidance decorator is `@guidance(stateless=False)` by default, meaning that a function with this decorator depends on the lm state to execute (either prior state or state generated within the function). For example:\n```python\n@guidance(stateless=False)\ndef test(lm):\n    lm += 'Should I say \"Scott\"?\\n' + select(['yes', 'no'], name='answer') + '\\n'\n    if lm['answer'] == 'yes':\n        lm += 'Scott'\n    else:\n        lm += 'Not Scott'\n    return lm\nllama2 + test()\n```\n\u003cimg width=\"159\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/5a55496b-aea0-46e9-8de6-b63655027653\"\u003e\u003cbr\u003e\n\n\n### Example: ReAct\nA big advantage of stateful control is that you don't have to write any intermediate parsers, and adding follow-up 'prompting' is easy, even if the follow up depends on what the model generates.\nFor example, let's say we want to implement the first example of ReAct prompt in [this](https://www.promptingguide.ai/techniques/react), and let's say the valid acts are only 'Search' or 'Finish'. We might write it like this:\n```python\n@guidance\ndef react_prompt_example(lm, question, max_rounds=10):\n    lm += f'Question: {question}\\n'\n    i = 1\n    while True:\n        lm += f'Thought {i}: ' + gen(suffix='\\n')\n        lm += f'Act {i}: ' + select(['Search', 'Finish'], name='act') \n        lm += '[' + gen(name='arg', suffix=']') + '\\n'\n        if lm['act'] == 'Finish' or i == max_rounds:\n            break\n        else:\n            lm += f'Observation {i}: ' + search(lm['arg']) + '\\n'\n        i += 1\n    return lm\n```\nNotice how we don't have to write a parser for Act and argument and hope that the model generates something valid: we enforce it. Notice also that the loop only stops once the model chooses to act with 'Finish' (or once we hit a maximum number of rounds).\n\n### Example: Changing intermediate step of a Chat session\nWe can also hide or change some of what the model generates. For example, below we get a Chat model (notice we use special `role` blocks) to name some experts to answer a question, but we always remove 'Ferriss' from the list if he is mentioned:\n```python\nfrom guidance import user, system, assistant\nlm = llama2\nquery = 'How can I be more productive?'\nwith system():\n    lm += 'You are a helpful and terse assistant.'\nwith user():\n    lm += f'I want a response to the following question:\\n{query}\\n'\n    lm += 'Name 3 world-class experts (past or present) who would be great at answering this.'\nwith assistant():\n    temp_lm = lm\n    for i in range(1, 4):\n        # This regex only allows strings that look like names (where every word is capitalized)\n        # list_append appends the result to a list\n        temp_lm += f'{i}. ' + gen(regex='([A-Z][a-z]*\\s*)+', suffix='\\n',\n                                  name='experts', list_append=True)\n    experts = [x for x in temp_lm['experts'] if 'Ferriss' not in x]\n    # Notice that even if the model generates 'Ferriss' above,\n    # it doesn't get added to `lm`, only to `temp_lm`\n    lm += ', '.join(experts)\nwith user():\n    lm += 'Please answer the question as if these experts had collaborated in writing an anonymous answer.'\nwith assistant():\n    lm += gen(max_tokens=100)\n```\n\u003cimg width=\"688\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/d274f8b8-52e7-41a5-9635-b34f70ed50e0\"\u003e\u003cbr\u003e\n\n### Automatic interleaving of control and generation: tool use\nTool use is a common case of stateful control. To make it easy to do so, `gen` calls take `tools` as an optional argument, where each tool is defined by (1) a grammar that triggers its call and captures the arguments (if any), and (2) the actual tool call. Then, as generation unrolls, whenever the model generates something that matches the grammar of a tool call, it (1) stops generation, (2) calls the tool (which can append whatever it wants to the LM session), and (3) continues generation.\n\nFor example, here is how we might implement a calculator tool, leveraging our `expression` grammar above:\n```python\nfrom guidance import capture, Tool\n@guidance(stateless=True)\ndef calculator_call(lm):\n    # capture just 'names' the expression, to be saved in the LM state\n    return lm + 'calculator(' + capture(expression(), 'tool_args') + ')'\n\n@guidance\ndef calculator(lm):\n    expression = lm['tool_args']\n    # You typically don't want to run eval directly for security reasons\n    # Here we are guaranteed to only have mathematical expressions\n    lm += f' = {eval(expression)}'\n    return lm\ncalculator_tool = Tool(calculator_call(), calculator)\nlm = llama2 + 'Here are five expressions:\\ncalculator(3 *3) = 33\\ncalculator(2 + 1 * 3) = 5\\n'\nlm += gen(max_tokens=30, tools=[calculator_tool], stop='\\n\\n')\n```\n\u003cimg width=\"201\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/2d9b840a-4fad-4dab-b3e7-20887539b447\"\u003e\u003cbr\u003e\n\n\n### Gsm8k example\nNotice that the calculator is just called seamlessly during generation. Here is a more realistic example of the model solving a gsm8k question:\n\n```python\n@guidance\ndef math_with_calc(lm, question):\n    # Two-shot example\n    lm += '''\\\n    Question: John starts with 2 balls. He then quintupled his number of balls. Then he lost half of them. He then gave 3 to his brother. How many does he have left?\n    Reasoning:\n    1. He quintupled his balls. So he has calculator(2 * 5) = 10 balls.\n    1. He lost half. So he has calculator(10 / 2) = 5 balls.\n    3. He gave 3 to his brother. So he has calculator(5 - 3) = 2 balls.\n    Answer: 2\n\n    Question: Jill get 7 dollars a day in allowance. She uses 1 each day to by a bus pass, then gives half away. How much does she have left each day?\n    Reasoning:\n    1. She gets 7 dollars a day.\n    1. She spends 1 on a bus pass. So she has calculator(5 - 1) = 6.\n    3. She gives half away. So that makes calculator(6 / 2) = 3.\n    Answer: 3\n\n    '''\n    lm += f'Question: {question}\\n'\n    lm += 'Reasoning:\\n' + gen(max_tokens=200, tools=[calculator_tool], stop='Answer')\n    # Only numbers or commas\n    lm += 'Answer: ' + gen(regex='[-\\d,]+')\n    return lm\n\nquestion = '''Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?'''\nllama2 + math_with_calc(question)\n```\n\u003cimg width=\"685\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/0c7b8da0-b295-46cd-a312-604ecfba7b33\"\u003e\u003cbr\u003e\n\n### Automatic call grammar for @guidance functions\nYou can also initialize a `Tool` with any `@guidance`-decorated function, and the default call grammar will be like a python call. Here is an example of using multiple such tools in the same `gen` call:\n```python\n@guidance\ndef say_scott(lm, n):\n    lm += '\\n'\n    for _ in range(int(n)):\n        lm += 'Scott\\n'\n    return lm\n\n@guidance\ndef say_marco(lm, n):\n    lm += '\\n'\n    for _ in range(int(n)):\n        lm += 'marco\\n'\n    return lm\n\ntools = [Tool(callable=say_scott), Tool(callable=say_marco)]\nllama2 + '''\\\nI am going to call say_scott and say_marco a few times:\nsay_scott(1)\nScott\n''' + gen(max_tokens=20, tools=tools)\n```\n\u003cimg width=\"395\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/8025699b-59a1-4a3f-8b1e-a895a54924e2\"\u003e\u003cbr\u003e\n\n\n## Text, not tokens\nThe standard greedy tokenizations used by most language models introduce a variety of subtle and powerful biases, which that can have all kinds of unintended consequences for your prompts.\nFor example, take the following prompt, given to gpt-2 (standard greedy tokenization):\n\nhf_gen(prompt, max_tokens=10)\n```python\nfrom transformers import pipeline\npipe = pipeline(\"text-generation\", model=\"gpt2\")\ndef hf_gen(prompt, max_tokens=100):\n    return pipe(prompt, do_sample=False, max_length=max_tokens, return_full_text=False)[0]['generated_text']\n\nprompt = 'http:'\nhf_gen(prompt, max_tokens=10)\n```\n\u003cimg width=\"198\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/a0fe3e81-89e0-4b4a-8981-edf8b1a8a723\"\u003e\u003cbr\u003e\n\n \n Notice how the output generated by the LLM does not complete the URL with the obvious next characters (two forward slashes). It instead creates an invalid URL string with a space in the middle. Why? Because the string `://` is its own token, and so once the model sees a colon by itself, it assumes that the next characters cannot be `//`; otherwise, the tokenizer would not have used `:`, and instead would have used `://`. This is why there are warnings about ending prompts in whitespace, but the problem is way more pervasive than that: any boundary that may span multiple tokens will cause problems, e.g. notice how a partial word causes incorrect completion:\n\n ```python\nprompt = 'John is a'\nhf_gen(prompt, max_tokens=5)\n```\n\u003cimg width=\"133\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/44906e57-c4ca-4dc3-a1c3-2fdba040259b\"\u003e\u003cbr\u003e\n\n\n ```python\nprompt = 'John is a fo'\nhf_gen(prompt, max_tokens=5)\n```\n\u003cimg width=\"52\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/df649320-ec8e-468a-bb2f-e1994f16c9b6\"\u003e\u003cbr\u003e\n\nWhile problematic enough for normal prompts, these problems would be a disaster in the kinds of prompts we wrote in this readme, where there is interleaving of prompting and generation happening multiple times (and thus multiple opportunities for problems). This is why `{guidance}` implements [token healing](https://towardsdatascience.com/the-art-of-prompt-design-prompt-boundaries-and-token-healing-3b2448b0be38), a feature that deals with prompt boundaries automatically, allowing users to just think in terms of **text** rather than tokens. For example:\n\n```python\nfrom guidance import models\ngpt = models.Transformers('gpt2')\nprompt = 'http:'\ngpt + prompt + gen(max_tokens=10)\n```\n\u003cimg width=\"244\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/c9f26a58-52f2-457c-958a-e048f68eb388\"\u003e\u003cbr\u003e\n\n\n\n```python\nprompt = 'John is a fo'\ngpt + prompt + gen(max_tokens=2)\n```\n\u003cimg width=\"186\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/bc5e4cd4-9b82-4c09-9db2-9e890dad1d69\"\u003e\u003cbr\u003e\n\n## Fast\n### Integrated stateful control is faster\nWe have full control of the decoding loop in our integration with `transformers` and `llamacpp`, allowing us to add control and additional prompt without any extra cost.  \nIf instead we're calling a server, we pay the extra cost of making additional requests, which might be ok if the server has caching, but quickly becomes impractical if the server does not have fine-grained caching. For example, note again the output from the [gsm8k example with calculator](#gsm8k-example) above:\n\n\u003cimg width=\"624\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/2c75b0f2-6997-43d9-b10e-cb9f6f2e2de5\"\u003e\n\nEvery time we call `calculator`, we have to stop generation, append the result to the prompt, and resume generation. To avoid slowing down after the first call, a server would need to keep the KV cache up to '3 for breakfast. So she has calculator(16 - 3)', then roll forward generation from that point on. Even servers that _do_ have caching often don't have a way to guarantee state is preserved at each stop and start, and so user's pay a significant overhead at each interruption. The normal approach of considering everything as a new prompt would cause significant slow downs every time `calculator` is called.\n\n### Guidance acceleration\nIn addition to the benefit above, `{guidance}` calls are often **faster** than running equivalent prompts the traditional way, because we can batch any additional text that is added by the user as execution unrolls (rather than generating it). Take the example below, where we generate a json with a GGUF compressed `llama2` 7B executed using llama.cpp:\n```python\n@guidance\ndef character_maker(lm, id, description, valid_weapons):\n    lm += f\"\"\"\\\n    The following is a character profile for an RPG game in JSON format.\n    ```json\n    {{\n        \"id\": \"{id}\",\n        \"description\": \"{description}\",\n        \"name\": \"{gen('name', stop='\"')}\",\n        \"age\": {gen('age', regex='[0-9]+', stop=',')},\n        \"armor\": \"{select(options=['leather', 'chainmail', 'plate'], name='armor')}\",\n        \"weapon\": \"{select(options=valid_weapons, name='weapon')}\",\n        \"class\": \"{gen('class', stop='\"')}\",\n        \"mantra\": \"{gen('mantra', stop='\"')}\",\n        \"strength\": {gen('strength', regex='[0-9]+', stop=',')},\n        \"items\": [\"{gen('item', list_append=True, stop='\"')}\", \"{gen('item', list_append=True, stop='\"')}\", \"{gen('item', list_append=True, stop='\"')}\"]\n    }}```\"\"\"\n    return lm\na = time.time()\nlm = llama2 + character_maker(1, 'A nimble fighter', ['axe', 'sword', 'bow'])\ntime.time() - a\n```\n\u003cimg width=\"480\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/85b5a181-6e6a-4582-9203-730f49353aeb\"\u003e\u003cbr\u003e\n\nEverything that is not green is not actually generated by the model, and is thus batched (much faster). This prompt takes about 1.2 seconds on an A100 GPU. Now, if we let the model generate everything (as in the roughly equivalent prompt below), it takes roughly `2.6` seconds (not only is it slower, we also have less control over generation). \n```python\n@guidance\ndef character_maker2(lm, id, description):\n    lm += f\"\"\"\\\n    The following is a character profile for an RPG game in JSON format. It has fields 'id', 'description', 'name', 'age', 'armor', weapon', 'class', 'mantra', 'strength', and 'items (just the names of 3 items)'\n    please set description to '{description}'\n    ```json\"\"\" + gen(stop='```')\n    return lm\na = time.time()\nlm = llama2 + character_maker2(1, 'A nimble fighter')\ntime.time() - a\n```\n\u003cimg width=\"586\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/9c55500d-4c90-4f42-9343-43aa2a25efa4\"\u003e\u003cbr\u003e\n\n## Loading models\n### llama.cpp\nInstall the python bindings:\n```bash\nCMAKE_ARGS=\"-DGGML_CUDA=on\" pip install llama-cpp-python\n```\nLoading the model:\n```python\nfrom guidance import models\nlm = models.LlamaCpp(path_to_model, n_gpu_layers=-1)\n```\n\n### Transformers\nInstall transformers:\n```python\nfrom guidance import models\nlm = models.Transformers(model_name_or_path)\n```\n\n\n```python\n@guidance(stateless=True) # Note the stateless=True flag in the decorator -- this enables maximal efficiency on the guidance program execution\ndef character_maker(lm, id, description, valid_weapons):\n    lm += f\"\"\"\\\n    The following is a character profile for an RPG game in JSON format.\n    ```json\n    {{\n        \"id\": \"{id}\",\n        \"description\": \"{description}\",\n        \"name\": \"{gen('name', stop='\"')}\",\n        \"age\": {gen('age', regex='[0-9]+', stop=',')},\n        \"armor\": \"{select(options=['leather', 'chainmail', 'plate'], name='armor')}\",\n        \"weapon\": \"{select(options=valid_weapons, name='weapon')}\",\n        \"class\": \"{gen('class', stop='\"')}\",\n        \"mantra\": \"{gen('mantra', stop='\"')}\",\n        \"strength\": {gen('strength', regex='[0-9]+', stop=',')},\n        \"items\": [\"{gen('item', list_append=True, stop='\"')}\", \"{gen('item', list_append=True, stop='\"')}\", \"{gen('item', list_append=True, stop='\"')}\"]\n    }}```\"\"\"\n    return lm\ncharacter_lm = lm + character_maker(1, 'A nimble fighter', ['axe', 'sword', 'bow']) # Runs on Azure and streams results back\n```\n\n### Vertex AI\nRemote endpoints that don't have explicit guidance integration are run \"optimistically\". This means that all the text that can be forced is given to the model as a prompt (or chat context) and then the model is run in streaming mode without hard constraints (since the remote API doesn't support them). If the model ever violates the contraints then the model stream is stopped and we optionally try it again at that point. This means that all the API-supported control work as expected, and more complex controls/parsing that is not supported by the API work if the model stays consistent with the program.\n```python\npalm2 = models.VertexAI(\"text-bison@001\")\n\nwith instruction():\n    lm = palm2 + \"What is one funny fact about Seattle?\"\n\nlm + gen(\"fact\", max_tokens=100)\n```\n\u003cimg width=\"635\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/693ae08f-68f7-4368-bd25-19afc9bfc0a5\"\u003e\u003cbr\u003e\n\n### OpenAI\nOpenAI endpoint don't have direct support for guidance grammars, but through optimistic running we can still control them in ways that match the model type:\n\n*Legacy completion models:*\n```python\ncurie = models.OpenAI(\"text-curie-001\")\n\ncurie + \"The smallest cats are\" + gen(stop=\".\")\n```\n\u003cimg width=\"263\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/116a906c-ea77-4a13-a83a-682029d5e5c8\"\u003e\u003cbr\u003e\n\n*Instruct tuned models:*\n```python\ngpt_instruct = models.OpenAI(\"gpt-3.5-turbo-instruct\")\n\nwith instruction():\n    lm = gpt_instruct + \"What are the smallest cats?\"\n    \nlm += gen(stop=\".\")\n```\n\u003cimg width=\"574\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/56a53ce1-89f5-4e9d-bdb8-86fb3eebf309\"\u003e\u003cbr\u003e\n\n*Chat models:*\n```python\ngpt = models.OpenAI(\"gpt-3.5-turbo\")\n\nwith system():\n    lm = gpt + \"You are a cat expert.\"\n\nwith user():\n    lm += \"What are the smallest cats?\"\n\nwith assistant():\n    lm += gen(\"answer\", stop=\".\")\n```\n\u003cimg width=\"367\" alt=\"image\" src=\"https://github.com/guidance-ai/guidance/assets/3740613/46102f0f-37dc-4bb1-99b7-e5895bdee772\"\u003e\u003cbr\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidance-ai%2Fguidance","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguidance-ai%2Fguidance","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguidance-ai%2Fguidance/lists"}