{"id":13577032,"url":"https://github.com/zhudotexe/kani","last_synced_at":"2026-03-04T05:07:29.506Z","repository":{"id":182438737,"uuid":"666461223","full_name":"zhudotexe/kani","owner":"zhudotexe","description":"kani (カニ) is a highly hackable microframework for tool-calling language models. (NLP-OSS @ EMNLP 2023)","archived":false,"fork":false,"pushed_at":"2026-02-27T05:04:35.000Z","size":46724,"stargazers_count":599,"open_issues_count":7,"forks_count":29,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-03-02T15:43:28.784Z","etag":null,"topics":["chatgpt","framework","function-calling","gpt-4","large-language-models","llama","llms","microframework","openai","tool-use"],"latest_commit_sha":null,"homepage":"https://kani.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhudotexe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"citation.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"zhudotexe","ko_fi":"zhuexe"}},"created_at":"2023-07-14T15:18:23.000Z","updated_at":"2026-03-01T16:56:34.000Z","dependencies_parsed_at":"2026-03-02T14:01:00.655Z","dependency_job_id":null,"html_url":"https://github.com/zhudotexe/kani","commit_stats":{"total_commits":445,"total_committers":6,"mean_commits":74.16666666666667,"dds":0.04943820224719098,"last_synced_commit":"42d5244922a1ef4a74586b003673616d89ba5a83"},"previous_names":["zhudotexe/kani"],"tags_count":43,"template":false,"template_full_name":null,"purl":"pkg:github/zhudotexe/kani","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fkani","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fkani/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fkani/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fkani/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhudotexe","download_url":"https://codeload.github.com/zhudotexe/kani/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhudotexe%2Fkani/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30071918,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-04T03:25:38.285Z","status":"ssl_error","status_checked_at":"2026-03-04T03:25:05.086Z","response_time":59,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chatgpt","framework","function-calling","gpt-4","large-language-models","llama","llms","microframework","openai","tool-use"],"created_at":"2024-08-01T15:01:17.109Z","updated_at":"2026-03-04T05:07:29.465Z","avatar_url":"https://github.com/zhudotexe.png","language":"Python","funding_links":["https://github.com/sponsors/zhudotexe","https://ko-fi.com/zhuexe"],"categories":["Python","SDK, Libraries, Frameworks","Langchain"],"sub_categories":["Python"],"readme":"\u003cp align=\"center\"\u003e\n  \u003cimg width=\"256\" height=\"256\" alt=\"kani\" src=\"docs/_static/kani-logo@256.png\"\u003e\n\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://github.com/zhudotexe/kani/actions/workflows/pytest.yml\"\u003e\n    \u003cimg alt=\"Test Package\" src=\"https://github.com/zhudotexe/kani/actions/workflows/pytest.yml/badge.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://kani.readthedocs.io/en/latest/?badge=latest\"\u003e\n    \u003cimg alt=\"Documentation Status\" src=\"https://readthedocs.org/projects/kani/badge/?version=latest\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://pypi.org/project/kani/\"\u003e\n    \u003cimg alt=\"PyPI\" src=\"https://img.shields.io/pypi/v/kani\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://colab.research.google.com/github/zhudotexe/kani/blob/main/examples/colab_examples.ipynb\"\u003e\n    \u003cimg alt=\"Quickstart in Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://discord.gg/eTepTNDxYT\"\u003e\n    \u003cimg alt=\"Discord\" src=\"https://img.shields.io/discord/1150902904773935214?color=5865F2\u0026label=discord\u0026logo=discord\u0026logoColor=white\"\u003e\n  \u003c/a\u003e\n  \u003cbr/\u003e\n  \u003ca href=\"examples/4_engines_zoo.py\"\u003e\n    \u003cimg alt=\"Model zoo\" src=\"https://img.shields.io/badge/examples-model_zoo-blue\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"examples/5_advanced_retrieval.py\"\u003e\n    \u003cimg alt=\"Retrieval example\" src=\"https://img.shields.io/badge/examples-retrieval-blue\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# kani (カニ)\n\nkani (カニ) is a lightweight and highly hackable framework for chat-based language models with tool usage/function\ncalling.\n\nCompared to other LM frameworks, kani is less opinionated and offers more fine-grained customizability\nover the parts of the control flow that matter, making it the perfect choice for NLP researchers, hobbyists, and\ndevelopers alike.\n\nkani comes with support for the following models out of the box, with a model-agnostic framework to add support for many\nmore:\n\n**Hosted Models**\n\n- OpenAI Models\n- Anthropic Models\n\n**Open Source Models**\n\nkani supports _every_ chat model available on Hugging Face through `transformers` or `llama.cpp`!\n\n\u003c!--\nIn particular, we have reference implementations for the following base models, and their fine-tunes:\n\n- [LLaMA 3](https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6) (all sizes)\n- [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2),\n  [Mixtral-8x7B](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1),\n  and [Mixtral-8x22B](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1)\n- [Command R](https://huggingface.co/CohereForAI/c4ai-command-r-v01)\n  and [Command R+](https://huggingface.co/CohereForAI/c4ai-command-r-plus)\n- [Gemma](https://huggingface.co/collections/google/gemma-release-65d5efbccdbb8c4202ec078b) (all sizes)\n- [LLaMA 2](https://huggingface.co/meta-llama) (all sizes)\n- [Vicuna v1.3](https://huggingface.co/lmsys/vicuna-7b-v1.3)\n--\u003e\n\nCheck out the [Model Zoo](examples/4_engines_zoo.py) for some examples of loading popular models in Kani!\n\n**Interested in contributing? Check out our\n[guide](https://kani.readthedocs.io/en/latest/community/contributing.html).**\n\n[Read the docs on ReadTheDocs!](http://kani.readthedocs.io/)\n\n[Read our paper on arXiv!](https://arxiv.org/abs/2309.05542)\n\n## Features\n\n- **Lightweight and high-level** - kani implements common boilerplate to interface with language models without forcing\n  you to use opinionated prompt frameworks or complex library-specific tooling.\n- **Model agnostic** - kani provides a simple interface to implement: token counting and completion generation.\n  kani lets developers switch which language model runs on the backend without major code refactors.\n- **Automatic chat memory management** - Allow chat sessions to flow without worrying about managing the number of\n  tokens in the history - kani takes care of it.\n- **Function calling with model feedback and retry** - Give models access to functions in just one line of code.\n  kani elegantly provides feedback about hallucinated parameters and errors and allows the model to retry calls.\n- **You control the prompts** - There are no hidden prompt hacks. We will never decide for you how to format your own\n  data, unlike other popular language model libraries.\n- **Fast to iterate and intuitive to learn** - With kani, you only write Python - we handle the rest.\n- **Asynchronous design from the start** - kani can scale to run multiple chat sessions in parallel easily, without\n  having to manage multiple processes or programs.\n\n## Installation\n\nkani requires Python 3.10 or above. To install model-specific dependencies, kani uses various extras (brackets after\nthe library name in `pip install`). To determine which extra(s) to install, see\nthe [model table](https://kani.readthedocs.io/en/latest/engines.html), or use the `[all]` extra to install everything.\n\n```shell\n# for OpenAI models\n$ pip install \"kani[openai]\"\n# for Hugging Face models\n$ pip install \"kani[huggingface]\" torch\n# or install everything:\n$ pip install \"kani[all]\"\n```\n\nFor the most up-to-date changes and new models, you can also install the development version from Git's `main` branch:\n\n```shell\n$ pip install \"kani[all] @ git+https://github.com/zhudotexe/kani.git@main\"\n```\n\n## Quickstart\n\n\u003ca href=\"https://colab.research.google.com/github/zhudotexe/kani/blob/main/examples/colab_examples.ipynb\"\u003e\n  \u003cimg alt=\"Quickstart in Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"\u003e\n\u003c/a\u003e\n\nkani requires Python 3.10 or above.\n\nFirst, install the library. In this quickstart, we'll use the OpenAI engine, though kani\nis [model-agnostic](https://kani.readthedocs.io/en/latest/engines.html).\n\n```shell\n$ pip install \"kani[openai]\"\n```\n\nThen, let's use kani to create a simple chatbot using ChatGPT as a backend.\n\n```python\n# import the library\nimport asyncio\nfrom kani import Kani, chat_in_terminal\nfrom kani.engines.openai import OpenAIEngine\n\n# Replace this with your OpenAI API key: https://platform.openai.com/account/api-keys\napi_key = \"sk-...\"\n\n# kani uses an Engine to interact with the language model. You can specify other model \n# parameters here, like temperature=0.7.\nengine = OpenAIEngine(api_key, model=\"gpt-4o-mini\")\n\n# The kani manages the chat state, prompting, and function calling. Here, we only give \n# it the engine to call ChatGPT, but you can specify other parameters like \n# system_prompt=\"You are...\" here.\nai = Kani(engine)\n\n# kani comes with a utility to interact with a kani through your terminal...\nchat_in_terminal(ai)\n\n\n# or you can use kani programmatically in an async function!\nasync def main():\n    resp = await ai.chat_round(\"What is the airspeed velocity of an unladen swallow?\")\n    print(resp.text)\n\n\nasyncio.run(main())\n```\n\nkani makes the time to set up a working chat model short, while offering the programmer deep customizability over\nevery prompt, function call, and even the underlying language model.\n\n## Function Calling\n\nFunction calling gives language models the ability to choose when to call a function you provide based off its\ndocumentation.\n\nWith kani, you can write functions in Python and expose them to the model with just one line of code: the `@ai_function`\ndecorator.\n\n```python\n# import the library\nimport asyncio\nfrom typing import Annotated\nfrom kani import AIParam, Kani, ai_function, chat_in_terminal, ChatRole\nfrom kani.engines.openai import OpenAIEngine\n\n# set up the engine as above\napi_key = \"sk-...\"\nengine = OpenAIEngine(api_key, model=\"gpt-4o-mini\")\n\n\n# subclass Kani to add AI functions\nclass MyKani(Kani):\n    # Adding the annotation to a method exposes it to the AI\n    @ai_function()\n    def get_weather(\n        self,\n        # and you can provide extra documentation about specific parameters\n        location: Annotated[str, AIParam(desc=\"The city and state, e.g. San Francisco, CA\")],\n    ):\n        \"\"\"Get the current weather in a given location.\"\"\"\n        # In this example, we mock the return, but you could call a real weather API\n        return f\"Weather in {location}: Sunny, 72 degrees fahrenheit.\"\n\n\nai = MyKani(engine)\n\n# the terminal utility allows you to test function calls...\nchat_in_terminal(ai)\n\n\n# and you can track multiple rounds programmatically.\nasync def main():\n    async for msg in ai.full_round(\"What's the weather in Tokyo?\"):\n        print(msg.role, msg.text)\n\n\nasyncio.run(main())\n```\n\nkani guarantees that function calls are valid by the time they reach your methods while allowing you to focus on\nwriting code. For more information, check\nout [the function calling docs](https://kani.readthedocs.io/en/latest/function_calling.html).\n\n## Streaming\n\nkani supports streaming responses from the underlying language model token-by-token, even in the presence of function\ncalls. Streaming is designed to be a drop-in superset of the ``chat_round`` and ``full_round`` methods, allowing you to\ngradually refactor your code without ever leaving it in a broken state.\n\n```python\nasync def stream_chat():\n    stream = ai.chat_round_stream(\"What does kani mean?\")\n    async for token in stream:\n        print(token, end=\"\")\n    print()\n    msg = await stream.message()  # or `await stream`\n\n\nasync def stream_with_function_calling():\n    async for stream in ai.full_round_stream(\"What's the weather in Tokyo?\"):\n        async for token in stream:\n            print(token, end=\"\")\n        print()\n        msg = await stream.message()\n```\n\n## Why kani?\n\nExisting frameworks for language models like LangChain and simpleaichat are opinionated and/or heavyweight - they edit\ndevelopers' prompts under the hood, are challenging to learn, and are difficult to customize without adding a lot of\nhigh-maintenance bloat to your codebase.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg style=\"max-width: 800px;\" alt=\"kani\" src=\"docs/_static/lib-comparison_white.png\"\u003e\n\u003c/p\u003e\n\nWe built kani as a more flexible, simple, and robust alternative. A good analogy between frameworks would be to say that\nkani is to LangChain as Flask (or FastAPI) is to Django.\n\nkani is appropriate for everyone from academic researchers to industry professionals to hobbyists to use without\nworrying about under-the-hood hacks.\n\n## Docs\n\nTo learn more about how\nto [customize kani with your own prompt wrappers](https://kani.readthedocs.io/en/latest/customization.html),\n[function calling](https://kani.readthedocs.io/en/latest/function_calling.html), and\nmore, [read the docs!](http://kani.readthedocs.io/)\n\nOr take a look at the hands-on examples [in this repo](https://github.com/zhudotexe/kani/tree/main/examples).\n\n## Demo\n\nWant to see kani in action? Using 4-bit quantization to shrink the model, we run LLaMA v2 as part of our test suite\nright on GitHub Actions:\n\nhttps://github.com/zhudotexe/kani/actions/workflows/pytest.yml?query=branch%3Amain+is%3Asuccess\n\nSimply click on the latest build to see LLaMA's output!\n\n## Who we are\n\n\u003cimg alt=\"University of Pennsylvania Logo\" src=\"docs/_static/penn-logo.jpg\" width=\"300\"\u003e\n\nThe core development team is made of three PhD students in the Department of Computer and Information Science at the\nUniversity of Pennsylvania. We're all members of\n[Prof. Chris Callison-Burch's](https://www.cis.upenn.edu/~ccb/) lab, working towards advancing the future of NLP.\n\n- [**Andrew Zhu**](https://zhu.codes/) started in Fall 2022. His research interests include natural language processing,\n  programming languages, distributed systems, and more. He's also a full-stack software engineer, proficient in all\n  manner of backend, devops, database, and frontend engineering. Andrew strives to make idiomatic, clean, performant,\n  and low-maintenance code — philosophies that are often rare in academia. His research is supported by the NSF Graduate\n  Research Fellowship.\n- [**Liam Dugan**](https://liamdugan.com/) started in Fall 2021. His research focuses primarily on large language models\n  and how humans interact with them. In particular, he is interested in human detection of generated text and whether we\n  can apply those insights to automatic detection systems. He is also interested in the practical application of large\n  language models to education.\n- [**Alyssa Hwang**](https://alyssahwang.com/) started in Fall 2020 and is advised by Chris Callison-Burch and Andrew\n  Head. Her research focuses on AI assistants that effectively communicate complex information, like voice assistants\n  guiding users through instructions or audiobooks allowing users to seamlessly navigate through spoken text. Beyond\n  research, Alyssa chairs the Penn CIS Doctoral Association, founded the CIS PhD Mentorship Program, and was supported\n  by the NSF Graduate Research Fellowship Program.\n\nWe use kani actively in our research, and aim to keep it up-to-date with modern NLP practices.\n\n## Citation\n\nIf you use Kani, please cite us as:\n\n```\n@inproceedings{zhu-etal-2023-kani,\n    title = \"Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications\",\n    author = \"Zhu, Andrew  and\n      Dugan, Liam  and\n      Hwang, Alyssa  and\n      Callison-Burch, Chris\",\n    editor = \"Tan, Liling  and\n      Milajevs, Dmitrijs  and\n      Chauhan, Geeticka  and\n      Gwinnup, Jeremy  and\n      Rippeth, Elijah\",\n    booktitle = \"Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)\",\n    month = dec,\n    year = \"2023\",\n    address = \"Singapore\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2023.nlposs-1.8\",\n    doi = \"10.18653/v1/2023.nlposs-1.8\",\n    pages = \"65--77\",\n}\n```\n\n### Acknowledgements\n\nWe would like to thank the members of the lab of Chris Callison-Burch for their testing and detailed feedback on the\ncontents of both our paper and the Kani repository. In addition, we’d like to thank Henry Zhu (no relation to the first\nauthor) for his early and enthusiastic support of the project.\n\nThis research is based upon work supported in part by the Air Force Research Laboratory (contract FA8750-23-C-0507), the\nIARPA HIATUS Program (contract 2022-22072200005), and the NSF (Award 1928631). Approved for Public Release, Distribution\nUnlimited. The views and conclusions contained herein are those of the authors and should not be interpreted as\nnecessarily representing the official policies, either expressed or implied, of IARPA, NSF, or the U.S. Government.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhudotexe%2Fkani","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhudotexe%2Fkani","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhudotexe%2Fkani/lists"}