{"id":13472339,"url":"https://github.com/MeetKai/functionary","last_synced_at":"2025-03-26T15:32:02.313Z","repository":{"id":180592740,"uuid":"665299585","full_name":"MeetKai/functionary","owner":"MeetKai","description":"Chat language model that can use tools and interpret the results","archived":false,"fork":false,"pushed_at":"2024-10-25T05:28:26.000Z","size":3565,"stargazers_count":1393,"open_issues_count":49,"forks_count":107,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-10-25T05:38:33.720Z","etag":null,"topics":["agents","ai","ai-agents","function-calling","llm","ml","python"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MeetKai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-11T22:44:15.000Z","updated_at":"2024-10-25T05:35:09.000Z","dependencies_parsed_at":"2023-09-26T09:03:11.678Z","dependency_job_id":"bd7a3ba4-04b6-4d74-935e-0a130a00a643","html_url":"https://github.com/MeetKai/functionary","commit_stats":null,"previous_names":["musabgultekin/functionary","meetkai/functionary"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeetKai%2Ffunctionary","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeetKai%2Ffunctionary/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeetKai%2Ffunctionary/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MeetKai%2Ffunctionary/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MeetKai","download_url":"https://codeload.github.com/MeetKai/functionary/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222062353,"owners_count":16924722,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["agents","ai","ai-agents","function-calling","llm","ml","python"],"created_at":"2024-07-31T16:00:53.909Z","updated_at":"2025-03-26T15:32:02.306Z","avatar_url":"https://github.com/MeetKai.png","language":"Python","readme":"# Functionary\n\n\u003ca href=\"https://meetkai.com/\"\u003e\n  \u003cimg align=\"right\" width=\"256\" height=\"256\" src=\"https://github.com/meetkai/functionary/assets/3749407/c7a1972d-6ad7-40dc-8000-dceabe6baabd\"\u003e\n\u003c/a\u003e\n\nFunctionary is a language model that can interpret and execute functions/plugins.\n\nThe model determines when to execute functions, whether in parallel or serially, and can understand their outputs. It only triggers functions as needed. Function definitions are given as JSON Schema Objects, similar to OpenAI GPT function calls. \n\n\nDocumentation and more examples: [functionary.meetkai.com](https://functionary.meetkai.com/)\n\n\u003cdetails\u003e\n\n  \u003csummary\u003eChangelog: (click to expand)\u003c/summary\u003e\n\n  + [2024/12/24] We release [meetkai/functionary-v4r-small-preview](https://huggingface.co/meetkai/functionary-v4r-small-preview) - our first version of Functionary that can generate the reasoning steps first before using the tools\n  + [2024/10/21] New server powered by [SGLang](https://github.com/sgl-project/sglang)!\n  + [2024/08/21] We release [meetkai/functionary-small-v3.2](https://huggingface.co/meetkai/functionary-small-v3.2) and [meetkai/functionary-medium-v3.2](https://huggingface.co/meetkai/functionary-medium-v3.2)\n  + [2024/08/11] Our newest model ([meetkai/functionary-medium-v3.1](https://huggingface.co/meetkai/functionary-medium-v3.1)) is ranked 2nd in [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html)\n  + [2024/08/08] We release 128k-context length 70B-model: [meetkai/functionary-medium-v3.1](https://huggingface.co/meetkai/functionary-medium-v3.1) that are based on [meta-llama/Meta-Llama-3.1-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B-Instruct)  \n  + [2024/08/07] We release 2 128k-context length models that are based on [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct): \n     + [meetkai/functionary-small-v3.1](https://huggingface.co/meetkai/functionary-small-v3.1): **using Meta's original prompt template** as described in: [User-defined Custom tool calling](https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1#user-defined-custom-tool-calling)\n     + [meetkai/functionary-small-v3.2](https://huggingface.co/meetkai/functionary-small-v3.2): using **our own prompt template**. This model is **better** than [meetkai/functionary-small-v3.1](https://huggingface.co/meetkai/functionary-small-v3.1)\n  + [2024/06/14] We release [meetkai/functionary-medium-v3.0](https://huggingface.co/meetkai/functionary-medium-v3.0) (based on [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct)) with better capability for function calling\n  + [2024/05/17] We release [meetkai/functionary-small-v2.5](https://huggingface.co/meetkai/functionary-small-v2.5) with better capability for function calling and code interpreter compared with [functionary-small-v2.4](https://huggingface.co/meetkai/functionary-small-v2.4)\n  + [2024/05/06] Streaming support for functionary v2 to v2.4 models is released in [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)!\n  + [2024/05/03] Added support for serverless vLLM deployment on [Modal.com](https://modal.com/)\n  + [2024/04/02] We release [meetkai/functionary-small-v2.4](https://huggingface.co/meetkai/functionary-small-v2.4) and [meetkai/functionary-medium-v2.4](https://huggingface.co/meetkai/functionary-medium-v2.4)! The first functionary models with code-interpreter ability (by passing in `{type: \"code_interpreter\"}` in tools)!\n\n\u003c/details\u003e\n\n## Getting Started\n\nFunctionary can be deployed using either our [vLLM](https://vllm.readthedocs.io/en/latest/getting_started/installation.html) or [SGLang](https://sglang.readthedocs.io/en/latest/install.html) servers. Choose either one depending on your preferences.\n\n### Installation\n\n**vLLM**\n```shell\npip install -e .[vllm]\n```\n**SGLang**\n```shell\npip install -e .[sglang] --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python\n```\n\n### Running the server\n\n#### Small Model\n\n**vLLM**\n```shell\npython3 server_vllm.py --model \"meetkai/functionary-v4r-small-preview\" --host 0.0.0.0 --port 8000 --max-model-len 8192\n```\n**SGLang**\n```shell\npython3 server_sglang.py --model-path \"meetkai/functionary-v4r-small-preview\" --host 0.0.0.0 --port 8000 --context-length 8192\n```\n\n#### Medium Model\n\nOur medium models require: 4xA6000 or 2xA100 80GB to run, need to use: `tensor-parallel-size` or `tp` (SGLang)\n\n**vLLM**\n```shell\n# vllm requires to run this first: https://github.com/vllm-project/vllm/issues/6152\nexport VLLM_WORKER_MULTIPROC_METHOD=spawn\n\npython server_vllm.py --model \"meetkai/functionary-medium-v3.1\" --host 0.0.0.0 --port 8000 --max-model-len 8192 --tensor-parallel-size 2\n```\n**SGLang**\n```shell\npython server_sglang.py --model-path \"meetkai/functionary-medium-v3.1\" --host 0.0.0.0 --port 8000 --context-length 8192 --tp 2\n```\n\n#### LoRA Support (Currently Only in vLLM)\n\nSimilar to [LoRA in vLLM](https://docs.vllm.ai/en/latest/models/lora.html), our server supports serving LoRA adapters both at startup and dynamically.\n\nTo serve a LoRA adapter at startup, run the server with the `--lora-modules` argument:\n\n```shell\npython server_vllm.py --model {BASE_MODEL} --enable-lora --lora-modules {name}={path} {name}={path} --host 0.0.0.0 --port 8000\n```\n\nTo serve a LoRA adapter dynamically, use the `/v1/load_lora_adapter` endpoint:\n```shell\npython server_vllm.py --model {BASE_MODEL} --enable-lora --host 0.0.0.0 --port 8000\n# Load a LoRA adapter dynamically\ncurl -X POST http://localhost:8000/v1/load_lora_adapter \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"lora_name\": \"my_lora\",\n    \"lora_path\": \"/path/to/my_lora_adapter\"\n  }'\n# Example chat request to lora adapter\ncurl -X POST http://localhost:8000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"my_lora\",\n    \"messages\": [...],\n    \"tools\": [...],\n    \"tool_choice\": \"auto\"\n  }'\n# Unload a LoRA adapter dynamically\ncurl -X POST http://localhost:8000/v1/unload_lora_adapter \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"lora_name\": \"my_lora\"\n  }'\n```\n\n\n### Text-Generation-Inference (TGI)\n\nWe also provide a service that performs inference on Functionary models using [Text-Generation-Inference](https://huggingface.co/docs/text-generation-inference/en/index) (TGI). Follow these steps to get started:\n\n1. Install Docker following [their installation instructions](https://docs.docker.com/get-docker/).\n\n2. Install the Docker SDK for Python\n\n```shell\npip install docker\n```\n\n3. Start up the Functionary TGI server\n\nAt start-up, the Functionary TGI server tries to connect to an existing TGI endpoint. In this case, you can run the following:\n\n```shell\npython3 server_tgi.py --model \u003cREMOTE_MODEL_ID_OR_LOCAL_MODEL_PATH\u003e --endpoint \u003cTGI_SERVICE_ENDPOINT\u003e\n```\n\nIf the TGI endpoint does not exist, the Functionary TGI server will start a new TGI endpoint container with the address provided in the `endpoint` CLI argument via the installed Docker Python SDK. Run the following commands for remote and local models respectively:\n\n```shell\npython3 server_tgi.py --model \u003cREMOTE_MODEL_ID\u003e --remote_model_save_folder \u003cPATH_TO_SAVE_AND_CACHE_REMOTE_MODEL\u003e --endpoint \u003cTGI_SERVICE_ENDPOINT\u003e\n```\n\n```shell\npython3 server_tgi.py --model \u003cLOCAL_MODEL_PATH\u003e --endpoint \u003cTGI_SERVICE_ENDPOINT\u003e\n```\n\n4. Make either [OpenAI-compatible](#openai-compatible-usage) or [raw HTTP](#raw-usage) requests to the Functionary TGI server.\n\n\n**Docker**\n\nIf you're having trouble with dependencies, and you have [nvidia-container-toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#setting-up-nvidia-container-toolkit), \nyou can start your environment like this: \n\n```shell\ncd \u003cROOT\u003e\n\n# vLLM\nsudo docker build -t functionary-vllm -f dockerfiles/Dockerfile.vllm .\nsudo docker run --runtime nvidia --gpus all -p 8000:8000 functionary-vllm\n\n# SGLang\nsudo docker build -t functionary-sglang -f dockerfiles/Dockerfile.sgl .\nsudo docker run --runtime nvidia --gpus all -p 8000:8000 functionary-sglang\n```\n\n### OpenAI Compatible Usage\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(base_url=\"http://localhost:8000/v1\", api_key=\"functionary\")\n\nclient.chat.completions.create(\n    model=\"meetkai/functionary-v4r-small-preview\",\n    messages=[{\"role\": \"user\",\n            \"content\": \"What is the weather for Istanbul?\"}\n    ],\n    tools=[{\n            \"type\": \"function\",\n            \"function\": {\n                \"name\": \"get_current_weather\",\n                \"description\": \"Get the current weather\",\n                \"parameters\": {\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"location\": {\n                            \"type\": \"string\",\n                            \"description\": \"The city and state, e.g. San Francisco, CA\"\n                        }\n                    },\n                    \"required\": [\"location\"]\n                }\n            }\n        }],\n    tool_choice=\"auto\"\n)\n```\n\n\n\n### Raw Usage:\n\n\u003cdetails\u003e\n  \u003csummary\u003eDetails (click to expand)\u003c/summary\u003e\n\n```python\nimport requests\n\ndata = {\n    'model': 'meetkai/functionary-v4r-small-preview', # model name here is the value of argument \"--model\" in deploying: server_vllm.py or server.py\n    'messages': [\n        {\n            \"role\": \"user\",\n            \"content\": \"What is the weather for Istanbul?\"\n        }\n    ],\n    'tools':[ # For functionary-7b-v2 we use \"tools\"; for functionary-7b-v1.4 we use \"functions\" = [{\"name\": \"get_current_weather\", \"description\":..., \"parameters\": ....}]\n        {\n            \"type\": \"function\",\n            \"function\": {\n                \"name\": \"get_current_weather\",\n                \"description\": \"Get the current weather\",\n                \"parameters\": {\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"location\": {\n                            \"type\": \"string\",\n                            \"description\": \"The city and state, e.g. San Francisco, CA\"\n                        }\n                    },\n                    \"required\": [\"location\"]\n                }\n            }\n        }\n    ]\n}\n\nresponse = requests.post(\"http://127.0.0.1:8000/v1/chat/completions\", json=data, headers={\n    \"Content-Type\": \"application/json\",\n    \"Authorization\": \"Bearer xxxx\"\n})\n\n# Print the response text\nprint(response.text)\n```\n\n\u003c/details\u003e\n\n\n## Models Available\n| Model                                                                                | Description                                                                                                                         | VRAM FP16 |\n|:-------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------|:------|\n| [meetkai/functionary-v4r-small-preview](https://huggingface.co/meetkai/functionary-v4r-small-preview) | 128k context, code interpreter, using **our own prompt template** | 24GB |\n| [functionary-medium-v3.2](https://huggingface.co/meetkai/functionary-medium-v3.2) | 128k context, code interpreter, using **our own prompt template** | 160GB |\n| [functionary-small-v3.2](https://huggingface.co/meetkai/functionary-small-v3.2) / [GGUF](https://huggingface.co/meetkai/functionary-small-v3.2-GGUF) | 128k context, code interpreter, using **our own prompt template** | 24GB |\n| [functionary-medium-v3.1](https://huggingface.co/meetkai/functionary-medium-v3.1) / [GGUF](https://huggingface.co/meetkai/functionary-medium-v3.1-GGUF) | 128k context, code interpreter, using **original Meta's prompt template** | 160GB |\n| [functionary-small-v3.1](https://huggingface.co/meetkai/functionary-small-v3.1) / [GGUF](https://huggingface.co/meetkai/functionary-small-v3.1-GGUF) | 128k context, code interpreter, using **original Meta's prompt template** | 24GB |\n| [functionary-medium-v3.0](https://huggingface.co/meetkai/functionary-medium-v3.0) / [GGUF](https://huggingface.co/meetkai/functionary-medium-v3.0-GGUF) | 8k context, based on [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | 160GB |\n| [functionary-small-v2.5](https://huggingface.co/meetkai/functionary-small-v2.5) / [GGUF](https://huggingface.co/meetkai/functionary-small-v2.5-GGUF) | 8k context, code interpreter | 24GB |\n| [functionary-small-v2.4](https://huggingface.co/meetkai/functionary-small-v2.4) / [GGUF](https://huggingface.co/meetkai/functionary-small-v2.4-GGUF) | 8k context, code interpreter | 24GB |\n| [functionary-medium-v2.4](https://huggingface.co/meetkai/functionary-medium-v2.4) / [GGUF](https://huggingface.co/meetkai/functionary-medium-v2.4-GGUF) | 8k context, code interpreter, better accuracy | 90GB |\n| [functionary-small-v2.2](https://huggingface.co/meetkai/functionary-small-v2.2) / [GGUF](https://huggingface.co/meetkai/functionary-small-v2.2-GGUF) | 8k context | 24GB |\n| [functionary-medium-v2.2](https://huggingface.co/meetkai/functionary-medium-v2.2) / [GGUF](https://huggingface.co/meetkai/functionary-medium-v2.2-GGUF) | 8k context| 90GB |\n| [functionary-7b-v2.1](https://huggingface.co/meetkai/functionary-7b-v2.1) / [GGUF](https://huggingface.co/meetkai/functionary-7b-v2.1-GGUF)            | 8k context | 24GB |\n| [functionary-7b-v2](https://huggingface.co/meetkai/functionary-7b-v2) / [GGUF](https://huggingface.co/meetkai/functionary-7b-v2-GGUF)                | Parallel function call support.   | 24GB |\n| [functionary-7b-v1.4](https://huggingface.co/meetkai/functionary-7b-v1.4) / [GGUF](https://huggingface.co/meetkai/functionary-7b-v1.4-GGUF)            | 4k context, better accuracy (deprecated) | 24GB |\n| [functionary-7b-v1.1](https://huggingface.co/meetkai/functionary-7b-v1.1)            | 4k context (deprecated)                                                                                                         | 24GB | \n| functionary-7b-v0.1            | 2k context (deprecated) Not recommended, use 2.1 onwards                                                                                                  | 24GB |\n\n### Compatibility information\n\n- v1 models are compatible with both OpenAI-python v0 and v1.\n- v2 models are designed for compatibility with OpenAI-python v1.\n  \nThe difference between OpenAI-python v0 and v1 you may refer to the official documentation [here](https://platform.openai.com/docs/api-reference/chat/create#chat-create-tools)\n\n## The Differences Between Related Projects\n| Feature/Project | [Functionary](https://github.com/MeetKai/functionary) | [NexusRaven](https://github.com/nexusflowai/NexusRaven) | [Gorilla](https://github.com/ShishirPatil/gorilla/tree/main/openfunctions) | [Glaive](https://huggingface.co/glaiveai/glaive-function-calling-v1)| [GPT-4-1106-preview](https://github.com/openai/openai-python) |\n|---|---|---|---|---|---|\n|Single Function Call | ✅ | ✅ | ✅ | ✅ | ✅ |\n|Parallel Function Calls | ✅ | ✅ | ✅ | ❌ | ✅ |\n|Following Up on Missing Function Arguments | ✅ | ❌ | ❌ | ❌ | ✅ |\n|Multi-turn | ✅ | ❌ | ❌ | ✅ | ✅ |\n|Generate Model Responses Grounded in Tools Execution Results | ✅ | ❌ | ❌ | ❌ | ✅ |\n|Chit-Chat | ✅ | ❌ | ✅ | ✅ | ✅ |\n|Code Interpreter | ✅ | ❌ | ❌ | ❌ | ✅ |\n\n**You can find more details of the features in [here](features_desc.md)**\n\n## Llama.cpp Inference\n\n### Llama.cpp Inference using Huggingface Tokenizer\nExample for inference using LLama-cpp-python can be found in: [llama_cpp_inference.py](functionary/examples/llama_cpp_inference.py). \n\n### Integration into Llama-cpp\nBesides, functionary was also integrated into LLama-cpp-python, however the integration might not be **quickly updated**, so if there is something wrong or weird in the result, please use: [llama_cpp_inference.py](functionary/examples/llama_cpp_inference.py) instead. Currently, v2.5 hasn't been integrated, so if you are using **functionary-small-v2.5-GGUF**, please use: [llama_cpp_inference.py](functionary/examples/llama_cpp_inference.py)\n\nMake sure that the latest version of [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) is successully installed in your system. Functionary v2 is fully integrated into llama-cpp-python. You can perform inference using Functionary's GGUF models either via normal chat completion or through llama-cpp-python's OpenAI-compatible server which behaves similarly to ours.\n\nThe following is the sample code using normal chat completion:\n\n```python\nfrom llama_cpp import Llama\nfrom llama_cpp.llama_tokenizer import LlamaHFTokenizer\n\n# We should use HF AutoTokenizer instead of llama.cpp's tokenizer because we found that Llama.cpp's tokenizer doesn't give the same result as that from Huggingface. The reason might be in the training, we added new tokens to the tokenizer and Llama.cpp doesn't handle this successfully\nllm = Llama.from_pretrained(\n    repo_id=\"meetkai/functionary-small-v2.4-GGUF\",\n    filename=\"functionary-small-v2.4.Q4_0.gguf\",\n    chat_format=\"functionary-v2\",\n    tokenizer=LlamaHFTokenizer.from_pretrained(\"meetkai/functionary-small-v2.4-GGUF\"),\n    n_gpu_layers=-1\n)\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"what's the weather like in Hanoi?\"}\n]\ntools = [ # For functionary-7b-v2 we use \"tools\"; for functionary-7b-v1.4 we use \"functions\" = [{\"name\": \"get_current_weather\", \"description\":..., \"parameters\": ....}]\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_current_weather\",\n            \"description\": \"Get the current weather\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"The city and state, e.g., San Francisco, CA\"\n                    }\n                },\n                \"required\": [\"location\"]\n            }\n        }\n    }\n]\n\nresult = llm.create_chat_completion(\n      messages = messages,\n      tools=tools,\n      tool_choice=\"auto\",\n)\n\nprint(result[\"choices\"][0][\"message\"])\n```\nThe output would be:\n```python\n{'role': 'assistant', 'content': None, 'tool_calls': [{'type': 'function', 'function': {'name': 'get_current_weather', 'arguments': '{\\n  \"location\": \"Hanoi\"\\n}'}}]}\n```\n\nFor more details, please refer to the [Function Calling section](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#function-calling) in llama-cpp-python. To use our Functionary GGUF models using llama-cpp-python's OpenAI-compatible server, please refer to [here](https://llama-cpp-python.readthedocs.io/en/latest/server/#function-calling) for more details and documentation.\n\n**Note:**\n- For Functionary in llama-cpp-python, the default system messages are added automatically during the API call. Therefore, there is no need to provide the default system messages in `messages`.\n- Streaming feature for Functionary models in both the normal chat completion and in llama-cpp-python's OpenAI-compatible server is officially supported from v0.2.70 onwards.\n\n\u003c/details\u003e\n\n\n\n## Call Real Python Function\n\nTo call the real python function, get the result and extract the result to respond, you can use [chatlab](https://github.com/rgbkrk/chatlab). The following example uses chatlab==0.16.0:\n\nPlease note that Chatlab currently doesn't support Parallel Function calls. This sample code is compatible only with Functionary Version 1.4 and may not work correctly with Functionary Version 2.0.\n```python\nfrom chatlab import Conversation\nimport openai\nimport os\nopenai.api_key = \"functionary\" # We just need to set this something other than None\nos.environ['OPENAI_API_KEY'] = \"functionary\" # chatlab requires us to set this too\nopenai.api_base = \"http://localhost:8000/v1\"\n\n# now provide the function with description\ndef get_car_price(car_name: str):\n    \"\"\"this function is used to get the price of the car given the name\n    :param car_name: name of the car to get the price\n    \"\"\"\n    car_price = {\n        \"tang\": {\"price\": \"$20000\"},\n        \"song\": {\"price\": \"$25000\"} \n    }\n    for key in car_price:\n        if key in car_name.lower():\n            return {\"price\": car_price[key]}\n    return {\"price\": \"unknown\"}\n\nchat = Conversation(model=\"meetkai/functionary-7b-v2\")\nchat.register(get_car_price)  # register this function\nchat.submit(\"what is the price of the car named Tang?\") # submit user prompt\n\n# print the flow\nfor message in chat.messages:\n    role = message[\"role\"].upper()\n    if \"function_call\" in message:\n        func_name = message[\"function_call\"][\"name\"]\n        func_param = message[\"function_call\"][\"arguments\"]\n        print(f\"{role}: call function: {func_name}, arguments:{func_param}\")\n    else:\n        content = message[\"content\"]\n        print(f\"{role}: {content}\")\n```\n\nThe output will look like this:\n```\nUSER: what is the price of the car named Tang?\nASSISTANT: call function: get_car_price, arguments:{\n  \"car_name\": \"Tang\"\n}\nFUNCTION: {'price': {'price': '$20000'}}\nASSISTANT: The price of the car named Tang is $20,000.\n```\n\n## Serverless Deployment using Modal.com\n\nServerless deployment of Functionary models is supported via the *modal_server_vllm.py* script. After signing up and installing Modal, follow these steps to deploy our vLLM server on Modal:\n\n1. **Create dev environment**\n\n```shell Python\nmodal environment create dev\n```\nIf you have a dev environment created already, there is no need to create another one. Just configure to it in the next step.\n\n2. **Configure dev environment**\n\n```shell Python\nmodal config set-environment dev\n``` \n\n\n3. **Serve Functionary Model**\n    \n```shell Python\nmodal serve modal_server_vllm\n```\n\n4. **Deploy Runner**\n\n```shell Python\nmodal deploy modal_server_vllm\n```\n  \n# Use Cases\n\nHere are a few examples of how you can use this function calling system:\n\n### Travel and Hospitality - Trip Planning\nThe function `plan_trip(destination: string, duration: int, interests: list)` can take user input such as \"I want to plan a 7-day trip to Paris with a focus on art and culture\" and generate an itinerary accordingly.\n\n\u003cdetails\u003e\n  \u003csummary\u003eDetails (click to expand)\u003c/summary\u003e\n\n```python\nclient.chat.completions.create((\n    model=\"meetkai/functionary-7b-v2\",\n    messages=[\n        {\"role\": \"user\", \"content\": 'I want to plan a 7-day trip to Paris with a focus on art and culture'},\n    ], \n    tools=[\n        {\n            \"type\": \"function\",\n            \"function\": {\n                \"name\": \"plan_trip\",\n                \"description\": \"Plan a trip based on user's interests\",\n                \"parameters\": {\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"destination\": {\n                            \"type\": \"string\",\n                            \"description\": \"The destination of the trip\",\n                        },\n                        \"duration\": {\n                            \"type\": \"integer\",\n                            \"description\": \"The duration of the trip in days\",\n                        },\n                        \"interests\": {\n                            \"type\": \"array\",\n                            \"items\": {\"type\": \"string\"},\n                            \"description\": \"The interests based on which the trip will be planned\",\n                        },\n                    },\n                    \"required\": [\"destination\", \"duration\", \"interests\"],\n                }\n            }\n        }    \n    ]\n)\n```\n\nResponse will have: \n\n```json\n{\"role\": \"assistant\", \"content\": null, \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"plan_trip\", \"arguments\": '{\\n  \"destination\": \"Paris\",\\n  \"duration\": 7,\\n  \"interests\": [\"art\", \"culture\"]\\n}'}}]}\n```\n\nThen you need to call ```plan_trip``` function with provided arguments. \nIf you would like a commentary from the model, then you'll call the model again with the response from the function, the model will write necessary commentary.\n\n\u003c/details\u003e\n\n\n### Real Estate - Property Valuation\nA function like estimate_property_value(property_details: dict) could allow users to input details about a property (such as location, size, number of rooms, etc.) and receive an estimated market value.\n\n\u003cdetails\u003e\n  \u003csummary\u003eDetails (click to expand)\u003c/summary\u003e\n\n```python\nclient.chat.completions.create(\n    model=\"meetkai/functionary-7b-v2\",\n    messages=[\n        {\n            \"role\": \"user\", \n            \"content\": 'What is the estimated value of a 3-bedroom house in San Francisco with 2000 sq ft area?'\n        },\n        {\n            \"role\": \"assistant\", \n            \"content\": None, \n            \"tool_calls\": [\n                {\n                    \"type\": \"function\", \n                    \"function\": {\n                        \"name\": \"estimate_property_value\", \n                        \"arguments\": '{\\n  \"property_details\": {\"location\": \"San Francisco\", \"size\": 2000, \"rooms\": 3}\\n}'\n                    }\n                }\n            ]\n        }\n    ], \n    tools=[\n        {\n            \"type\": \"function\",\n            \"function\": {\n                \"name\": \"estimate_property_value\",\n                \"description\": \"Estimate the market value of a property\",\n                \"parameters\": {\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"property_details\": {\n                            \"type\": \"object\",\n                            \"properties\": {\n                                \"location\": {\n                                    \"type\": \"string\",\n                                    \"description\": \"The location of the property\"\n                                },\n                                \"size\": {\n                                    \"type\": \"integer\",\n                                    \"description\": \"The size of the property in square feet\"\n                                },\n                                \"rooms\": {\n                                    \"type\": \"integer\",\n                                    \"description\": \"The number of rooms in the property\"\n                                }\n                            },\n                            \"required\": [\"location\", \"size\", \"rooms\"]\n                        }\n                    },\n                    \"required\": [\"property_details\"]\n                }\n            }\n        }\n    ],\n    tool_choice=\"auto\"\n)\n\n```\n\nResponse will have: \n\n```json\n{\"role\": \"assistant\", \"content\": null, \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"plan_trip\", \"arguments\": '{\\n  \"destination\": \"Paris\",\\n  \"duration\": 7,\\n  \"interests\": [\"art\", \"culture\"]\\n}'}}]}\n```\n\nThen you need to call ```plan_trip``` function with provided arguments. \nIf you would like a commentary from the model, then you'll call the model again with the response from the function, the model will write necessary commentary.\n\n\u003c/details\u003e\n\n\n### Telecommunications - Customer Support\nA function `parse_customer_complaint(complaint: {issue: string, frequency: string, duration: string})` could help in extracting structured information from a complex, narrative customer complaint, identifying the core issue and potential solutions. The `complaint` object could include properties such as `issue` (the main problem), `frequency` (how often the issue occurs), and `duration` (how long the issue has been occurring).\n\n\u003cdetails\u003e\n  \u003csummary\u003eDetails (click to expand)\u003c/summary\u003e\n\n```python\nclient.chat.completions.create(\n    model=\"meetkai/functionary-7b-v2\",\n    messages=[\n        {\"role\": \"user\", \"content\": 'My internet has been disconnecting frequently for the past week'},\n    ], \n    tools=[\n        {\n            \"type\": \"function\",\n            \"function\": {\n            \"name\": \"parse_customer_complaint\",\n            \"description\": \"Parse a customer complaint and identify the core issue\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"complaint\": {\n                        \"type\": \"object\",\n                        \"properties\": {\n                            \"issue\": {\n                                \"type\": \"string\",\n                                \"description\": \"The main problem\",\n                            },\n                            \"frequency\": {\n                                \"type\": \"string\",\n                                \"description\": \"How often the issue occurs\",\n                            },\n                            \"duration\": {\n                                \"type\": \"string\",\n                                \"description\": \"How long the issue has been occurring\",\n                            },\n                        },\n                        \"required\": [\"issue\", \"frequency\", \"duration\"],\n                    },\n                },\n                \"required\": [\"complaint\"],\n            }\n        }\n     }\n    ],\n    tool_choice=\"auto\"\n)\n```\n\nResponse will have:\n\n```json\n{\"role\": \"assistant\", \"content\": null, \"tool_calls\": [{\"type\": \"function\", \"function\": {\"name\": \"parse_customer_complaint\", \"arguments\": '{\\n  \"complaint\": {\"issue\": \"internet disconnecting\", \"frequency\": \"frequently\", \"duration\": \"past week\"}\\n}'}}]}\n```\n\nThen you need to call parse_customer_complaint function with provided arguments.\nIf you would like a commentary from the model, then you'll call the model again with the response from the function, the model will write necessary commentary.\n\n\u003c/details\u003e\n\n\n## How it Works?\n\nWe convert function definitions to a similar text to TypeScript definitions. \nThen we inject these definitions as system prompts. After that, we inject the default system prompt. \nThen we start the conversation messages. \n\nThe prompt example can be found here: [V1](tests/prompt_test_v1.txt) (v1.4), [V2](tests/prompt_test_v2.txt) (v2, v2.1, v2.2, v2.4) and [V2.llama3](tests/prompt_test_v2.llama3.txt) (v2.5)\n\n\nWe don't change the logit probabilities to conform to a certain schema, but the model itself knows how to conform. This allows us to use existing tools and caching systems with ease.\n\n## Evaluation\n\n### Berkeley Function-Calling Leaderboard\nWe are ranked 2nd in the [Berkeley Function-Calling Leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) (Last Updated: 2024-08-11)\n| Model Name          | Function Calling  Accuracy (Name \u0026 Arguments) |\n| :-------------------| ---------------------------: | \n| meetkai/functionary-medium-v3.1  |                       **88.88%**|\n| GPT-4-1106-Preview (Prompt)  |                       88.53%|\n| meetkai/functionary-small-v3.2  |                       82.82%|\n| meetkai/functionary-small-v3.1  |                       82.53%|\n| FireFunction-v2 (FC)  |                        78.82.47% |\n\n\n### ToolSandbox\nWe also evaluate our models on [ToolSandbox](https://github.com/apple/ToolSandbox), this benchmark is much more difficult than **Berkeley Function-Calling Leaderboard**. This benchmark includes stateful tool execution, implicit state dependencies between tools, a built-in user simulator supporting on-policy conversational evaluation and a dynamic evaluation strategy for intermediate and final milestones over an arbitrary trajectory. The authors of this benchmark showed that there is a huge performance gap between open source models and proprietary models.\n\nFrom our evaluation result, our models are comparable to best proprietary models and much better than other open source models.\n\n| Model Name | Average similarity score |\n| :----------| ------------------------: |\n| GPT-4o-2024-05-13 |\t73 |\n| Claude-3-Opus-20240229 |\t69.2 |\n| **Functionary-medium-v3.1** | 68.87 |\n| GPT-3.5-Turbo-0125 | 65.6 |\n| GPT-4-0125-Preview\t| 64.3 |\n| Claude-3-Sonnet-20240229 |\t63.8 |\n| **Functionary-small-v3.1** | 63.13 |\n| Gemini-1.5-Pro-001 | 60.4 |\n| **Functionary-small-v3.2** | 58.56 |\n| Claude-3-Haiku-20240307 | 54.9 |\n| Gemini-1.0-Pro | 38.1 |\t\n| Hermes-2-Pro-Mistral-7B | 31.4 |\n| Mistral-7B-Instruct-v0.3 | 29.8 |\n| C4AI-Command-R-v01 | 26.2 |\n| Gorilla-Openfunctions-v2 | 25.6 |\n| C4AI-Command R+ | 24.7 |\n\n\n\n### Function Prediction Evaluation\nEvaluation function call prediction in SGD dataset. The accuracy metric measures the overall correctness of predicted function calls, including function name prediction and arguments extraction.\n\n\u003cimg align=\"left\" width=\"800\" src=\"assets/Functionary_32.png\"\u003e\n\n| Dataset       | Model Name          | Function Calling  Accuracy (Name \u0026 Arguments) |\n| :-------------| :-------------------| ---------------------------: | \n| SGD | meetkai/functionary-medium-v3.1  |                       **88.11%**|\n| SGD | gpt-4o-2024-05-13  |                       82.75%|\n| SGD | gemini-1.5-flash  |                       79.64%|\n| SGD | c4ai-command-r-plus  |                        45.66% |\n\n\u003c/details\u003e\n\n## Training\n\nSee training [README](functionary/train/README.md)\n\n## Roadmap\n\n- [ ] OpenAPI specification based plugin support.\n- [X] Fast inference server \n  - [X] [vLLM](https://github.com/vllm-project/vllm) \n  - [X] [text-generation-inference](https://github.com/huggingface/text-generation-inference)\n  - [X] Streaming Support\n  - [X] function_call parameter to server\n- [X] Parallel function calling support\n- [X] Python function calling support (Automatic detection of type annotations and calling them automatically)\n- [X] Real world usage examples, such as creating agents.\n- [X] Train Mixtral based model\n- [X] Code interpreter support\n- **Please consider opening a PR for future requests**\n","funding_links":[],"categories":["Table of Contents","A01_文本生成_文本对话","Python","NLP","Building","Agent Integration \u0026 Deployment Tools","Agent Categories"],"sub_categories":["LLM Interpretability Tools","大语言对话模型及数据","3. Pretraining","Tools","AI Agent Deployment","\u003ca name=\"Unclassified\"\u003e\u003c/a\u003eUnclassified"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMeetKai%2Ffunctionary","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMeetKai%2Ffunctionary","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMeetKai%2Ffunctionary/lists"}