{"id":14801376,"url":"https://github.com/empower-ai/empower-functions","last_synced_at":"2025-04-08T16:07:52.340Z","repository":{"id":240973056,"uuid":"798953810","full_name":"empower-ai/empower-functions","owner":"empower-ai","description":"GPT-4 level function calling models for real-world tool using use cases","archived":false,"fork":false,"pushed_at":"2024-10-05T00:19:19.000Z","size":1346,"stargazers_count":229,"open_issues_count":1,"forks_count":19,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-01T15:12:20.781Z","etag":null,"topics":["ai","function-calling","llama3","llm","mixtral"],"latest_commit_sha":null,"homepage":"https://www.empower.dev","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/empower-ai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-05-10T20:47:31.000Z","updated_at":"2025-02-27T23:10:07.000Z","dependencies_parsed_at":"2024-05-21T18:43:34.767Z","dependency_job_id":"9bd7d4d8-15e0-451b-b002-96f4ff86d124","html_url":"https://github.com/empower-ai/empower-functions","commit_stats":{"total_commits":14,"total_committers":3,"mean_commits":4.666666666666667,"dds":0.2142857142857143,"last_synced_commit":"6570b0c752280732292b8f94313357151e4c1986"},"previous_names":["empower-ai/empower-functions"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empower-ai%2Fempower-functions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empower-ai%2Fempower-functions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empower-ai%2Fempower-functions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/empower-ai%2Fempower-functions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/empower-ai","download_url":"https://codeload.github.com/empower-ai/empower-functions/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247823854,"owners_count":21002151,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","function-calling","llama3","llm","mixtral"],"created_at":"2024-09-17T20:00:44.260Z","updated_at":"2025-04-08T16:07:52.325Z","avatar_url":"https://github.com/empower-ai.png","language":"Python","funding_links":[],"categories":["NLP","Building"],"sub_categories":["Tools"],"readme":"\u003cpicture\u003e\n  \u003ca href=\"https://empower.dev\"\u003e\u003cimg alt=\"Empower Logo\" width=\"200\" src=\"assets/logo.png\"\u003e\u003c/a\u003e\n\u003c/picture\u003e\n\n# Empower Functions\n\nEmpower Functions is a family of LLMs(large language models) that offer GPT-4 level capabilities for real-world \"tool using\" use cases, with full compatibility support to be served as a drop-in replacement.\n\n[Live Demo](https://app.empower.dev/chat-demo) • [Huggingface Repo](https://huggingface.co/collections/empower-dev/empower-functions-v11-66df72d78c1f7b80bda36f5f) • [Website](https://empower.dev) • [Discord](https://discord.gg/PVaggZ3z6r)\n\n## Update\n\n**New Empower Functions v1.1** \nWe have just launched new [v1.1 of the Empower Functions family](https://huggingface.co/collections/empower-dev/empower-functions-v11-66df72d78c1f7b80bda36f5f). The updated v1.1 family has been fine-tuned based on Llama3.1 using an enhanced curated dataset. It has achieved state-of-the-art performance on the Berkeley Function Calling leader board:\n\n![image](assets/bfcl.png)\n(captured on Sep 10, 2024)\n\n## What is real world \"tool using\" use cases?\n\n\"tool using\" refers to the ability of LLMs to interact with external APIs by recognizing when a function needs to be called and then generating JSON containing the necessary arguments based on user inputs. This capability is essential for building conversational agents and applications that convert natural language into API calls, facilitating tasks such as weather inquiries, data extraction, and interactions with knowledge bases.\n\nReal-world use cases, particularly those involving conversational agents, often introduce complex requirements for LLMs. Models must be capable of retrieving context from multiple round of conversations([multi-turn](docs/inference/multi-turn.md)), choosing between utilizing tools or engaging in standard dialogue (['auto' mode](docs/inference/introduction.md#tools-parameter)), and asking for clarification if any parameters are missing([clarification](docs/inference/clarification.md)). Furthermore, they should integrate responses with tool outputs in a [streaming](docs/inference/streaming.md) fashion. Additionally, when multiple tools are required to complete a task, models should efficiently execute multiple functions either in parallel ([parallel calling](docs/inference/parallel-calling.md)) or sequentially with dependencies ([sequential calling](docs/inference/sequential-calling.md)).\n\nFor example, below is a screenshot demonstrating how the model is used in a medical center coordinator bot. You can explore this further in our [live demo](https://app.empower.dev/chat-demo).\n![image](assets/demo_screenshot.png)\n\n## Family of Models\n\n| Model                          | Specs                                                                                             | Links                                                                                                                                                      | Notes                                 |\n| ------------------------------ | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- |\n| llama3-empower-functions-small | 128k context, based on [Llama3.1 8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B)               | [model](https://huggingface.co/empower-dev/llama3-empower-functions-small-v1.1), [gguf](https://huggingface.co/empower-dev/llama3-empower-functions-small-gguf-v1.1) | Most cost-effective, locally runnable |                                                                                    | Balance in accuracy and cost          |\n| llama3-empower-functions-large | 128k context, based on [Llama3.1 70B](https://huggingface.co/meta-llama/Meta-Llama-3.1-70B)             | [model](https://huggingface.co/empower-dev/llama3-empower-functions-large-v1.1)                                                                                 | Best accuracy                         |\n\n#### Hardware Requirement\n\nWe have tested and the family of models in following setup:\n\n- empower-functions-small: fp16 on 1xA100 40G, GGUF and 4bit GGUF on Macbook M2 Pro with 32G RAM, in minimal the 4bit GGUF version requires 7.56G RAM.\n- empower-functions-large: fp16 on 4xA100 80G\n\n## How to Use?\n\n#### Running Locally\n\n\u003e Running locally is only supported by the `llama3-empower-functions-small` model. To use other models, please use our API.\n\nLocal running is supported through the `empower_functions` pip package, make sure you install it first by running `pip install empower-functions`.\n\n\u003e If you encounter errors like RuntimeError: Failed to load shared library, (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), please re-install the llama-cpp-python package by running `CMAKE_ARGS=\"-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DLLAMA_METAL=on\" pip install --upgrade --verbose --force-reinstall --no-cache-dir llama-cpp-python`\n\n\u003cdetails\u003e\n\u003csummary\u003eRunning a Local OpenAI Compatible Server\u003c/summary\u003e\n\nWe leverage the `llama-cpp-python` project to run the model locally. To start a local OpenAI compatible server, you'll need to follow the steps below:\n\n1. Download the GGUF model from our [huggingface repo](TODO)\n2. Run the command `python -m empower_functions.server --model \u003cpath to GGUF model\u003e --chat_format empower-functions`\n\nYou should see the following output when the server is ready:\n\n`INFO:     Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)`\n\nThen you can use the OpenAI SDK to connect to the server. See below for a basic example:\n\n```python\nimport openai\nimport json\n\nclient = openai.OpenAI(\n    base_url = \"http://localhost:8000/v1\",\n    api_key = \"YOUR_API_KEY\"\n)\n\nmessages = [\n    {\"role\": \"user\", \"content\": \"What's the weather in San Francisco?\"}\n]\n\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_current_weather\",\n            \"description\": \"Get the current weather\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"The city and state, e.g., San Francisco, CA\"\n                    }\n                },\n                \"required\": [\"location\"]\n            }\n        }\n    }\n]\n\nchat_completion = client.chat.completions.create(\n    model=\"does_not_matter\",\n    messages=messages,\n    tools=tools,\n    temperature=0,\n    tool_choice=\"auto\"\n)\n\nprint(chat_completion)\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eRunning in a Python Environment\u003c/summary\u003e\n\nYou can directly call the model in your python environment through the `llama-cpp-python` package with the chat handler provided in the `empower_functions` package. See below for a basic example. For more detailed example, please refer to the [python script](https://github.com/empower-ai/empower-functions/blob/main/examples/llama_cpp_inference.py).\n\n```python\nimport json\nfrom empower_functions import EmpowerFunctionsCompletionHandler\nfrom llama_cpp.llama_tokenizer import LlamaHFTokenizer\nfrom llama_cpp import Llama\n\nllm = Llama.from_pretrained(\n    repo_id=\"empower-dev/llama3-empower-functions-small-gguf\",\n    filename=\"ggml-model-Q4_K_M.gguf\",\n    chat_format=\"llama-3\",\n    chat_handler=EmpowerFunctionsCompletionHandler(),\n    tokenizer=LlamaHFTokenizer.from_pretrained(\"empower-dev/llama3-empower-functions-small-gguf\"),\n    n_gpu_layers=0\n)\n\n# You can then use the llm object to chat with the model\nmessages = [\n    {\"role\": \"user\", \"content\": \"What's the weather in San Francisco?\"}\n]\n\ntools = [\n    {\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_current_weather\",\n            \"description\": \"Get the current weather\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"The city and state, e.g., San Francisco, CA\"\n                    }\n                },\n                \"required\": [\"location\"]\n            }\n        }\n    }\n]\n\nresult = llm.create_chat_completion(\n      messages = messages,\n      tools=tools,\n      tool_choice=\"auto\",\n      max_tokens=128\n)\nprint(json.dumps(result[\"choices\"][0], indent=2))\n\n```\n\n\u003c/details\u003e\n\u003cdetails\u003e\n\u003csummary\u003eRunning in Windows with Cuda\u003c/summary\u003e\n\n* install Nvidia toolkit (I used cuda 12.1): https://developer.nvidia.com/cuda-12-1-1-download-archive?target_os=Windows\u0026target_arch=x86_64\u0026target_version=11\u0026target_type=exe_local\n\n* install Visual Studio with:\nC++ CMake tools for Windows.\nC++ core features\n\n* run this command with the empower_functions virtual environment active in the Windows command prompt (command prompt, not PowerShell):\n`set FORCE_CMAKE=1 \u0026\u0026 set CMAKE_ARGS=-DGGML_CUDA=on -DLLAMA_AVX=off -DLLAMA_AVX2=off -DLLAMA_FMA=off \u0026\u0026 pip install llama-cpp-python --no-cache-dir --force-reinstall --verbose`\n\nThat will take awhile but will overwrite the normal llama-cpp-python module with the Cuda support one.\n\n* then run the server with the virtual environment active with a command like this:\n`python -m empower_functions.server --model C:\\Github\\empower-functions-gpu\\models\\ggml-model-Q4_K_M.gguf --chat_format empower-functions --port 8001 --n_ctx 8196 --n_gpu_layers 20`\n\nreplacing the path with the path where the model is saved on your computer and adjusting n_ctx to the desired context and n_gpu_layers to the amount of the layers to offload to the GPU.\n\u003c/details\u003e\n\n#### Using Empower API\n\nThe empower platform offers an API that is fully compatible with the OpenAI API, allowing you to directly use the OpenAI SDK. An example is shown below. See below for a basic example, more details can be found [here](/docs/inference/introduction.md).\n\nCurrently streaming and JSON model is only available in Empower API.\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"https://app.empower.dev/api/v1\",\n    api_key=\"YOU_API_KEY\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"empower-functions\",\n    messages=[{\"role\": \"user\",\n               \"content\": \"What's the weather in San Francisco and Los Angeles in Celsius?\"}],\n    temperature=0,\n    tools=[{\n        \"type\": \"function\",\n        \"function\": {\n            \"name\": \"get_current_weather\",\n            \"description\": \"Get the current weather in a given location\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"The city and state, e.g. San Francisco, CA\",\n                    },\n                    \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n                },\n                \"required\": [\"location\"],\n            },\n        },\n    }],\n)\nresponse_message = response.choices[0].message.tool_calls\nprint(response_message)\n```\n\n\u003c/details\u003e\n\n#### Prompt Raw Model\n\nThe Empower functions model family has been tuned to natively produce JSON. We provide utilities in our Python package to prompt OpenAI-formatted messages. See below for a basic example, more details can be found [here](/docs/model-prompt.md).\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom prompt import prompt_messages\n\ndevice = \"cuda\"\n\nmodel_path = 'empower-dev/empower-functions-small'\nmodel = AutoModelForCausalLM.from_pretrained(model_path, device_map=\"auto\")\ntokenizer = AutoTokenizer.from_pretrained(model_path)\n\nfunctions = [\n    {\n        \"name\": \"get_current_weather\",\n        \"description\": \"Get the current weather in a given location\",\n        \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"location\": {\n                        \"type\": \"string\",\n                        \"description\": \"The city and state, e.g. San Francisco, CA\",\n                    },\n                    \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]},\n                },\n            \"required\": [\"location\"],\n        },\n    }\n]\n\nmessages = [\n    {'role': 'user', 'content': 'What\\'s the weather in San Francisco and Los Angles in Celsius?'},\n]\n\nmessages = prompt_messages(messages, functions)\nmodel_inputs = tokenizer.apply_chat_template(\n    messages, return_tensors=\"pt\").to(model.device)\n\ngenerated_ids = model.generate(model_inputs, max_new_tokens=128)\ndecoded = tokenizer.batch_decode(generated_ids)\n\nprint(decoded[0])\n```\n\n## Training Approach\n\nEmpower's function models are fine-tuned based on state-of-the-art OSS models. We divided the training into two phases.\n\nFirst, we perform SFT(supervised fine-tuning) using over 100k rows of hand-curated, high-quality conversations involving function calling. These conversations cover different scenarios such as single turn, multi-turn, and parallel calling. Specifically, the model is trained to use beginning tokens to determine whether it is calling functions or returning regular conversation (using \u003cf\u003e and \u003cc\u003e tags). It then returns function calls as JSON or conversations as usual, making streaming integration very straightforward. The SFT sets the model up with a very strong foundation covering various scenarios for general use cases.\n\nNext, we apply DPO (Directly Preference Optimization) for trickier scenario where SFT (Supervised Fine-Tuning) is less effective. For instance, when function specifications include examples for arguments, we want to prevent the model from hallucinating argument values from these examples. We have found DPO to be very effective in correcting such misbehavior with a relatively small amount of data.\n\nFinally, we are committed to continuously optimizing the model for better quality across a wider range of use cases and scenarios :) We can further fine-tune the model based on your specific needs. Please contact us if you have any use-case-specific requirements!\n\n## Evaluation\n\nWe evaluate our models against the Berkeley Function Calling benchmark and both of the 8B and 70B version have achieved the state of the art performance on its size:\n\n![image](assets/bfcl.png)\n(captured on Sep 10, 2024)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fempower-ai%2Fempower-functions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fempower-ai%2Fempower-functions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fempower-ai%2Fempower-functions/lists"}