{"id":13430959,"url":"https://github.com/Mozilla-Ocho/llamafile","last_synced_at":"2025-03-16T06:31:52.597Z","repository":{"id":208323870,"uuid":"689773665","full_name":"Mozilla-Ocho/llamafile","owner":"Mozilla-Ocho","description":"Distribute and run LLMs with a single file.","archived":false,"fork":false,"pushed_at":"2024-10-26T08:31:33.000Z","size":18131,"stargazers_count":19970,"open_issues_count":135,"forks_count":1003,"subscribers_count":175,"default_branch":"main","last_synced_at":"2024-10-26T21:37:12.734Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://llamafile.ai","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Mozilla-Ocho.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-09-10T21:12:32.000Z","updated_at":"2024-10-26T21:23:33.000Z","dependencies_parsed_at":"2023-12-16T15:44:46.324Z","dependency_job_id":"43fa41a5-2c6f-4591-99f0-fb86857c5eee","html_url":"https://github.com/Mozilla-Ocho/llamafile","commit_stats":null,"previous_names":["mozilla-ocho/llamafile"],"tags_count":30,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mozilla-Ocho%2Fllamafile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mozilla-Ocho%2Fllamafile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mozilla-Ocho%2Fllamafile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Mozilla-Ocho%2Fllamafile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Mozilla-Ocho","download_url":"https://codeload.github.com/Mozilla-Ocho/llamafile/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":221656503,"owners_count":16858784,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T02:00:59.326Z","updated_at":"2025-03-16T06:31:52.583Z","avatar_url":"https://github.com/Mozilla-Ocho.png","language":"C++","funding_links":[],"categories":["Open-Source Local LLM Projects","C++","Agentic AI ![](https://img.shields.io/badge/_-AGENTIC-22d3ee?style=flat-square\u0026logo=openai\u0026logoColor=white)","AI Сервисы для разработки ПО","others","Supporting Infrastructure","By Industry","A01_文本生成_文本对话","⚡ LLM Inference \u0026 Local Serving Engines","Repos","By Language","🏠 Local and Self-Hosted AI","\u003ca name=\"cpp\"\u003e\u003c/a\u003eC++","Local LLM Tools","8. Inference Engines"],"sub_categories":["Apple Silicon optimized (MLX / CoreML / ANE)","Инструменты для развертывания AI-моделей","Local LLM Runners","Other","大语言对话模型及数据","C++","Desktop / Local"],"readme":"# llamafile\n\n[![ci status](https://github.com/Mozilla-Ocho/llamafile/actions/workflows/ci.yml/badge.svg)](https://github.com/Mozilla-Ocho/llamafile/actions/workflows/ci.yml)\u003cbr/\u003e\n[![](https://dcbadge.vercel.app/api/server/YuMNeuKStr)](https://discord.gg/YuMNeuKStr)\u003cbr/\u003e\u003cbr/\u003e\n\n\u003cimg src=\"llamafile/llamafile-640x640.png\" width=\"320\" height=\"320\"\n     alt=\"[line drawing of llama animal head in front of slightly open manilla folder filled with files]\"\u003e\n\n**llamafile lets you distribute and run LLMs with a single file. ([announcement blog post](https://hacks.mozilla.org/2023/11/introducing-llamafile/))**\n\nOur goal is to make open LLMs much more\naccessible to both developers and end users. We're doing that by\ncombining [llama.cpp](https://github.com/ggerganov/llama.cpp) with [Cosmopolitan Libc](https://github.com/jart/cosmopolitan) into one\nframework that collapses all the complexity of LLMs down to\na single-file executable (called a \"llamafile\") that runs\nlocally on most computers, with no installation.\u003cbr/\u003e\u003cbr/\u003e\n\n\u003ca href=\"https://future.mozilla.org\"\u003e\u003cimg src=\"llamafile/mozilla-logo-bw-rgb.png\" width=\"150\"\u003e\u003c/a\u003e\u003cbr/\u003e\nllamafile is a Mozilla Builders project.\u003cbr/\u003e\u003cbr/\u003e\n\n## Quickstart\n\nThe easiest way to try it for yourself is to download our example\nllamafile for the [LLaVA](https://llava-vl.github.io/) model (license: [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/),\n[OpenAI](https://openai.com/policies/terms-of-use)). LLaVA is a new LLM that can do more\nthan just chat; you can also upload images and ask it questions\nabout them. With llamafile, this all happens locally; no data\never leaves your computer.\n\n1. Download [llava-v1.5-7b-q4.llamafile](https://huggingface.co/Mozilla/llava-v1.5-7b-llamafile/resolve/main/llava-v1.5-7b-q4.llamafile?download=true) (4.29 GB).\n\n2. Open your computer's terminal.\n\n3. If you're using macOS, Linux, or BSD, you'll need to grant permission\nfor your computer to execute this new file. (You only need to do this\nonce.)\n\n```sh\nchmod +x llava-v1.5-7b-q4.llamafile\n```\n\n4. If you're on Windows, rename the file by adding \".exe\" on the end.\n\n5. Run the llamafile. e.g.:\n\n```sh\n./llava-v1.5-7b-q4.llamafile\n```\n\n6. Your browser should open automatically and display a chat interface.\n(If it doesn't, just open your browser and point it at http://localhost:8080)\n\n7. When you're done chatting, return to your terminal and hit\n`Control-C` to shut down llamafile.\n\n**Having trouble? See the \"Gotchas\" section below.**\n\n### JSON API Quickstart\n\nWhen llamafile is started, in addition to hosting a web\nUI chat server at \u003chttp://127.0.0.1:8080/\u003e, an [OpenAI\nAPI](https://platform.openai.com/docs/api-reference/chat) compatible\nchat completions endpoint is provided too. It's designed to support the\nmost common OpenAI API use cases, in a way that runs entirely locally.\nWe've also extended it to include llama.cpp specific features (e.g.\nmirostat) that may also be used. For further details on what fields and\nendpoints are available, refer to both the [OpenAI\ndocumentation](https://platform.openai.com/docs/api-reference/chat/create)\nand the [llamafile server\nREADME](llama.cpp/server/README.md#api-endpoints).\n\n\u003cdetails\u003e\n\u003csummary\u003eCurl API Client Example\u003c/summary\u003e\n\nThe simplest way to get started using the API is to copy and paste the\nfollowing curl command into your terminal.\n\n```shell\ncurl http://localhost:8080/v1/chat/completions \\\n-H \"Content-Type: application/json\" \\\n-H \"Authorization: Bearer no-key\" \\\n-d '{\n  \"model\": \"LLaMA_CPP\",\n  \"messages\": [\n      {\n          \"role\": \"system\",\n          \"content\": \"You are LLAMAfile, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\"\n      },\n      {\n          \"role\": \"user\",\n          \"content\": \"Write a limerick about python exceptions\"\n      }\n    ]\n}' | python3 -c '\nimport json\nimport sys\njson.dump(json.load(sys.stdin), sys.stdout, indent=2)\nprint()\n'\n```\n\nThe response that's printed should look like the following:\n\n```json\n{\n   \"choices\" : [\n      {\n         \"finish_reason\" : \"stop\",\n         \"index\" : 0,\n         \"message\" : {\n            \"content\" : \"There once was a programmer named Mike\\nWho wrote code that would often choke\\nHe used try and except\\nTo handle each step\\nAnd his program ran without any hike.\",\n            \"role\" : \"assistant\"\n         }\n      }\n   ],\n   \"created\" : 1704199256,\n   \"id\" : \"chatcmpl-Dt16ugf3vF8btUZj9psG7To5tc4murBU\",\n   \"model\" : \"LLaMA_CPP\",\n   \"object\" : \"chat.completion\",\n   \"usage\" : {\n      \"completion_tokens\" : 38,\n      \"prompt_tokens\" : 78,\n      \"total_tokens\" : 116\n   }\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003ePython API Client example\u003c/summary\u003e\n\nIf you've already developed your software using the [`openai` Python\npackage](https://pypi.org/project/openai/) (that's published by OpenAI)\nthen you should be able to port your app to talk to llamafile instead,\nby making a few changes to `base_url` and `api_key`. This example\nassumes you've run `pip3 install openai` to install OpenAI's client\nsoftware, which is required by this example. Their package is just a\nsimple Python wrapper around the OpenAI API interface, which can be\nimplemented by any server.\n\n```python\n#!/usr/bin/env python3\nfrom openai import OpenAI\nclient = OpenAI(\n    base_url=\"http://localhost:8080/v1\", # \"http://\u003cYour api-server IP\u003e:port\"\n    api_key = \"sk-no-key-required\"\n)\ncompletion = client.chat.completions.create(\n    model=\"LLaMA_CPP\",\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests.\"},\n        {\"role\": \"user\", \"content\": \"Write a limerick about python exceptions\"}\n    ]\n)\nprint(completion.choices[0].message)\n```\n\nThe above code will return a Python object like this:\n\n```python\nChatCompletionMessage(content='There once was a programmer named Mike\\nWho wrote code that would often strike\\nAn error would occur\\nAnd he\\'d shout \"Oh no!\"\\nBut Python\\'s exceptions made it all right.', role='assistant', function_call=None, tool_calls=None)\n```\n\n\u003c/details\u003e\n\n## New v2 Server\n\nWe have a new server that has a better web gui. It also implements\nOpenAI API compatible endpoints, including embeddings. It's designed to\nbe more reliable. It's better able to recycle context windows across\nmultiple slots. To try it, run:\n\n```\nllamafile --server --v2 --help\nllamafile --server --v2\n```\n\n## Other example llamafiles\n\nWe also provide example llamafiles for other models, so you can easily\ntry out llamafile with different kinds of LLMs.\n\n| Model                   | Size     | License                                                                                                                            | llamafile                                                                                                                                                                                      | other quants                                                                        |\n| ---                     | ---      | ---                                                                                                                                | ---                                                                                                                                                                                            | ---                                                                                 |\n| LLaMA 3.2 3B Instruct   | 2.62 GB  | [LLaMA 3.2](https://huggingface.co/Mozilla/Llama-3.2-3B-Instruct-llamafile/blob/main/LICENSE)                                      | [Llama-3.2-3B-Instruct.Q6\\_K.llamafile](https://huggingface.co/Mozilla/Llama-3.2-3B-Instruct-llamafile/blob/main/Llama-3.2-3B-Instruct.Q6_K.llamafile?download=true)                           | [See HF repo](https://huggingface.co/Mozilla/Llama-3.2-3B-Instruct-llamafile)       |\n| LLaMA 3.2 1B Instruct   | 1.11 GB  | [LLaMA 3.2](https://huggingface.co/Mozilla/Llama-3.2-1B-Instruct-llamafile/blob/main/LICENSE)                                      | [Llama-3.2-1B-Instruct.Q6\\_K.llamafile](https://huggingface.co/Mozilla/Llama-3.2-1B-Instruct-llamafile/blob/main/Llama-3.2-1B-Instruct.Q6_K.llamafile?download=true)                           | [See HF repo](https://huggingface.co/Mozilla/Llama-3.2-1B-Instruct-llamafile)       |\n| Gemma 2 2B Instruct     | 2.32 GB  | [Gemma 2](https://huggingface.co/Mozilla/gemma-2-2b-it-llamafile/blob/main/LICENSE)                                                | [gemma-2-2b-it.Q6\\_K.llamafile](https://huggingface.co/Mozilla/gemma-2-2b-it-llamafile/blob/main/gemma-2-2b-it.Q6_K.llamafile?download=true)                                                   | [See HF repo](https://huggingface.co/Mozilla/gemma-2-2b-it-llamafile)               |\n| Gemma 2 9B Instruct     | 7.76 GB  | [Gemma 2](https://huggingface.co/Mozilla/gemma-2-9b-it-llamafile/blob/main/LICENSE)                                                | [gemma-2-9b-it.Q6\\_K.llamafile](https://huggingface.co/Mozilla/gemma-2-9b-it-llamafile/blob/main/gemma-2-9b-it.Q6_K.llamafile?download=true)                                                   | [See HF repo](https://huggingface.co/Mozilla/gemma-2-9b-it-llamafile)               |\n| Gemma 2 27B Instruct    | 22.5 GB  | [Gemma 2](https://huggingface.co/Mozilla/gemma-2-27b-it-llamafile/blob/main/LICENSE)                                               | [gemma-2-27b-it.Q6\\_K.llamafile](https://huggingface.co/Mozilla/gemma-2-27b-it-llamafile/blob/main/gemma-2-27b-it.Q6_K.llamafile?download=true)                                                | [See HF repo](https://huggingface.co/Mozilla/gemma-2-27b-it-llamafile)              |\n| LLaVA 1.5               | 3.97 GB  | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)                                                     | [llava-v1.5-7b-q4.llamafile](https://huggingface.co/Mozilla/llava-v1.5-7b-llamafile/resolve/main/llava-v1.5-7b-q4.llamafile?download=true)                                                     | [See HF repo](https://huggingface.co/Mozilla/llava-v1.5-7b-llamafile)               |\n| TinyLlama-1.1B          | 2.05 GB  | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/)                                                                      | [TinyLlama-1.1B-Chat-v1.0.F16.llamafile](https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-llamafile/resolve/main/TinyLlama-1.1B-Chat-v1.0.F16.llamafile?download=true)                  | [See HF repo](https://huggingface.co/Mozilla/TinyLlama-1.1B-Chat-v1.0-llamafile)    |\n| Mistral-7B-Instruct     | 3.85 GB  | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/)                                                                      | [mistral-7b-instruct-v0.2.Q4\\_0.llamafile](https://huggingface.co/Mozilla/Mistral-7B-Instruct-v0.2-llamafile/resolve/main/mistral-7b-instruct-v0.2.Q4_0.llamafile?download=true)               | [See HF repo](https://huggingface.co/Mozilla/Mistral-7B-Instruct-v0.2-llamafile)    |\n| Phi-3-mini-4k-instruct  | 7.67 GB  | [Apache 2.0](https://huggingface.co/Mozilla/Phi-3-mini-4k-instruct-llamafile/blob/main/LICENSE)                                    | [Phi-3-mini-4k-instruct.F16.llamafile](https://huggingface.co/Mozilla/Phi-3-mini-4k-instruct-llamafile/resolve/main/Phi-3-mini-4k-instruct.F16.llamafile?download=true)                        | [See HF repo](https://huggingface.co/Mozilla/Phi-3-mini-4k-instruct-llamafile)      |\n| Mixtral-8x7B-Instruct   | 30.03 GB | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/)                                                                      | [mixtral-8x7b-instruct-v0.1.Q5\\_K\\_M.llamafile](https://huggingface.co/Mozilla/Mixtral-8x7B-Instruct-v0.1-llamafile/resolve/main/mixtral-8x7b-instruct-v0.1.Q5_K_M.llamafile?download=true)    | [See HF repo](https://huggingface.co/Mozilla/Mixtral-8x7B-Instruct-v0.1-llamafile)  |\n| WizardCoder-Python-34B  | 22.23 GB | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)                                                     | [wizardcoder-python-34b-v1.0.Q5\\_K\\_M.llamafile](https://huggingface.co/Mozilla/WizardCoder-Python-34B-V1.0-llamafile/resolve/main/wizardcoder-python-34b-v1.0.Q5_K_M.llamafile?download=true) | [See HF repo](https://huggingface.co/Mozilla/WizardCoder-Python-34B-V1.0-llamafile) |\n| WizardCoder-Python-13B  | 7.33 GB  | [LLaMA 2](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)                                                     | [wizardcoder-python-13b.llamafile](https://huggingface.co/jartine/wizardcoder-13b-python/resolve/main/wizardcoder-python-13b.llamafile?download=true)                                          | [See HF repo](https://huggingface.co/jartine/wizardcoder-13b-python)                |\n| LLaMA-3-Instruct-70B    | 37.25 GB | [llama3](https://huggingface.co/Mozilla/Meta-Llama-3-8B-Instruct-llamafile/blob/main/Meta-Llama-3-Community-License-Agreement.txt) | [Meta-Llama-3-70B-Instruct.Q4\\_0.llamafile](https://huggingface.co/Mozilla/Meta-Llama-3-70B-Instruct-llamafile/resolve/main/Meta-Llama-3-70B-Instruct.Q4_0.llamafile?download=true)            | [See HF repo](https://huggingface.co/Mozilla/Meta-Llama-3-70B-Instruct-llamafile)   |\n| LLaMA-3-Instruct-8B     | 5.37 GB  | [llama3](https://huggingface.co/Mozilla/Meta-Llama-3-8B-Instruct-llamafile/blob/main/Meta-Llama-3-Community-License-Agreement.txt) | [Meta-Llama-3-8B-Instruct.Q5\\_K\\_M.llamafile](https://huggingface.co/Mozilla/Meta-Llama-3-8B-Instruct-llamafile/resolve/main/Meta-Llama-3-8B-Instruct.Q5_K_M.llamafile?download=true)          | [See HF repo](https://huggingface.co/Mozilla/Meta-Llama-3-8B-Instruct-llamafile)    |\n| Rocket-3B               | 1.89 GB  | [cc-by-sa-4.0](https://creativecommons.org/licenses/by-sa/4.0/deed.en)                                                             | [rocket-3b.Q5\\_K\\_M.llamafile](https://huggingface.co/Mozilla/rocket-3B-llamafile/resolve/main/rocket-3b.Q5_K_M.llamafile?download=true)                                                       | [See HF repo](https://huggingface.co/Mozilla/rocket-3B-llamafile)                   |\n| OLMo-7B                 | 5.68 GB  | [Apache 2.0](https://huggingface.co/Mozilla/OLMo-7B-0424-llamafile/blob/main/LICENSE)                                              | [OLMo-7B-0424.Q6\\_K.llamafile](https://huggingface.co/Mozilla/OLMo-7B-0424-llamafile/resolve/main/OLMo-7B-0424.Q6_K.llamafile?download=true)                                                   | [See HF repo](https://huggingface.co/Mozilla/OLMo-7B-0424-llamafile)                |\n| *Text Embedding Models* |          |                                                                                                                                    |                                                                                                                                                                                                |                                                                                     |\n| E5-Mistral-7B-Instruct  | 5.16 GB  | [MIT](https://choosealicense.com/licenses/mit/)                                                                                    | [e5-mistral-7b-instruct-Q5_K_M.llamafile](https://huggingface.co/Mozilla/e5-mistral-7b-instruct/resolve/main/e5-mistral-7b-instruct-Q5_K_M.llamafile?download=true)                            | [See HF repo](https://huggingface.co/Mozilla/e5-mistral-7b-instruct)                |\n| mxbai-embed-large-v1    | 0.7 GB   | [Apache 2.0](https://choosealicense.com/licenses/apache-2.0/)                                                                      | [mxbai-embed-large-v1-f16.llamafile](https://huggingface.co/Mozilla/mxbai-embed-large-v1-llamafile/resolve/main/mxbai-embed-large-v1-f16.llamafile?download=true)                              | [See HF Repo](https://huggingface.co/Mozilla/mxbai-embed-large-v1-llamafile)        |\n\nHere is an example for the Mistral command-line llamafile:\n\n```sh\n./mistral-7b-instruct-v0.2.Q5_K_M.llamafile --temp 0.7 -p '[INST]Write a story about llamas[/INST]'\n```\n\nAnd here is an example for WizardCoder-Python command-line llamafile:\n\n```sh\n./wizardcoder-python-13b.llamafile --temp 0 -e -r '```\\n' -p '```c\\nvoid *memcpy_sse2(char *dst, const char *src, size_t size) {\\n'\n```\n\nAnd here's an example for the LLaVA command-line llamafile:\n\n```sh\n./llava-v1.5-7b-q4.llamafile --temp 0.2 --image lemurs.jpg -e -p '### User: What do you see?\\n### Assistant:'\n```\n\nAs before, macOS, Linux, and BSD users will need to use the \"chmod\"\ncommand to grant execution permissions to the file before running these\nllamafiles for the first time.\n\nUnfortunately, Windows users cannot make use of many of these example\nllamafiles because Windows has a maximum executable file size of 4GB,\nand all of these examples exceed that size. (The LLaVA llamafile works\non Windows because it is 30MB shy of the size limit.) But don't lose\nheart: llamafile allows you to use external weights; this is described\nlater in this document.\n\n**Having trouble? See the \"Gotchas\" section below.**\n\n## How llamafile works\n\nA llamafile is an executable LLM that you can run on your own\ncomputer. It contains the weights for a given open LLM, as well\nas everything needed to actually run that model on your computer.\nThere's nothing to install or configure (with a few caveats, discussed\nin subsequent sections of this document).\n\nThis is all accomplished by combining llama.cpp with Cosmopolitan Libc,\nwhich provides some useful capabilities:\n\n1. llamafiles can run on multiple CPU microarchitectures. We\nadded runtime dispatching to llama.cpp that lets new Intel systems use\nmodern CPU features without trading away support for older computers.\n\n2. llamafiles can run on multiple CPU architectures. We do\nthat by concatenating AMD64 and ARM64 builds with a shell script that\nlaunches the appropriate one. Our file format is compatible with WIN32\nand most UNIX shells. It's also able to be easily converted (by either\nyou or your users) to the platform-native format, whenever required.\n\n3. llamafiles can run on six OSes (macOS, Windows, Linux,\nFreeBSD, OpenBSD, and NetBSD). If you make your own llama files, you'll\nonly need to build your code once, using a Linux-style toolchain. The\nGCC-based compiler we provide is itself an Actually Portable Executable,\nso you can build your software for all six OSes from the comfort of\nwhichever one you prefer most for development.\n\n4. The weights for an LLM can be embedded within the llamafile.\nWe added support for PKZIP to the GGML library. This lets uncompressed\nweights be mapped directly into memory, similar to a self-extracting\narchive. It enables quantized weights distributed online to be prefixed\nwith a compatible version of the llama.cpp software, thereby ensuring\nits originally observed behaviors can be reproduced indefinitely.\n\n5. Finally, with the tools included in this project you can create your\n*own* llamafiles, using any compatible model weights you want. You can\nthen distribute these llamafiles to other people, who can easily make\nuse of them regardless of what kind of computer they have.\n\n## Using llamafile with external weights\n\nEven though our example llamafiles have the weights built-in, you don't\n*have* to use llamafile that way. Instead, you can download *just* the\nllamafile software (without any weights included) from our releases page.\nYou can then use it alongside any external weights you may have on hand.\nExternal weights are particularly useful for Windows users because they\nenable you to work around Windows' 4GB executable file size limit.\n\nFor Windows users, here's an example for the Mistral LLM:\n\n```sh\ncurl -L -o llamafile.exe https://github.com/Mozilla-Ocho/llamafile/releases/download/0.8.17/llamafile-0.8.17\ncurl -L -o mistral.gguf https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q4_K_M.gguf\n./llamafile.exe -m mistral.gguf\n```\n\nWindows users may need to change `./llamafile.exe` to `.\\llamafile.exe`\nwhen running the above command.\n\n\n## Gotchas and troubleshooting\n\nOn any platform, if your llamafile process is immediately killed, check\nif you have CrowdStrike and then ask to be whitelisted.\n\n### Mac\n\nOn macOS with Apple Silicon you need to have Xcode Command Line Tools\ninstalled for llamafile to be able to bootstrap itself.\n\nIf you use zsh and have trouble running llamafile, try saying `sh -c\n./llamafile`. This is due to a bug that was fixed in zsh 5.9+. The same\nis the case for Python `subprocess`, old versions of Fish, etc.\n\n\n#### Mac error \"... cannot be opened because the developer cannot be verified\"\n\n1. Immediately launch System Settings, then go to Privacy \u0026 Security. llamafile should be listed at the bottom, with a button to Allow.\n2. If not, then change your command in the Terminal to be `sudo spctl --master-disable; [llama launch command]; sudo spctl --master-enable`. This is because `--master-disable` disables _all_ checking, so you need to turn it back on after quitting llama. \n\n### Linux \n\nOn some Linux systems, you might get errors relating to `run-detectors`\nor WINE. This is due to `binfmt_misc` registrations. You can fix that by\nadding an additional registration for the APE file format llamafile\nuses:\n\n```sh\nsudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf\nsudo chmod +x /usr/bin/ape\nsudo sh -c \"echo ':APE:M::MZqFpD::/usr/bin/ape:' \u003e/proc/sys/fs/binfmt_misc/register\"\nsudo sh -c \"echo ':APE-jart:M::jartsr::/usr/bin/ape:' \u003e/proc/sys/fs/binfmt_misc/register\"\n```\n\n### Windows\nAs mentioned above, on Windows you may need to rename your llamafile by\nadding `.exe` to the filename.\n\nAlso as mentioned above, Windows also has a maximum file size limit of 4GB\nfor executables. The LLaVA server executable above is just 30MB shy of\nthat limit, so it'll work on Windows, but with larger models like\nWizardCoder 13B, you need to store the weights in a separate file. An\nexample is provided above; see \"Using llamafile with external weights.\"\n\nOn WSL, there are many possible gotchas. One thing that helps solve them\ncompletely is this:\n\n```\n[Unit]\nDescription=cosmopolitan APE binfmt service\nAfter=wsl-binfmt.service\n\n[Service]\nType=oneshot\nExecStart=/bin/sh -c \"echo ':APE:M::MZqFpD::/usr/bin/ape:' \u003e/proc/sys/fs/binfmt_misc/register\"\n\n[Install]\nWantedBy=multi-user.target\n```\n\nPut that in `/etc/systemd/system/cosmo-binfmt.service`.\n\nEnsure that the APE loader is installed to `/usr/bin/ape`:\n\n```sh\nsudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf\nsudo chmod +x /usr/bin/ape\n```\n\nThen run `sudo systemctl enable --now cosmo-binfmt`.\n\nAnother thing that's helped WSL users who experience issues, is to\ndisable the WIN32 interop feature:\n\n```sh\nsudo sh -c \"echo -1 \u003e /proc/sys/fs/binfmt_misc/WSLInterop\"\n```\n\nIn Windows 11 with WSL 2 the location of the interop flag has changed, as such\nthe following command be required instead/additionally:\n\n```sh\nsudo sh -c \"echo -1 \u003e /proc/sys/fs/binfmt_misc/WSLInterop-late\"\n```\n\nIn the instance of getting a `Permission Denied` on disabling interop\nthrough CLI, it can be permanently disabled by adding the following in\n`/etc/wsl.conf`\n\n```sh\n[interop]\nenabled=false\n```\n\n## Supported OSes\n\nllamafile supports the following operating systems, which require a minimum\nstock install:\n\n- Linux 2.6.18+ (i.e. every distro since RHEL5 c. 2007)\n- Darwin (macOS) 23.1.0+ [1] (GPU is only supported on ARM64)\n- Windows 10+ (AMD64 only)\n- FreeBSD 13+\n- NetBSD 9.2+ (AMD64 only)\n- OpenBSD 7+ (AMD64 only)\n\nOn Windows, llamafile runs as a native portable executable. On UNIX\nsystems, llamafile extracts a small loader program named `ape` to\n`$TMPDIR/.llamafile` or `~/.ape-1.9` which is used to map your model\ninto memory.\n\n[1] Darwin kernel versions 15.6+ *should* be supported, but we currently\n    have no way of testing that.\n\n## Supported CPUs\n\nllamafile supports the following CPUs:\n\n- **AMD64** microprocessors must have AVX. Otherwise llamafile will\n  print an error and refuse to run. This means that if you have an Intel\n  CPU, it needs to be Intel Core or newer (circa 2006+), and if you have\n  an AMD CPU, then it needs to be K8 or newer (circa 2003+). Support for\n  AVX512, AVX2, FMA, F16C, and VNNI are conditionally enabled at runtime\n  if you have a newer CPU. For example, Zen4 has very good AVX512 that\n  can speed up BF16 llamafiles.\n\n- **ARM64** microprocessors must have ARMv8a+. This means everything\n  from Apple Silicon to 64-bit Raspberry Pis will work, provided your\n  weights fit into memory.\n\n## GPU support\n\nllamafile supports the following kinds of GPUs:\n\n- Apple Metal\n- NVIDIA\n- AMD\n\nGPU on MacOS ARM64 is supported by compiling a small module using the\nXcode Command Line Tools, which need to be installed. This is a one time\ncost that happens the first time you run your llamafile. The DSO built\nby llamafile is stored in `$TMPDIR/.llamafile` or `$HOME/.llamafile`.\nOffloading to GPU is enabled by default when a Metal GPU is present.\nThis can be disabled by passing `-ngl 0` or `--gpu disable` to force\nllamafile to perform CPU inference.\n\nOwners of NVIDIA and AMD graphics cards need to pass the `-ngl 999` flag\nto enable maximum offloading. If multiple GPUs are present then the work\nwill be divided evenly among them by default, so you can load larger\nmodels. Multiple GPU support may be broken on AMD Radeon systems. If\nthat happens to you, then use `export HIP_VISIBLE_DEVICES=0` which\nforces llamafile to only use the first GPU.\n\nWindows users are encouraged to use our release binaries, because they\ncontain prebuilt DLLs for both NVIDIA and AMD graphics cards, which only\ndepend on the graphics driver being installed. If llamafile detects that\nNVIDIA's CUDA SDK or AMD's ROCm HIP SDK are installed, then llamafile\nwill try to build a faster DLL that uses cuBLAS or rocBLAS. In order for\nllamafile to successfully build a cuBLAS module, it needs to be run on\nthe x64 MSVC command prompt. You can use CUDA via WSL by enabling\n[Nvidia CUDA on\nWSL](https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl)\nand running your llamafiles inside of WSL. Using WSL has the added\nbenefit of letting you run llamafiles greater than 4GB on Windows.\n\nOn Linux, NVIDIA users will need to install the CUDA SDK (ideally using\nthe shell script installer) and ROCm users need to install the HIP SDK.\nThey're detected by looking to see if `nvcc` or `hipcc` are on the PATH.\n\nIf you have both an AMD GPU *and* an NVIDIA GPU in your machine, then\nyou may need to qualify which one you want used, by passing either\n`--gpu amd` or `--gpu nvidia`.\n\nIn the event that GPU support couldn't be compiled and dynamically\nlinked on the fly for any reason, llamafile will fall back to CPU\ninference.\n\n## Source installation\n\nDeveloping on llamafile requires a modern version of the GNU `make`\ncommand (called `gmake` on some systems), `sha256sum` (otherwise `cc`\nwill be used to build it), `wget` (or `curl`), and `unzip` available at\n[https://cosmo.zip/pub/cosmos/bin/](https://cosmo.zip/pub/cosmos/bin/).\nWindows users need [cosmos bash](https://justine.lol/cosmo3/) shell too.\n\n```sh\nmake -j8\nsudo make install PREFIX=/usr/local\n```\n\nHere's an example of how to generate code for a libc function using the\nllama.cpp command line interface, utilizing WizardCoder-Python-13B\nweights:\n\n```sh\nllamafile \\\n  -m wizardcoder-python-13b-v1.0.Q8_0.gguf \\\n  --temp 0 -r '}\\n' -r '```\\n' \\\n  -e -p '```c\\nvoid *memcpy(void *dst, const void *src, size_t size) {\\n'\n```\n\nHere's a similar example that instead utilizes Mistral-7B-Instruct\nweights for prose composition:\n\n```sh\nllamafile -ngl 9999 \\\n  -m mistral-7b-instruct-v0.1.Q4_K_M.gguf \\\n  -p '[INST]Write a story about llamas[/INST]'\n```\n\nHere's an example of how llamafile can be used as an interactive chatbot\nthat lets you query knowledge contained in training data:\n\n```sh\nllamafile -m llama-65b-Q5_K.gguf -p '\nThe following is a conversation between a Researcher and their helpful AI assistant Digital Athena which is a large language model trained on the sum of human knowledge.\nResearcher: Good morning.\nDigital Athena: How can I help you today?\nResearcher:' --interactive --color --batch_size 1024 --ctx_size 4096 \\\n--keep -1 --temp 0 --mirostat 2 --in-prefix ' ' --interactive-first \\\n--in-suffix 'Digital Athena:' --reverse-prompt 'Researcher:'\n```\n\nHere's an example of how you can use llamafile to summarize HTML URLs:\n\n```sh\n(\n  echo '[INST]Summarize the following text:'\n  links -codepage utf-8 \\\n        -force-html \\\n        -width 500 \\\n        -dump https://www.poetryfoundation.org/poems/48860/the-raven |\n    sed 's/   */ /g'\n  echo '[/INST]'\n) | llamafile -ngl 9999 \\\n      -m mistral-7b-instruct-v0.2.Q5_K_M.gguf \\\n      -f /dev/stdin \\\n      -c 0 \\\n      --temp 0 \\\n      -n 500 \\\n      --no-display-prompt 2\u003e/dev/null\n```\n\nHere's how you can use llamafile to describe a jpg/png/gif/bmp image:\n\n```sh\nllamafile -ngl 9999 --temp 0 \\\n  --image ~/Pictures/lemurs.jpg \\\n  -m llava-v1.5-7b-Q4_K.gguf \\\n  --mmproj llava-v1.5-7b-mmproj-Q4_0.gguf \\\n  -e -p '### User: What do you see?\\n### Assistant: ' \\\n  --no-display-prompt 2\u003e/dev/null\n```\n\nIt's possible to use BNF grammar to enforce the output is predictable\nand safe to use in your shell script. The simplest grammar would be\n`--grammar 'root ::= \"yes\" | \"no\"'` to force the LLM to only print to\nstandard output either `\"yes\\n\"` or `\"no\\n\"`. Another example is if you\nwanted to write a script to rename all your image files, you could say:\n\n```sh\nllamafile -ngl 9999 --temp 0 \\\n    --image lemurs.jpg \\\n    -m llava-v1.5-7b-Q4_K.gguf \\\n    --mmproj llava-v1.5-7b-mmproj-Q4_0.gguf \\\n    --grammar 'root ::= [a-z]+ (\" \" [a-z]+)+' \\\n    -e -p '### User: What do you see?\\n### Assistant: ' \\\n    --no-display-prompt 2\u003e/dev/null |\n  sed -e's/ /_/g' -e's/$/.jpg/'\na_baby_monkey_on_the_back_of_a_mother.jpg\n```\n\nHere's an example of how to run llama.cpp's built-in HTTP server. This\nexample uses LLaVA v1.5-7B, a multimodal LLM that works with llama.cpp's\nrecently-added support for image inputs.\n\n```sh\nllamafile -ngl 9999 \\\n  -m llava-v1.5-7b-Q8_0.gguf \\\n  --mmproj llava-v1.5-7b-mmproj-Q8_0.gguf \\\n  --host 0.0.0.0\n```\n\nThe above command will launch a browser tab on your personal computer to\ndisplay a web interface. It lets you chat with your LLM and upload\nimages to it.\n\n## Creating llamafiles\n\nIf you want to be able to just say:\n\n```sh\n./llava.llamafile\n```\n\n...and have it run the web server without having to specify arguments,\nthen you can embed both the weights and a special `.args` inside, which\nspecifies the default arguments. First, let's create a file named\n`.args` which has this content:\n\n```sh\n-m\nllava-v1.5-7b-Q8_0.gguf\n--mmproj\nllava-v1.5-7b-mmproj-Q8_0.gguf\n--host\n0.0.0.0\n-ngl\n9999\n...\n```\n\nAs we can see above, there's one argument per line. The `...` argument\noptionally specifies where any additional CLI arguments passed by the\nuser are to be inserted. Next, we'll add both the weights and the\nargument file to the executable:\n\n```sh\ncp /usr/local/bin/llamafile llava.llamafile\n\nzipalign -j0 \\\n  llava.llamafile \\\n  llava-v1.5-7b-Q8_0.gguf \\\n  llava-v1.5-7b-mmproj-Q8_0.gguf \\\n  .args\n\n./llava.llamafile\n```\n\nCongratulations. You've just made your own LLM executable that's easy to\nshare with your friends.\n\n## Distribution\n\nOne good way to share a llamafile with your friends is by posting it on\nHugging Face. If you do that, then it's recommended that you mention in\nyour Hugging Face commit message what git revision or released version\nof llamafile you used when building your llamafile. That way everyone\nonline will be able verify the provenance of its executable content. If\nyou've made changes to the llama.cpp or cosmopolitan source code, then\nthe Apache 2.0 license requires you to explain what changed. One way you\ncan do that is by embedding a notice in your llamafile using `zipalign`\nthat describes the changes, and mention it in your Hugging Face commit.\n\n## Documentation\n\nThere's a manual page for each of the llamafile programs installed when you\nrun `sudo make install`. The command manuals are also typeset as PDF\nfiles that you can download from our GitHub releases page. Lastly, most\ncommands will display that information when passing the `--help` flag.\n\n## Running llamafile with models downloaded by third-party applications\n\nThis section answers the question *\"I already have a model downloaded locally by application X, can I use it with llamafile?\"*. The general answer is \"yes, as long as those models are locally stored in GGUF format\" but its implementation can be more or less hacky depending on the application. A few examples (tested on a Mac) follow.\n\n### LM Studio\n[LM Studio](https://lmstudio.ai/) stores downloaded models in `~/.cache/lm-studio/models`, in subdirectories with the same name of the models (following HuggingFace's `account_name/model_name` format), with the same filename you saw when you chose to download the file.\n\n So if you have downloaded e.g. the `llama-2-7b.Q2_K.gguf` file for `TheBloke/Llama-2-7B-GGUF`, you can run llamafile as follows:\n\n```\ncd ~/.cache/lm-studio/models/TheBloke/Llama-2-7B-GGUF\nllamafile -m llama-2-7b.Q2_K.gguf\n```\n\n### Ollama\n\nWhen you download a new model with [ollama](https://ollama.com), all its metadata will be stored in a manifest file under `~/.ollama/models/manifests/registry.ollama.ai/library/`. The directory and manifest file name are the model name as returned by `ollama list`. For instance, for `llama3:latest` the manifest file will be named `.ollama/models/manifests/registry.ollama.ai/library/llama3/latest`.\n\nThe manifest maps each file related to the model (e.g. GGUF weights, license, prompt template, etc) to a sha256 digest. The digest corresponding to the element whose `mediaType` is `application/vnd.ollama.image.model` is the one referring to the model's GGUF file.\n\nEach sha256 digest is also used as a filename in the `~/.ollama/models/blobs` directory (if you look into that directory you'll see *only* those sha256-* filenames). This means you can directly run llamafile by passing the sha256 digest as the model filename. So if e.g. the `llama3:latest` GGUF file digest is `sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29`, you can run llamafile as follows:\n\n```\ncd ~/.ollama/models/blobs\nllamafile -m sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29\n```\n\n## Technical details\n\nHere is a succinct overview of the tricks we used to create the fattest\nexecutable format ever. The long story short is llamafile is a shell\nscript that launches itself and runs inference on embedded weights in\nmilliseconds without needing to be copied or installed. What makes that\npossible is mmap(). Both the llama.cpp executable and the weights are\nconcatenated onto the shell script. A tiny loader program is then\nextracted by the shell script, which maps the executable into memory.\nThe llama.cpp executable then opens the shell script again as a file,\nand calls mmap() again to pull the weights into memory and make them\ndirectly accessible to both the CPU and GPU.\n\n### ZIP weights embedding\n\nThe trick to embedding weights inside llama.cpp executables is to ensure\nthe local file is aligned on a page size boundary. That way, assuming\nthe zip file is uncompressed, once it's mmap()'d into memory we can pass\npointers directly to GPUs like Apple Metal, which require that data be\npage size aligned. Since no existing ZIP archiving tool has an alignment\nflag, we had to write about [500 lines of code](llamafile/zipalign.c) to\ninsert the ZIP files ourselves. However, once there, every existing ZIP\nprogram should be able to read them, provided they support ZIP64. This\nmakes the weights much more easily accessible than they otherwise would\nhave been, had we invented our own file format for concatenated files.\n\n### Microarchitectural portability\n\nOn Intel and AMD microprocessors, llama.cpp spends most of its time in\nthe matmul quants, which are usually written thrice for SSSE3, AVX, and\nAVX2. llamafile pulls each of these functions out into a separate file\nthat can be `#include`ed multiple times, with varying\n`__attribute__((__target__(\"arch\")))` function attributes. Then, a\nwrapper function is added which uses Cosmopolitan's `X86_HAVE(FOO)`\nfeature to runtime dispatch to the appropriate implementation.\n\n### Architecture portability\n\nllamafile solves architecture portability by building llama.cpp twice:\nonce for AMD64 and again for ARM64. It then wraps them with a shell\nscript which has an MZ prefix. On Windows, it'll run as a native binary.\nOn Linux, it'll extract a small 8kb executable called [APE\nLoader](https://github.com/jart/cosmopolitan/blob/master/ape/loader.c)\nto `${TMPDIR:-${HOME:-.}}/.ape` that'll map the binary portions of the\nshell script into memory. It's possible to avoid this process by running\nthe\n[`assimilate`](https://github.com/jart/cosmopolitan/blob/master/tool/build/assimilate.c)\nprogram that comes included with the `cosmocc` compiler. What the\n`assimilate` program does is turn the shell script executable into\nthe host platform's native executable format. This guarantees a fallback\npath exists for traditional release processes when it's needed.\n\n### GPU support\n\nCosmopolitan Libc uses static linking, since that's the only way to get\nthe same executable to run on six OSes. This presents a challenge for\nllama.cpp, because it's not possible to statically link GPU support. The\nway we solve that is by checking if a compiler is installed on the host\nsystem. For Apple, that would be Xcode, and for other platforms, that\nwould be `nvcc`. llama.cpp has a single file implementation of each GPU\nmodule, named `ggml-metal.m` (Objective C) and `ggml-cuda.cu` (Nvidia\nC). llamafile embeds those source files within the zip archive and asks\nthe platform compiler to build them at runtime, targeting the native GPU\nmicroarchitecture. If it works, then it's linked with platform C library\ndlopen() implementation. See [llamafile/cuda.c](llamafile/cuda.c) and\n[llamafile/metal.c](llamafile/metal.c).\n\nIn order to use the platform-specific dlopen() function, we need to ask\nthe platform-specific compiler to build a small executable that exposes\nthese interfaces. On ELF platforms, Cosmopolitan Libc maps this helper\nexecutable into memory along with the platform's ELF interpreter. The\nplatform C library then takes care of linking all the GPU libraries, and\nthen runs the helper program which longjmp()'s back into Cosmopolitan.\nThe executable program is now in a weird hybrid state where two separate\nC libraries exist which have different ABIs. For example, thread local\nstorage works differently on each operating system, and programs will\ncrash if the TLS register doesn't point to the appropriate memory. The\nway Cosmopolitan Libc solves that on AMD is by using SSE to recompile\nthe executable at runtime to change `%fs` register accesses into `%gs`\nwhich takes a millisecond. On ARM, Cosmo uses the `x28` register for TLS\nwhich can be made safe by passing the `-ffixed-x28` flag when compiling\nGPU modules. Lastly, llamafile uses the `__ms_abi__` attribute so that\nfunction pointers passed between the application and GPU modules conform\nto the Windows calling convention. Amazingly enough, every compiler we\ntested, including nvcc on Linux and even Objective-C on MacOS, all\nsupport compiling WIN32 style functions, thus ensuring your llamafile\nwill be able to talk to Windows drivers, when it's run on Windows,\nwithout needing to be recompiled as a separate file for Windows. See\n[cosmopolitan/dlopen.c](https://github.com/jart/cosmopolitan/blob/master/libc/dlopen/dlopen.c)\nfor further details.\n\n## A note about models\n\nThe example llamafiles provided above should not be interpreted as\nendorsements or recommendations of specific models, licenses, or data\nsets on the part of Mozilla.\n\n## Security\n\nllamafile adds pledge() and SECCOMP sandboxing to llama.cpp. This is\nenabled by default. It can be turned off by passing the `--unsecure`\nflag. Sandboxing is currently only supported on Linux and OpenBSD on\nsystems without GPUs; on other platforms it'll simply log a warning.\n\nOur approach to security has these benefits:\n\n1. After it starts up, your HTTP server isn't able to access the\n   filesystem at all. This is good, since it means if someone discovers\n   a bug in the llama.cpp server, then it's much less likely they'll be\n   able to access sensitive information on your machine or make changes\n   to its configuration. On Linux, we're able to sandbox things even\n   further; the only networking related system call the HTTP server will\n   allowed to use after starting up, is accept(). That further limits an\n   attacker's ability to exfiltrate information, in the event that your\n   HTTP server is compromised.\n\n2. The main CLI command won't be able to access the network at all. This\n   is enforced by the operating system kernel. It also won't be able to\n   write to the file system. This keeps your computer safe in the event\n   that a bug is ever discovered in the GGUF file format that lets\n   an attacker craft malicious weights files and post them online. The\n   only exception to this rule is if you pass the `--prompt-cache` flag\n   without also specifying `--prompt-cache-ro`. In that case, security\n   currently needs to be weakened to allow `cpath` and `wpath` access,\n   but network access will remain forbidden.\n\nTherefore your llamafile is able to protect itself against the outside\nworld, but that doesn't mean you're protected from llamafile. Sandboxing\nis self-imposed. If you obtained your llamafile from an untrusted source\nthen its author could have simply modified it to not do that. In that\ncase, you can run the untrusted llamafile inside another sandbox, such\nas a virtual machine, to make sure it behaves how you expect.\n\n## Licensing\n\nWhile the llamafile project is Apache 2.0-licensed, our changes\nto llama.cpp are licensed under MIT (just like the llama.cpp project\nitself) so as to remain compatible and upstreamable in the future,\nshould that be desired.\n\nThe llamafile logo on this page was generated with the assistance of DALL·E 3.\n\n\n[![Star History Chart](https://api.star-history.com/svg?repos=Mozilla-Ocho/llamafile\u0026type=Date)](https://star-history.com/#Mozilla-Ocho/llamafile\u0026Date)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMozilla-Ocho%2Fllamafile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FMozilla-Ocho%2Fllamafile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FMozilla-Ocho%2Fllamafile/lists"}