{"id":15432894,"url":"https://github.com/simonw/llm-gguf","last_synced_at":"2025-04-06T19:11:12.988Z","repository":{"id":249871103,"uuid":"832856278","full_name":"simonw/llm-gguf","owner":"simonw","description":"Run models distributed as GGUF files using LLM","archived":false,"fork":false,"pushed_at":"2024-11-21T07:15:57.000Z","size":12,"stargazers_count":58,"open_issues_count":6,"forks_count":7,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-23T09:30:04.636Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":"simonw"}},"created_at":"2024-07-23T21:30:35.000Z","updated_at":"2025-02-21T08:51:16.000Z","dependencies_parsed_at":"2025-03-23T17:12:24.663Z","dependency_job_id":"651241d7-30e0-4c94-9d2e-892c84d08083","html_url":"https://github.com/simonw/llm-gguf","commit_stats":null,"previous_names":["simonw/llm-gguf"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-gguf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-gguf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-gguf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-gguf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/llm-gguf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247535516,"owners_count":20954576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T18:29:02.240Z","updated_at":"2025-04-06T19:11:12.969Z","avatar_url":"https://github.com/simonw.png","language":"Python","funding_links":["https://github.com/sponsors/simonw"],"categories":["Python"],"sub_categories":[],"readme":"# llm-gguf\n\n[![PyPI](https://img.shields.io/pypi/v/llm-gguf.svg)](https://pypi.org/project/llm-gguf/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/llm-gguf?include_prereleases\u0026label=changelog)](https://github.com/simonw/llm-gguf/releases)\n[![Tests](https://github.com/simonw/llm-gguf/actions/workflows/test.yml/badge.svg)](https://github.com/simonw/llm-gguf/actions/workflows/test.yml)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/llm-gguf/blob/main/LICENSE)\n\nRun models distributed as GGUF files using [LLM](https://llm.datasette.io/)\n\n## Installation\n\nInstall this plugin in the same environment as LLM:\n```bash\nllm install llm-gguf\n```\n## Usage\n\nThis plugin runs models that have been distributed as GGUF files.\n\nYou can either ask the plugin to download these directly, or you can register models you have already downloaded.\n\nTo download the LM Studio GGUF of Llama 3.1 8B Instruct, run the following command:\n\n```bash\nllm gguf download-model \\\n  https://huggingface.co/lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \\\n  --alias llama-3.1-8b-instruct --alias l31i\n```\nThe `--alias` options set aliases for that model, you can omit them if you don't want to set any.\n\nThis command will download the 4.92GB file and store it in the directory revealed by running `llm gguf models-dir` - on macOS this will be `~/Library/Application Support/io.datasette.llm/gguf/models`.\n\nRun `llm models` to confirm that the model has been installed.\n\nYou can then run prompts through that model like this:\n```bash\nllm -m gguf/Meta-Llama-3.1-8B-Instruct-Q4_K_M 'Five great names for a pet lemur'\n```\nOr using one of the aliases that you set like this:\n```bash\nllm -m l31i 'Five great names for a pet lemur'\n```\nYou can start a persistent chat session with the model using `llm chat` - this will avoid having to load the model into memory for each prompt:\n```bash\nllm chat -m l31i\n```\n```\nChatting with gguf/Meta-Llama-3.1-8B-Instruct-Q4_K_M\nType 'exit' or 'quit' to exit\nType '!multi' to enter multiple lines, then '!end' to finish\n\u003e tell me a joke about a walrus, a pelican and a lemur getting lunch\nHere's one: Why did the walrus, the pelican, and the lemur go to the cafeteria for lunch? ...\n```\n\nIf you have downloaded the model already you can register it with the plugin while keeping the file in its current location like this:\n```bash\nllm gguf register-model \\\n  ~/Downloads/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \\\n  --alias llama-3.1-8b-instruct --alias l31i\n```\n\nThis plugin **currently only works with chat models** - these are usually distributed in files with the prefix `-Instruct` or `-Chat` or similar.\n\nFor non-chat models you may have better luck with the older [llm-llama-cpp plugin](https://github.com/simonw/llm-llama-cpp).\n\n## Embedding models\n\nThis plugin also supports [embedding models](https://llm.datasette.io/en/stable/embeddings/index.html) that are distributed as GGUFs.\n\nThese are managed using the `llm gguf embed-models`, `llm gguf download-embed-model` and `llm gguf register-embed-model` commands.\n\nFor example, to start using the excellent and tiny [mxbai-embed-xsmall-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1) model you can download the 30.8MB GGUF version like this:\n\n```bash\nllm gguf download-embed-model \\\n  https://huggingface.co/mixedbread-ai/mxbai-embed-xsmall-v1/resolve/main/gguf/mxbai-embed-xsmall-v1-q8_0.gguf\n```\nThis will store the model in the directory shown if you run `llm gguf models-dir`.\n\nConfirm that the new model is available by running this:\n\n```bash\nllm embed-models\n```\nYou should see `gguf/mxbai-embed-xsmall-v1-q8_0` in the list.\n\nThen try that model out like this:\n\n```bash\nllm embed -m gguf/mxbai-embed-xsmall-v1-q8_0 -c 'hello'\n```\nThis will output a 384 element floating point JSON array.\n\nConsult the [LLM documentation](https://llm.datasette.io/en/stable/embeddings/index.html) for more information on how to use these embeddings.\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n```bash\ncd llm-gguf\npython3 -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\nllm install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-gguf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fllm-gguf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-gguf/lists"}