{"id":21532509,"url":"https://github.com/monk1337/auto-ollama","last_synced_at":"2025-04-10T00:32:25.569Z","repository":{"id":230528846,"uuid":"779558888","full_name":"monk1337/auto-ollama","owner":"monk1337","description":"run ollama \u0026 gguf easily with a single command","archived":false,"fork":false,"pushed_at":"2024-05-15T12:55:28.000Z","size":97,"stargazers_count":50,"open_issues_count":5,"forks_count":5,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-28T15:47:07.733Z","etag":null,"topics":["autogguf","autoollama","gguf","inference","llama","llm","llm-inference","lora","mergelora","mistral","ollama","openai"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/monk1337.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-30T06:24:55.000Z","updated_at":"2025-03-26T07:32:15.000Z","dependencies_parsed_at":"2024-05-16T02:25:03.748Z","dependency_job_id":null,"html_url":"https://github.com/monk1337/auto-ollama","commit_stats":{"total_commits":59,"total_committers":4,"mean_commits":14.75,"dds":0.0847457627118644,"last_synced_commit":"a5a86a8dd6f634ce511265c470037db5b4d84b3b"},"previous_names":["monk1337/auto-ollama"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monk1337%2Fauto-ollama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monk1337%2Fauto-ollama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monk1337%2Fauto-ollama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/monk1337%2Fauto-ollama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/monk1337","download_url":"https://codeload.github.com/monk1337/auto-ollama/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247814898,"owners_count":21000664,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autogguf","autoollama","gguf","inference","llama","llm","llm-inference","lora","mergelora","mistral","ollama","openai"],"created_at":"2024-11-24T02:20:23.606Z","updated_at":"2025-04-10T00:32:25.539Z","avatar_url":"https://github.com/monk1337.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003eAuto-Ollama \u0026 Auto-GGUF ⚡️ \u003c/h1\u003e\n\u003cp\u003eInference or Quantize Large Language Models (LLMs) Locally with a Single Command\u003c/p\u003e\n\u003c/div\u003e\n\n## Overview\n\nAuto-Ollama is a toolkit designed to simplify the inference or Quantization of Large Language Models (LLMs) directly on your local environment. With an emphasis on ease of use and flexibility, Auto-Ollama supports both direct usage and conversion of models into an efficient format for local deployment.\n\nFor quantization, check out the new package called [Auto-QuantLLM ⚡️](https://github.com/monk1337/AutoQuantLLM/tree/main). It's currently under development, but it aims to provide a streamlined and user-friendly approach to quantizing large language models (LLMs) with different Quantization methods.\n\n## Getting Started\n\n### Installation\n\nClone the repository to get started with Auto-Ollama:\n\n```bash\ngit clone https://github.com/monk1337/auto-ollama.git\ncd auto-ollama\n```\n\n### Quick Tour\nRunning Auto-Ollama\nUse the autollama.sh script to quickly inference LLMs. This script requires the model name and the quantized file name as arguments.\n\n```bash\n# Deploy Large Language Models (LLMs) locally with Auto-Ollama\n# Usage:\n# ./scripts/autollama.sh -m \u003cmodel path\u003e -g \u003cgguf file name\u003e\n\n\n# Example command:\n./scripts/autollama.sh -m TheBloke/MistralLite-7B-GGUF -g mistrallite.Q4_K_M.gguf\n```\n\n\n\n### Handling Non-Quantized Models with AutoGGUF\nIf your desired model is not available in a quantized format suitable for local deployment, Auto-Ollama offers the AutoGGUF utility. This tool can convert any Hugging Face model into the GGUF format and upload it to the Hugging Face model hub.\n\n```bash\n# Convert your Hugging Face model to GGUF format for local deployment\n# Usage:\n# ./scripts/autogguf.sh -m \u003cMODEL_ID\u003e [-u USERNAME] [-t TOKEN] [-q QUANTIZATION_METHODS]\n\n# Example command:\n./scripts/autogguf.sh -m unsloth/gemma-2b\n```\n\n### More Options\n```bash\n# if want to upload the gguf model to hub after the conversion, provide the user and token\n# Example command:\n./scripts/autogguf.sh -m unsloth/gemma-2b -u user_name -t hf_token\n\n\n#if wants to provide QUANTIZATION_METHODS\n# Example command:\n./scripts/autogguf.sh -m unsloth/gemma-2b -u user_name -t hf_token -q \"q4_k_m,q5_k_m\"\n```\n\n## Quantization Recommendations\n- **Use Q5_K_M** for the best performance-resource balance.\n- **Q4_K_M** is a good choice if you need to save memory.\n- **K_M** versions generally perform better than K_S.\n\n### Support and Contributions\nFor issues, suggestions, or contributions, please open an issue or pull request in the GitHub repository. We welcome contributions from the community to make Auto-Ollama even better!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonk1337%2Fauto-ollama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmonk1337%2Fauto-ollama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmonk1337%2Fauto-ollama/lists"}