{"id":14964779,"url":"https://github.com/jaslatendresse/llm-demo","last_synced_at":"2025-10-25T10:30:36.422Z","repository":{"id":227089465,"uuid":"770364689","full_name":"jaslatendresse/llm-demo","owner":"jaslatendresse","description":"This repository demonstrates how to do inference using llama.cpp on a machine with minimal specs.","archived":false,"fork":false,"pushed_at":"2025-03-24T19:23:42.000Z","size":887,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-07T11:46:14.317Z","etag":null,"topics":["large-language-models","llama3","llamacpp","tutorial"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jaslatendresse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-03-11T12:29:51.000Z","updated_at":"2025-05-19T21:33:06.000Z","dependencies_parsed_at":"2025-04-30T02:38:24.476Z","dependency_job_id":"52ef04b7-9722-48f8-872c-2f9e30118a91","html_url":"https://github.com/jaslatendresse/llm-demo","commit_stats":null,"previous_names":["jaslatendresse/llm-demo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jaslatendresse/llm-demo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaslatendresse%2Fllm-demo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaslatendresse%2Fllm-demo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaslatendresse%2Fllm-demo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaslatendresse%2Fllm-demo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jaslatendresse","download_url":"https://codeload.github.com/jaslatendresse/llm-demo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jaslatendresse%2Fllm-demo/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019158,"owners_count":26086685,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["large-language-models","llama3","llamacpp","tutorial"],"created_at":"2024-09-24T13:33:46.097Z","updated_at":"2025-10-25T10:30:36.410Z","avatar_url":"https://github.com/jaslatendresse.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Local LLM Inference with `llama.cpp`\n\nThis repository is a complete tutorial and workspace to run LLMs locally using [`llama.cpp`](https://github.com/ggerganov/llama.cpp).\n\nIt walks you through:\n* Setting up your environment\n* Downloading models\n* Converting a model from HF to GGUF\n* Quantizing a model\n* Running inference via CLI or launching a local web server.\n\n## Repository Structure\n\n```\nllm-demo/\t\t\t# Local clone of llm-demo\n├── inference-cli.sh       \t# Script to run inference in CLI mode (llama-cli)\n├── inference-web.sh       \t# Script to run llama-server for web-based inference\n│\n├── models/                \t# Model storage (downloaded, converted, quantized)\n│   ├── hf_models/         \t# Hugging Face models (original)\n│   └── model.gguf         \t# Converted GGUF model (example)\n│\n├── scripts/               \t# Utility scripts for model preparation\n│   ├── download_model.py  \t# Script to download models from HuggingFace\n│   ├── convert.sh         \t# Converts HF model to GGUF using llama.cpp\n│   └── quantize.sh        \t# Quantizes the converted GGUF model using llama.cpp\n│\n├── bencharmking/\t\t# Benchmarking tool\n│\n├── .gitignore             \t# Git ignore file (to exclude models, etc.)\n├── README.md              \t# Project overview and instructions\n└── requirements.txt       \t# Python dependencies\n```\n\n## Setup your workspacce\n\n1. **Create a virtual environment**:\n```bash\nconda create -n myenv python=3.10\nconda activate myenv\n```\n\n2. **Clone llama.cpp**:\n```bash\ngit clone https://github.com/ggerganov/llama.cpp.git\ncd llama.cpp\ncmake -B build \u0026\u0026 cmake --build build --config Release  # Build llama.cpp\npip install -r requirements.txt\n```\n\n3. **Clone this repository**:\n```bash\ngit clone https://github.com/jaslatendresse/llm-demo.git\ncd llm-demo\nmkdir models\nmkdir models/hf_models\npip install -r requirements.txt\n```\n\n4. **Create a [HuggingFace token](https://huggingface.co/security-checkup?next=%2Fsettings%2Ftokens) with write permission and copy to clipboard.**\n\n5. **Follow the instructions after running**:\n```bash\nhuggingface-cli login\n```\n\n## Model Preparation\n\nYou can skip this step and directly download quantized models in GGUF format. Otherwise, this section walks you through the process of converting and quantizing a model.\n\n1. **Download a model from HuggingFace**:\n\nSet the model ID in `download_model.py` and run:\n```bash\npython3 scripts/download_model.py\n```\n\nThis will download a model from HuggingFace and save it in `models/hf_models/` in HF format.\n\n2. **Convert the model to GGUF**:\n\n```bash\ncd scripts\n./convert.sh --cpp \"PATH_TO_LLAMA.CPP\" \\\n\t\t--hf \"PATH_TO_HF_MODEL\" \\\n\t\t--output \"PATH_TO_OUTPUT_MODELS\" \\\n\t\t--quant \"QUANTIZE TYPE \u003cf32, f16, q8_0, q4_0\u003e\"\n```\n\nThis will convert the model to GGUF format and save it in `models/`. In this step, you can already choose to quantize the model. The next step allows you to quantize it further.\n\nIf you get a \"permission denied\" error, run:\n```bash\nchmod +x scripts/convert.sh\n```\n\n3. **Quantize the model**:\n\nThis step allows you to quantize your model. You can skip this step if you have a very strong machine or if you have already quantized the model in the previous step.\n\nQuantization types can be found in the [llama.cpp repository](https://github.com/ggml-org/llama.cpp/tree/master/examples/quantize).\n\n```bash\ncd scripts\n./quantize.sh --cpp \"PATH_TO_LLAMA.CPP\" \\\n\t\t--input \"PATH_TO_GGUF_MODEL\" \\\n\t\t--output \"OUTPUT_PATH\" \\\n\t\t--model \"MODEL_NAME\" \\\n\t\t--quant \"QUANTIZE TYPE\"\n```\n\nThis will quantize the model and save it in `models/` or the specified output path.\n\n## Running Inference\n\nThese steps should be performed from the root of the workspace.\n\n### CLI Mode\n\n```bash\n./inference-cli.sh --cpp \"PATH/TO/LLAMA.CPP\" \\\n\t\t\t--model \"PATH/TO/MODEL.gguf\" \\\n\t\t\t--prompt \"YOUR PROMPT TEXT\" \\\n\t\t\t--temp 0.7 \\\n\t\t\t--threads 0 \\\n\t\t\t--gpu_layers 99\n```\n\n### Web Mode\n\n```bash\n./inference-web.sh --cpp \"PATH/TO/LLAMA.CPP\" \\\n\t\t\t--model \"PATH/TO/MODEL.gguf\" \\\n\t\t\t--temp 0.8 \\\n\t\t\t--threads 0 \\\n\t\t\t--gpu_layers 99 \\\n\t\t\t--port 8080\n```\n\nNavigate to `http://localhost:8080` to access the web interface.\n\n## Bencharking your models\n\nThis repository includes a workspace to benchmark your models on tasks like completion, generation, etc. The benchmarking tool is located in the `benchmarking/` folder.\n\nThe tasks are defined within [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/).\n\nTo know more about how to run this tool, see the [README](https://github.com/jaslatendresse/llm-demo/tree/main/benchmarking/README.md) in the `benchmarking/` folder.\n\n## References\n* [llama.cpp](https://github.com/ggml-org/llama.cpp/)\n* [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness/)\n\n## Other Useful Links\n* [HuggingFace](https://huggingface.co/)\n* [EleutherAI](https://www.eleuther.ai/)\n* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)\n    * Python bindings for llama.cpp (essentially allows you to pip install llama.cpp and use it in Python)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaslatendresse%2Fllm-demo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjaslatendresse%2Fllm-demo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjaslatendresse%2Fllm-demo/lists"}