{"id":31959532,"url":"https://github.com/huggingface/huggingface-llama-recipes","last_synced_at":"2025-10-14T15:33:11.976Z","repository":{"id":249840894,"uuid":"832257345","full_name":"huggingface/huggingface-llama-recipes","owner":"huggingface","description":null,"archived":false,"fork":false,"pushed_at":"2025-04-30T08:53:41.000Z","size":15446,"stargazers_count":682,"open_issues_count":24,"forks_count":80,"subscribers_count":30,"default_branch":"main","last_synced_at":"2025-09-29T00:56:42.146Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":".github/README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-22T16:32:07.000Z","updated_at":"2025-09-20T04:47:21.000Z","dependencies_parsed_at":"2024-10-17T10:28:26.668Z","dependency_job_id":null,"html_url":"https://github.com/huggingface/huggingface-llama-recipes","commit_stats":null,"previous_names":["huggingface/huggingface-llama-recipes"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/huggingface-llama-recipes","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fhuggingface-llama-recipes","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fhuggingface-llama-recipes/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fhuggingface-llama-recipes/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fhuggingface-llama-recipes/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/huggingface-llama-recipes/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Fhuggingface-llama-recipes/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019320,"owners_count":26086711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T15:32:19.008Z","updated_at":"2025-10-14T15:33:11.968Z","avatar_url":"https://github.com/huggingface.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hugging Face Llama Recipes\n\n![thumbnail for repository](../assets/hf-llama-recepies.png)\n\n🤗🦙Welcome! This repository contains *minimal* recipes to get started quickly\nwith **Llama 3.x** models, including **Llama 3.1**, **Llama 3.2**, and **Llama 3.3**.\n\n* To get an overview of Llama 3.1, please visit the [Hugging Face announcement blog post (3.1)](https://huggingface.co/blog/llama31).\n* To get an overview of Llama 3.2, please visit the [Hugging Face announcement blog post (3.2)](https://huggingface.co/blog/llama32).\n* For more advanced end-to-end use cases with open ML, please visit the [Open Source AI Cookbook](https://huggingface.co/learn/cookbook/index).\n\nThis repository is WIP so that you might see considerable changes in the coming days.\n\n\u003e [!NOTE]\n\u003e To use Llama 3.x, you need to accept the license and request permission\nto access the models. Please visit [the Hugging Face repos](https://huggingface.co/meta-llama)\nand submit your request. You only need to do this once per collection; you'll get access to\nall the repos in the collection if your request is approved.\n\n## Getting Started\n\nThe easiest way to quickly run a Llama 🦙 on your machine would be with the\n🤗 `transformers` repository. Make sure you have the latest release installed.\n\n```shell\n$ pip install -U transformers\n```\n\nLet us conversate with an instruction tuned model.\n\n```python\nimport torch\nfrom transformers import pipeline\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nllama_31 = \"meta-llama/Llama-3.1-8B-Instruct\" # \u003c-- llama 3.1\nllama_32 = \"meta-llama/Llama-3.2-3B-Instruct\" # \u003c-- llama 3.2\n\nprompt = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant, that responds as a pirate.\"},\n    {\"role\": \"user\", \"content\": \"What's Deep Learning?\"},\n]\n\ngenerator = pipeline(model=llama_32, device=device, torch_dtype=torch.bfloat16)\ngeneration = generator(\n    prompt,\n    do_sample=False,\n    temperature=1.0,\n    top_p=1,\n    max_new_tokens=50\n)\n\nprint(f\"Generation: {generation[0]['generated_text']}\")\n\n# Generation:\n# [\n#   {'role': 'system', 'content': 'You are a helpful assistant, that responds as a pirate.'},\n#   {'role': 'user', 'content': \"What's Deep Learning?\"},\n#   {'role': 'assistant', 'content': \"Yer lookin' fer a treasure trove o'\n#             knowledge on Deep Learnin', eh? Alright then, listen close and\n#             I'll tell ye about it.\\n\\nDeep Learnin' be a type o' machine\n#             learnin' that uses neural networks\"}\n# ]\n```\n\n## Local Inference\n\nWould you like to run inference of the Llama models locally?\nSo do we! The memory requirements depend on the model size and the\nprecision of the weights. Here's a table showing the approximate\nmemory needed for different configurations:\n\n| Model Size | Llama Variant | BF16/FP16 | FP8 | INT4(AWQ/GPTQ/bnb) |\n| :--: | :--: | :--: | :--: | :--: |\n| 1B | 3.2 | 2.5 GB | 1.25GB | 0.75GB |\n| 3B | 3.2 |6.5 GB | 3.2GB | 1.75GB |\n| 8B | 3.1 |16 GB | 8GB | 4GB |\n| 70B | 3.1 and 3.3 | 140 GB | 70GB | 35GB |\n|405B | 3.1 |810 GB | 405GB | 204GB |\n\n\n\u003e [!NOTE]\n\u003e These are estimated values and may vary based on specific\nimplementation details and optimizations.\n\nWorking with the capable Llama 3.1 8B models:\n\n* [Run Llama 3.1 8B in 4-bits with bitsandbytes](../local_inference/4bit_bnb.ipynb)\n* [Run Llama 3.1 8B in 8-bits with bitsandbytes](../local_inference/8bit_bnb.ipynb)\n* [Run Llama 3.1 8B with AWQ \u0026 fused ops](../local_inference/awq.ipynb)\n\nWorking on the 🐘 big Llama 3.1 405B model:\n\n* [Run Llama 3.1 405B FP8](../local_inference/fp8-405B.ipynb)\n* [Run Llama 3.1 405B quantized to INT4 with AWQ](../local_inference/awq_generation.py)\n* [Run Llama 3.1 405B quantized to INT4 with GPTQ](../local_inference/gptq_generation.py)\n\n## Model Fine Tuning:\n\nIt is often not enough to run inference on the model.\nMany times, you need to fine-tune the model on some \ncustom dataset. Here are some scripts showing \nhow to fine-tune the models.\n\nFine tune models on your custom dataset:\n* [Fine tune Llama 3.2 Vision on a custom dataset](../fine_tune/Llama-Vision%20FT.ipynb)\n* [Supervised Fine Tuning on Llama 3.2 Vision with TRL](../fine_tune/sft_vlm.py)\n* [How to fine-tune Llama 3.1 8B on consumer GPU with PEFT and QLoRA with bitsandbytes](../fine_tune/peft_finetuning.py)\n* [Execute a distributed fine tuning job for the Llama 3.1 405B model on a SLURM-managed computing cluster](../fine_tune/qlora_405B.slurm)\n\n## Assisted Decoding Techniques\n\nDo you want to use the smaller Llama 3.2 models to speed up text generation\nfor bigger models? These notebooks showcase assisted decoding (speculative decoding), which gives you upto 2x speedups for text generation on Llama 3.1 70B (with greedy decoding).\n\n* [Run assisted decoding with 🐘 Llama 3.1 70B and 🤏 Llama 3.2 3B](../assisted_decoding/assisted_decoding_70B_3B.ipynb)\n* [Run assisted decoding with Llama 3.1 8B and Llama 3.2 1B](../assisted_decoding/assisted_decoding_8B_1B.ipynb)\n* [Assisted Decoding with 405B model](../assisted_decoding/assisted_decoding.py)\n\n## Performance Optimization\n\nLet us optimize performace shall we?\n\n* [Accelerate your inference using torch.compile](../performance_optimization/torch_compile.py)\n* [Accelerate your inference using torch.compile and 4-bit quantization with torchao](../performance_optimization/torch_compile_with_torchao.ipynb)\n* [Quantize KV Cache to lower memory requirements](../performance_optimization/quantized_cache.py)\n* [How to reuse prompts with dynamic caching](../performance_optimization/prompt_reuse.py)\n* [How to setup distributed training utilizing DeepSpeed with mixed-precision and Zero-3 optimization](../performance_optimization/deepspeed_zero3.yaml)\n\n## API inference\n\nAre these models too large for you to run at home? Would you like to experiment with Llama 70B? Try out the following examples!\n\n* [Use the Inference API for PRO users](../api_inference/inference-api.ipynb)\n\n## Llama Guard and Prompt Guard\n\nIn addition to the generative models, Meta released two new models: Llama Guard 3 and Prompt Guard. Prompt Guard is a small classifier that detects jailbreaks and prompt injections. Llama Guard 3 is a safeguard model that can classify LLM inputs and generations. Learn how to use them as done in the following notebooks:\n\n* [Detecting jailbreaks and prompt injection with Prompt Guard](../llama_guard/prompt_guard.ipynb)\n* [Integrating Llama Guard in LLM Workflows for detecting prompt safety](../llama_guard/llama_guard_3_1B.ipynb)\n\n## Synthetic Data Generation\nWith the ever hungry models, the need for synthetic data generation is\non the rise. Here we show you how to build your very own synthetic dataset.\n\n* [Generate synthetic data with `distilabel`](../synthetic_data_gen/synthetic-data-with-llama.ipynb)\n\n\n## Llama RAG \nSeeking an entry-level RAG pipeline? This notebook guides you through building a very simple streamlined RAG experiment using Llama and Hugging Face.\n\n* [Simple RAG Pipeline](../llama_rag/llama_rag_pipeline.ipynb)\n\n\n## Text Generation Inference (TGI) \u0026 API Inference with Llama Models\nText Generation Inference (TGI) framework enables efficient and  scalable deployment of Llama models. In this notebook we'll learn how to integrate TGI for fast text generation and to consume already deployed Llama models via  the Inference API:\n\n* [Text Generation Inference (TGI) with Llama Models](../llama_tgi_api_inference/tgi_api_inference_recipe.ipynb) \n\n## Chatbot Demo with Llama Models \nWould you like to build a chatbot with Llama models? Here's a simple example to get you started.\n\n* [Chatbot with Llama Models](../gradio_demos/chatbot_demo.ipynb)\n\n## Tool Calling\n\nIn this notebook, we explore how to leverage the **tool-calling capabilities** of Llama models, also using and integrating the `chat_template` functionality for tool interactions.\n\n* [Tool calling with Llama Models](../tool_calling/tool_calling.ipynb)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fhuggingface-llama-recipes","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Fhuggingface-llama-recipes","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Fhuggingface-llama-recipes/lists"}