{"id":22733845,"url":"https://github.com/samthinkgit/llama2-for-windows","last_synced_at":"2025-07-28T19:10:56.310Z","repository":{"id":190263037,"uuid":"682223003","full_name":"SamthinkGit/llama2-for-windows","owner":"SamthinkGit","description":"How to Install/Setup Llama2 Model in Windows (No WSL)","archived":false,"fork":false,"pushed_at":"2023-08-24T23:08:58.000Z","size":26,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-05T04:32:35.737Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SamthinkGit.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-23T17:56:10.000Z","updated_at":"2023-08-23T21:01:34.000Z","dependencies_parsed_at":"2025-02-05T04:42:26.886Z","dependency_job_id":null,"html_url":"https://github.com/SamthinkGit/llama2-for-windows","commit_stats":null,"previous_names":["samthinkgit/llama2-for-windows"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SamthinkGit%2Fllama2-for-windows","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SamthinkGit%2Fllama2-for-windows/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SamthinkGit%2Fllama2-for-windows/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SamthinkGit%2Fllama2-for-windows/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SamthinkGit","download_url":"https://codeload.github.com/SamthinkGit/llama2-for-windows/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246266245,"owners_count":20749754,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-10T20:16:24.140Z","updated_at":"2025-03-30T02:11:35.212Z","avatar_url":"https://github.com/SamthinkGit.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# How to Install Llama2 Locally\nThis guide will explain how to set up everything in Windows to run new Meta Llama2 70B model on your local computer without WebUI or WSL needed.\n\n## Prerequisites\n\n- CUDA capable computer (NVIDIA's graphics card).\n- NVIDIA RTX 3070 or higher recommended (I'm using this one, and works right on the edge).\n- 8GB VRAM (determined by the graphics card)\n- 12GB RAM at least\n- Some Command Line Skills \u0026 Patience\n\n## Install Cuda Toolkit\nSkip this step if already installed. This toolkit is necessary to harness the full potential of your computer. Trying to run Llama2 on CPU barely works. All the instalation guide can be found in this [CUDA Guide](https://docs.nvidia.com/cuda/cuda-installation-guide-microsoft-windows/index.html). However here is a summary of the process:\n1. Check the compatibility of your NVIDIA graphics card with CUDA.\n2. Update the drivers for your NVIDIA graphics card.\n3. Download the CUDA Toolkit installer from the [NVIDIA official website](https://developer.nvidia.com/cuda-downloads).\n4. Run the CUDA Toolkit installer.\n5. Make sure the environment variables are set (specifically PATH).\n6. Restart your computer.\n\nOnce it is installed in your computer verify the installation running `nvcc --version` in PowerShell. It should appear some info like this:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/SamthinkGit/llama2-for-windows/assets/92941012/928d9b1b-e7b8-4c72-8986-07d0f98b5943\" alt=\"NVIDIA CUDA installation\" height=100/\u003e\n\u003c/p\u003e\n\n\n## Installing Dependencies\nEnsure you have previously installed [Python](https://www.tutorialspoint.com/how-to-install-python-in-windows) and [pip](https://phoenixnap.com/kb/install-pip-windows). Then download all the necessary libraries using the terminal:\n\n```powershell\n# Base Dependencies\npip install transformers torch yaml\n```\n\nLlama2 isn't often used directly, so it is also necesary to integrate 4bit-optimization into the model. For this we must use [bitsandbytes](https://github.com/TimDettmers/bitsandbytes), however currently (v0.41.0) it has only CUDA support on Linux, so we will need to install a precompiled wheel in Windows. For this follow the next steps:\n1. Check your CUDA version using `nvcc --version`\n2. Download your wheel from [this repository](https://github.com/jllllll/bitsandbytes-windows-webui/releases/tag/wheels) (thanks to jllllll)\n   **Note**: Must use at least \u003e= 0.39.1 to work with 4bit-optimization\n4. Install the library\n\n```powershell\n# Go to your wheel directory\ncd (path-to-download)\n\n# Replace with your selected wheel\npip install bitsandbytes-0.41.1-py3-none-win_amd64.whl\n\n# Check it has been succesfully installed\npip show bitsandbytes\n\n# Check if it has been compiled with CUDA support. Some versions can fail, however it is\n# only necesary that the warning \"This version has not been compiled with CUDA\" DO NOT pop up\n# (even if it crashes some lines afterwards there is no problem)\npython -m bitsandbytes\n```\n\nFinally with bitsandbytes installed we will also add the accelerate library to optimice the model\n```powershell\npip install accelerate\n```\n\n## Obtaining Access to Llama2\n1. First, we need to create an accout into the [Hugging Face](https://huggingface.co/) page and get our **access token** to load the model in the computer. You can follow [this guide](https://huggingface.co/docs/hub/security-tokens#:~:text=To%20create%20an%20access%20token,clicking%20on%20the%20Manage%20button.) but is as simple as going to Settings \u003e Access Tokens \u003e New Token \u003e Write.\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/SamthinkGit/llama2-for-windows/assets/92941012/58371a09-407a-4323-b74d-5178a412055c\" alt=\"Hugging Face get a Key\" height=400/\u003e\n\u003c/p\u003e\n\n2. With the **same email as the used in Hugging Face** we must request access to the model to Meta in [AI.meta.com](https://ai.meta.com/resources/models-and-libraries/llama-downloads/).\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/SamthinkGit/llama2-for-windows/assets/92941012/17842fde-9157-4c89-998b-433856904c2b\" alt=\"Hugging Face get a Key\" height=300/\u003e\n\u003c/p\u003e\n\n3. Go to Hugging Face, log into your account and select one of the three Llama2 open source models. Then request access to them in [this link](https://huggingface.co/meta-llama/Llama-2-13b-hf). When your access has been granted (1-2h) you'll receive an email and also the site will update to be fully enabled. Then you can look at all the models in [HuggingFace meta-llama models](https://huggingface.co/meta-llama)\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/SamthinkGit/llama2-for-windows/assets/92941012/d7624fc0-dba4-4a05-a9c3-4145cea438d5\" alt=\"Hugging Face get a Key\" height=300/\u003e\n\u003c/p\u003e\n\n## Running Llama2\nYou now have everything you needed to run the Llama2 Model on your GPU. You can test this installation using the scripts added to this repository:\n```powershell\n# Get the repository and fill the data\ngit clone https://github.com/SamthinkGit/llama2-for-windows.git\ncd .\\llama2-for-windows\npython.exe .\\setup.py\n\n# Llama2 only-terminal mode\npython.exe .\\llama2.py\n\n# Llama2 Web based Interface\npip install zmq streamlit\npython.exe .\\llama2-web.py\n\n# Don't forget to give a star to this repo if it worked for u :D\n```\n\nIf you want to build your own code for Llama2 or more purpouses continue with the guide.\n\n## Writing your first Q\u0026A Llama2 Agent\nWe will now write our first code to make Llama2 talk and answer some questions. We wont go in much further detail since this is not a LLM course. The code adapted from [Haystack-Llama2-Guide](https://github.com/anakin87/llama2-haystack/blob/main/llama2-haystack.ipynb) from anakin87\n\n1. First, create a new `config.yaml` file where we will write some information of the model.  The `\u003cBATCH_SIZE\u003e` will determine how your GPU uses your memory. I use a value of 25 and it's fine but it can be up to 100 or more if you are fine with it. The model selected is listed in the yaml, I recommend using the 7b one for the minimum computer requisites. The 13b will need around 12GB VRAM to work fine and 24GB for the 70b model. Remember to fill `\u003cYOUR_HUGGING_FACE_TOKEN\u003e` with the key obtained few steps ago.\n```powershell\n# -\u003e powershell\nNew-Item config.yaml\nnotepad.exe config.yaml\n```\n\n```yaml\n# -\u003e config.yaml\n# Possible llama models:\n# Llama-2-7b-hf\n# Llama-2-7b-chat-hf\n# Llama-2-13b-hf\n# Llama-2-13b-chat-hf\n# Llama-2-70b-hf\n# Llama-2-70b-chat-hf\n\ngeneral:\n  logging_level: WARNING\n  pytorch_cuda_config: max_split_size_mb:\u003cBATCH_SIZE\u003e\n\nmodel:\n  token: \u003cYOUR_HUGGING_FACE_TOKEN\u003e\n  id: meta-llama/Llama-2-7b-chat-hf\n```\n\n2. Create a Python file `model.py` with your IDE or by using the notepad.exe in some dir. Then initialice the model settings written in the yaml.\n```python\nimport logging\nimport torch\nimport os\nimport yaml\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Open config file\nwith open(\"config.yaml\", 'r') as stream:\n    config = yaml.safe_load(stream)\n\n# Some logs for the errors\nlogging.basicConfig(format=\"%(levelname)s - %(name)s -  %(message)s\", level=config['general']['logging_level'])\n\n# Initializing\nos.environ['PYTORCH_CUDA_ALLOC_CONF'] = config['general']['pytorch_cuda_config']\ntorch.cuda.empty_cache()  # Clean the cache, recommended for low GPU's\n\n# Obtaining some variables\nhf_token = config['model']['token']\nmodel_id = config['model']['id']\n```\n\n3. Finally we just need to build the model and write a query\n```python\nmodel = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, use_auth_token=hf_token)\nmodel.config.pretraining_tp = 1\ntokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_token)\n\nwhile True:\n    torch.cuda.empty_cache()\n    input_text = input(\"Query: \")\n    input_ids = tokenizer(input_text, return_tensors=\"pt\").input_ids.to(\"cuda\")\n    outputs = model.generate(input_ids, max_length=100) # You can modify the length of the answer\n    print(tokenizer.decode(outputs[0]))\n```\n\n### Results\n```powershell\npython.exe .\\model.py\nLoading checkpoint shards: 100%|█████████████████████████████████████| 2/2 [00:15\u003c00:00,  7.52s/it]\nQuestion: Explain briefly how to play Dark Souls I\n[AI] \u003cs\u003e Explain briefly how to play Dark Souls I and II in multiplayer.\nHopefully this will help you get started with playing Dark Souls in multiplayer.\nDark Souls is a challenging action RPG with a unique sense of...\n```\n\n## Building a WebInterface for Llama2\nWe will now use streamlit and zmq to build a small server for our model to talk with the user. We can make it all in one script, however it is better to optimice it in two parallel executions releasing a bit the GPU from the loading of the web.\n\n1. Add new socket parameters into the `config.yaml` file\n```yaml\n# Select the a free port, in this case 12443 is fine\nsocket:\n  REQ: tcp://localhost:12443\n  REP: tcp://*:12443\n```\n\n2. Create 2 new files, `model-back.py` and `model-front.py`. In the back we will copy the content of `model.py` and modify the code to accept queries from a server. In the front we will simply request a query to the user and print the generation in the screen.\n```python\n#model-back.py\nimport zmq\n# Same imports...\n\n# Initialize all variables (step 2 model.py)...\n\n# Build the model\nmodel = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, use_auth_token=hf_token)\nmodel.config.pretraining_tp = 1\ntokenizer = AutoTokenizer.from_pretrained(model_id, use_auth_token=hf_token)\n\n# Function to generate a output from a query\ndef get_answer(input_text):\n\n    input_ids = tokenizer(input_text, return_tensors=\"pt\").input_ids.to(\"cuda\")\n    outputs = model.generate(input_ids, max_length = 100)\n    return tokenizer.decode(outputs[0])\n\n# Building Socket\ncontext = zmq.Context()\nsocket = context.socket(zmq.REP)\nprint(f\"Wating for connection...\")\nsocket.bind(config['socket']['REP'])\nprint(f\"[SUCCESS] Connection Stablished\")\n\n# New loop for receiving queries\nwhile True:\n\n    input_text = socket.recv_string()\n\n    print(f\"Generating Output...\")\n    output = get_answer(input_text)\n    print(f\"Generation Finished\")\n\n    torch.cuda.empty_cache()\n    socket.send_string(output)\n```\n\n```python\n# model-front.py\nimport streamlit as st\nimport zmq\nimport yaml\n\n# Open yaml\nwith open(\"config.yaml\", 'r') as stream:\n    config = yaml.safe_load(stream)\n\n# Build socket\ncontext = zmq.Context()\nsocket = context.socket(zmq.REQ)\nsocket.connect(config['socket']['REQ'])\n\n# Build interface\nst.title('[ Talk with Meta-Llama2 ]')\nuser_input = st.text_input(\"Input:\")\n\n# Write the query\nif st.button(\"Send\"):\n    socket.send_string(user_input)\n    st.write(socket.recv_string())\n```\n3. Now you only need to launch this 2 scripts to have Llama2 fully working!\n```powershell\npython.exe model-back.py\nLoading checkpoint shards: 100%|█████████████████████████████████████████████████| 2/2 [00:07\u003c00:00,  3.75s/it]\nWating for connection...\n[SUCCESS] Connection Stablished\n...\n```\n```powershell\n# In other terminal\nstreamlit run model-front.py\n\n  You can now view your Streamlit app in your browser.\n\n  Local URL: http://localhost:8501\n  Network URL: http://192.168.1.128:8501\n\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/SamthinkGit/llama2-for-windows/assets/92941012/4fc6dc1e-d043-4a77-a070-ec1e22da4c91\" alt=\"Hugging Face get a Key\" height=300/\u003e\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamthinkgit%2Fllama2-for-windows","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsamthinkgit%2Fllama2-for-windows","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsamthinkgit%2Fllama2-for-windows/lists"}