https://github.com/vsx-d/localai
This guide helps you set up and run Large Language Models (LLMs) locally on your machine. Using WSL , Docker and Linux
https://github.com/vsx-d/localai
ai cli docker-container linux llm
Last synced: about 2 months ago
JSON representation
This guide helps you set up and run Large Language Models (LLMs) locally on your machine. Using WSL , Docker and Linux
- Host: GitHub
- URL: https://github.com/vsx-d/localai
- Owner: vsx-d
- Created: 2025-01-01T06:24:13.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-01T13:49:06.000Z (over 1 year ago)
- Last Synced: 2025-02-23T17:13:43.094Z (over 1 year ago)
- Topics: ai, cli, docker-container, linux, llm
- Homepage:
- Size: 11.7 KB
- Stars: 2
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Run Large Language Models locally in WSL or natively in Linux with Docker 🚀
### Introduction
This wiki provides step-by-step instructions for setting up and running Large Language Models (LLMs) locally in Windows Subsystem for Linux (WSL) or Linux natively with Docker. This guide is perfect for anyone who wants to host LLaMA, Stable Diffusion, or similar models on their local machine without relying on cloud services.
### Disclaimer⚠️
This tutorial assumes a slight knowledge of
* **Linux**.
* **WSL**
* **Docker**
* **CLI**
While the steps are straightforward, familiarity with these topics will help you troubleshoot any issues that might arise.
However, if you're new to these concepts, don't worry! Just follow the instructions step by step, and you'll be able to set everything up successfully. If any issues come up you can feel free to contact me or you can always revisit the steps to make sure everything is set up correctly.
### Prerequisites
Before you begin, ensure that you meet the following requirements:
* Windows 10 or later..
* An NVIDIA GPU (optional but highly recommended for better performance).
* At least 8 GB of RAM (more is recommended for larger models).
### Happy building! 🚀
***
## 1. Setting up WSL (Linux users can skip this step)
* Open terminal and execute the following command to install WSL.
`wsl --install`
* Now create username and password (Write down the password somewhere safe).
* After creating a user open a new session of the terminal and run.
`wsl`
* Update and Upgrade packages in ubuntu by running below commands
`sudo apt update` and `sudo apt upgrade`
## 2. Downloading and running [Ollama](https://ollama.com/download/linux)
* Paste this command to download Ollama inside WSL or Linux CLI.
`curl -fsSL https://ollama.com/install.sh | sh`
* Download LLaMa models
`ollama pull llama3.1`
`ollama pull llava`
* You can download other models too by executing
`ollama pull _followed by model name_`
* Run the LLaMA 3.1 model using Ollama, execute the following command in WSL.
`ollama run llama3.1`
### Now you can start chatting with AI right here in the Command line.
## 3. Installing [Docker Container](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository) to set-up a webUI for LLM.
* Add Docker's official GPG key:
`sudo apt-get update`
`sudo apt-get install ca-certificates curl`
`sudo install -m 0755 -d /etc/apt/keyrings`
`sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc`
`sudo chmod a+r /etc/apt/keyrings/docker.asc`
* Add the repository to Apt sources: (press enter after typing "echo \")
`echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update`
* Create a container
`sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin`
## 4.Deploying open-webUI container.
* Simply copy paste this command.
`docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main`
* After above steps go to the web browser and search
`localhost:8080`
* Sign up with your email and create a password.
# And Boom!🔥 You’ve got your own AI Model running locally.🚀🤖💻
# Tips🖊️
* To analyze images and recognize files, use the LLaVA 7B model.
* For image generation, use Stable Diffusion Automatic 1111. Stay tuned, as I’ll be writing a detailed wiki on how to set it up very soon! 🎨✨