Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/Highlyhotgames/fast_txtgen_7B

Installation Script for LLaMa 7B 4bit 128g on WSL
https://github.com/Highlyhotgames/fast_txtgen_7B

Last synced: about 2 months ago
JSON representation

Installation Script for LLaMa 7B 4bit 128g on WSL

Host: GitHub
URL: https://github.com/Highlyhotgames/fast_txtgen_7B
Owner: Highlyhotgames
License: agpl-3.0
Created: 2023-04-05T17:29:01.000Z (over 1 year ago)
Default Branch: main
Last Pushed: 2024-04-04T08:36:07.000Z (6 months ago)
Last Synced: 2024-06-28T13:33:45.560Z (3 months ago)
Language: Shell
Size: 213 KB
Stars: 29
Watchers: 1
Forks: 5
Open Issues: 2
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

## Deprecated, this project's functionality is now obsolete since oobabooga's text-generation-webui now has an easy installer. This repo is just for educational purposes.

# Installation Script for LLaMa 7B 4bit 128g on WSL

This is a tutorial on how to install LLaMa on your Windows machine using WSL (Windows Subsystem for Linux).
Although with some tweaks you may get this to work properly on another hardware or on multi-GPU setups,
this tutorial is specifically designed to work with Nvidia graphics cards - and I only cover a Single-GPU configuration.
I'll be using my RTX2060 with 6GB to load Llama model 7B with 4-bit quantization,
but you can try this tutorial with other models that fit your Nvidia hardware.

----------------------------------------------------------------------------------

These instructions are delivered for educational purposes only, and it assumes that you have a fresh install of
windows 10/11 latest version (and it is obvious that you can try it without having to reinstall Windows).

While we will need to restart the Ubuntu terminal a few times, I've done my best to simplify the instructions
to minimize the number of steps. This will help you explore these AI models as fast as possible.

----------------------------------------------------------------------------------
----------------------------------------------------------------------------------

Software requirements:
----------------------------------------------------------------------------------

Open windows update (right-click on start menu; settings; update & security), click on advanced options and
enable “receive updates for other Microsoft products when you update Windows”, then
check for updates and restart your PC if required.

Go to nvidia website (or through your Geforce Experience app if u have installed) and install latest version
of your nvidia driver (make sure it is the “Game ready” version), restart again if required.

----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
----------------------------------------------------------------------------------

Instructions:
----------------------------------------------------------------------------------

Click on start menu; type cmd and click on “run as administrator”

wsl --install

It will ask to restart PC if this is the first WSL installation on this machine

After reboot WSL will continue installation...

While it does, click on start menu; type cmd and open

notepad .wslconfig

Click on yes to create a new file; copy and paste the content below:

[wsl2]
memory=6GB
processors=2
swap=20GB
localhostforwarding=true

Close notepad, click on save; close prompt;

When WSL installation ends, enter a username and a password

clear && git clone https://github.com/Highlyhotgames/fast_txtgen_7B.git > /dev/null 2>&1 && cd fast_txtgen_7B && chmod +x requirements && ./requirements

Enter password and wait for message to close terminal; close

To ensure WSL2 is activated and to update:

Click on Start menu; type cmd and run as Administrator

wsl --set-version Ubuntu 2 & wsl --update

Without closing prompt, open windows update and check for updates - it will get an update for WSL;

After update, go back to Admin prompt and run this line:

cd lxss\lib & del libcuda.so & del libcuda.so.1 & wsl -e /bin/bash -c "ln -s libcuda.so.1.1 libcuda.so.1 && ln -s libcuda.so.1.1 libcuda.so" & wsl --shutdown & exit

Open Ubuntu:

./install

When "checking CUDA Installation" it will display a message "release 11.7" in cyan color.

After installation you may run server:

./run

Default URL:

—> http://127.0.0.1:7860

To start server, open Ubuntu:

conda activate webui && cd webui/text-generation-webui

Run:

python server.py --wbits 4 --model llama-7b-4bit-128g --groupsize 128 --chat

To remove completely (including WSL), open cmd as Admin:

wsl --shutdown & wsl --unregister Ubuntu