https://github.com/biosfood/intel-llm-guide
A guide on how to run LLMs on intel CPUs
https://github.com/biosfood/intel-llm-guide
guide intel llm llm-inference llm-serving machine-learning setup setup-development-environment tutorial
Last synced: 10 months ago
JSON representation
A guide on how to run LLMs on intel CPUs
- Host: GitHub
- URL: https://github.com/biosfood/intel-llm-guide
- Owner: biosfood
- License: gpl-3.0
- Created: 2024-01-15T21:24:50.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2024-01-23T20:00:19.000Z (about 2 years ago)
- Last Synced: 2025-02-14T20:40:31.403Z (about 1 year ago)
- Topics: guide, intel, llm, llm-inference, llm-serving, machine-learning, setup, setup-development-environment, tutorial
- Language: Python
- Homepage:
- Size: 20.5 KB
- Stars: 2
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# intel-llm-guide
A guide on how to run large language models on intel CPUs with limited resources on a linux platform.
Please note that this guide is __still in development__, so some inaccuracies are expected.
## Prerequesites
Im am assuming you have an intel-CPU with an integrated graphics chip. I have tested and developed this guide using an `11th Gen Intel i7-1165G7` and 16 GB of RAM, but it should work with many more models. You might also want to check out the [intel extension for pytorch system requirements](https://github.com/intel/intel-extension-for-transformers/blob/main/docs/installation.md#system-requirements)
You should also have the `python3` package installed (`sudo apt install python` or `sudo pacman -S python`).
You will also need the `git` and `git-lfs` packages.
### Virtual environment
First of all, create a new virtual python environment. This should probably be located in the `/opt` folder. My AI development environment path for example is `/opt/python-envs/AI`. After deciding on your environment path, run:
```bash
python -m venv
```
To ativate this enironment, run `source /bin/acitvate`. Because the `/opt` folder is normally not owned by the user, you might need to run `sudo chown $USER -R` to give your user permission to use it.
To check if this step worked, check which file the python command points to now using `which python` and you should get a response of the form `/bin/python`.
You will need to do this every time when using the environment, so maybe consider adding this line to your `.bashrc` but be careful as this might break other python dependencies.
## Dependencies
To install the `intel extensions for pytorch` and all other needed packages, run
```bash
pip install intel-extension-for-transformers torch tokenizers sentencepiece protobuf accelerate
```
## Huggingface models
As of Janurary 2023, there are some problems with the intel-extension-for-transformers module, making "normal" usage not possible.
First of all, create a new directory to store all of your models with `mkdir /opt/models && sudo chown $USER /opt/models`.
To correctly use a language model from the [huggingface](https://huggingface.co/) server, you first have to create an account and add your SSH key in the options menu. Then, use the clone script like this:
```bash
./clone_model.sh
```
After this, you can start using the model:
```python
from transformers import AutoTokenizer, TextStreamer
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
import torch
model = AutoModelForCausalLM.from_pretrained("/opt/llms/" + model_id)
```