https://github.com/dahlitzflorian/llm-on-ppc64le-setup
https://github.com/dahlitzflorian/llm-on-ppc64le-setup
Last synced: 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/dahlitzflorian/llm-on-ppc64le-setup
- Owner: DahlitzFlorian
- License: mit
- Created: 2024-06-12T13:58:11.000Z (11 months ago)
- Default Branch: main
- Last Pushed: 2024-09-11T07:52:09.000Z (8 months ago)
- Last Synced: 2025-01-08T13:27:12.872Z (4 months ago)
- Language: Dockerfile
- Size: 10.7 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# LLM on IBM POWER Setup
- [Description](#description)
- [Usage](#usage)
- [onnxruntime](#onnxruntime)
- [Llama CPP](#llama-cpp)
## Description
Sample setup for a Large Language Model (LLM) on IBM POWER10 (ppc64le).
## Usage
Different backends are provided in subfolders to utilise different runtimes.
There is always a shell script, which can be customised and executed to set up the environment.
A `Dockerfile` is provided as an alternative.
Build arguments for the container images can be passed via `--build-arg =`.
### onnxruntime
```shell
$ export MODEL="TinyLlama/TinyLlama-1.1B-Chat-v1.0"
$ docker image build -t llm-ppc64le-onnxruntime --build-arg MODEL=${MODEL} onnxruntime/
$ docker container run --rm -itd \
--name llm-ppc64le-onnxruntime \
-p 8000:8000 \
llm-ppc64le-onnxruntime
```
Supported build arguments:
- `MODEL`: Huggingface model repository (default: `TinyLlama/TinyLlama-1.1B-Chat-v1.0`)
### Llama CPP
- [README.md](/llama.cpp/README.md)