https://github.com/evilfreelancer/llm-msnp-tests
https://github.com/evilfreelancer/llm-msnp-tests
Last synced: 10 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/evilfreelancer/llm-msnp-tests
- Owner: EvilFreelancer
- Created: 2024-09-16T22:46:09.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-16T09:31:56.000Z (about 1 year ago)
- Last Synced: 2025-02-09T15:15:36.825Z (11 months ago)
- Language: Jupyter Notebook
- Size: 157 KB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# LLM MSNP Tests
A small and collection of results, if checks speed of GGUF model on different combination of GPUs and CPUs.
Inspired due to conversation in Telegram chat.
## MSNP Leaderboards
https://huggingface.co/spaces/evilfreelancer/msnp-leaderboard
## How to test
First you need to install [ollama](https://ollama.com/download) to server where you will make tests.
Download models:
```shell
ollama pull llama3.1:8b-instruct-q4_0
ollama pull llama3.1:70b-instruct-q4_0
```
> Please use models with __quantization__ in name
Then you need to create Python Virtual Environment, then chroot to it:
```shell
mkdir msnp-tests
cd msnp-tests
python3 -m venv venv
source venv/bin/activate
```
Then download requirements and tests.py files:
```shell
wget https://raw.githubusercontent.com/EvilFreelancer/llm-msnp-tests/refs/heads/main/requirements.txt
wget https://raw.githubusercontent.com/EvilFreelancer/llm-msnp-tests/refs/heads/main/test.py
```
Install dependencies:
```shell
pip install -r requirements.txt
```
And run test:
```shell
python3 test.py
```
In result will be something like this:
