https://github.com/argmaxinc/mlx-bench
Benchmark MLX Performance across commits
https://github.com/argmaxinc/mlx-bench
Last synced: about 1 year ago
JSON representation
Benchmark MLX Performance across commits
- Host: GitHub
- URL: https://github.com/argmaxinc/mlx-bench
- Owner: argmaxinc
- Created: 2024-02-25T05:52:59.000Z (over 2 years ago)
- Default Branch: main
- Last Pushed: 2024-03-06T05:45:48.000Z (over 2 years ago)
- Last Synced: 2025-04-06T10:47:18.295Z (over 1 year ago)
- Language: Python
- Size: 13.7 KB
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# mlx-bench
Benchmark MLX Performance across forks and commits
## Installation
```bash
pip install -r requirements.txt
git submodule init && git submodule update
```
## Usage
Example command to test [ml-explore/mlx#735](https://github.com/ml-explore/mlx/pull/735) for Mistral 7b:
```bash
python bench_mistral.py \
--repo-a ml-explore/mlx --commit-a cbcf44a4caf3fb504ed29ef78091126134e197a3 \
--repo-b ml-explore/mlx --commit-b 14b4e51a7c6455a61a74d24da9f47dfeb161023f \
--output-dir external --hub-model-name mlx-community/Mistral-7B-Instruct-v0.2-4-bit \
--max-context-length 800 \
--fail-for-mismatch-before-n-tokens 800
```
This benchmark tests the two repo+commit pairs for correctness and plot the performance. These are saved in the output directory.