https://github.com/deepsweet/mlx-eval
Utilities to evaluate MLX quantizations
https://github.com/deepsweet/mlx-eval
Last synced: about 1 month ago
JSON representation
Utilities to evaluate MLX quantizations
- Host: GitHub
- URL: https://github.com/deepsweet/mlx-eval
- Owner: deepsweet
- License: mit
- Created: 2026-04-23T17:33:16.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2026-06-09T10:34:46.000Z (about 1 month ago)
- Last Synced: 2026-06-09T12:17:14.481Z (about 1 month ago)
- Language: Python
- Homepage:
- Size: 859 KB
- Stars: 13
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE.md
Awesome Lists containing this project
README
# mlx-eval
Utilities to evaluate MLX quantizations.
See [detailed results](./results) for more information:
## Usage
```sh
# clone the repo
git clone git@github.com:deepsweet/mlx-eval.git
cd mlx-eval/
# install dependencies
uv sync
# prepare an original reference MLX model
# fof example, text-only using mlx-lm, or multimodal using mlx-vlm:
uv tool install mlx-vlm --with torchvision
mlx_vlm.convert \
--hf-path Qwen/Qwen3.6-35B-A3B \
--mlx-path /path/to/Qwen3.6-35B-A3B-MLX
# prepare a quantized target MLX model
# for example:
mlx_vlm.convert \
--hf-path Qwen/Qwen3.6-35B-A3B \
--mlx-path /path/to/Qwen3.6-35B-A3B-MLX-Q4 \
--quantize \
--q-bits 4
# compute and store the reference model data into outputs/
# mlx_eval.reference
uv run mlx_eval.reference /path/to/Qwen3.6-35B-A3B-MLX 16 8192
# and compare the target quantized model against it
# mlx_eval.compare
uv run mlx_eval.compare /path/to/Qwen3.6-35B-A3B-MLX-Q4 16
```
## Generate chart
```sh
uv run results/.py
```
## Lint and test
```sh
uv sync --group dev
uv run ruff check .
uv run pytest .
```
## License
MIT.
The evaluation prompt is derived from [Aes Sedai](https://huggingface.co/AesSedai)'s
`combined_all_micro.txt`.