{"id":50725810,"url":"https://github.com/deepsweet/mlx-eval","last_synced_at":"2026-06-10T04:01:10.612Z","repository":{"id":353636445,"uuid":"1219256143","full_name":"deepsweet/mlx-eval","owner":"deepsweet","description":"Utilities to evaluate MLX quantizations","archived":false,"fork":false,"pushed_at":"2026-06-09T10:34:46.000Z","size":880,"stargazers_count":13,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2026-06-09T12:17:14.481Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/deepsweet.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-23T17:33:16.000Z","updated_at":"2026-06-09T10:41:03.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/deepsweet/mlx-eval","commit_stats":null,"previous_names":["deepsweet/mlx-kld","deepsweet/mlx-eval"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/deepsweet/mlx-eval","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepsweet%2Fmlx-eval","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepsweet%2Fmlx-eval/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepsweet%2Fmlx-eval/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepsweet%2Fmlx-eval/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/deepsweet","download_url":"https://codeload.github.com/deepsweet/mlx-eval/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/deepsweet%2Fmlx-eval/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34136112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-10T02:00:07.152Z","response_time":89,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-06-10T04:01:09.778Z","updated_at":"2026-06-10T04:01:10.604Z","avatar_url":"https://github.com/deepsweet.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mlx-eval\n\nUtilities to evaluate MLX quantizations.\n\nSee [detailed results](./results) for more information:\n\n![Qwen3.6-35B-A3B KLD/RAM chart](./results/Qwen3.6-35B-A3B.svg)\n\n## Usage\n\n```sh\n# clone the repo\ngit clone git@github.com:deepsweet/mlx-eval.git\ncd mlx-eval/\n\n# install dependencies\nuv sync\n\n# prepare an original reference MLX model \n# fof example, text-only using mlx-lm, or multimodal using mlx-vlm:\nuv tool install mlx-vlm --with torchvision\nmlx_vlm.convert \\\n  --hf-path Qwen/Qwen3.6-35B-A3B \\\n  --mlx-path /path/to/Qwen3.6-35B-A3B-MLX\n\n# prepare a quantized target MLX model\n# for example:\nmlx_vlm.convert \\\n  --hf-path Qwen/Qwen3.6-35B-A3B \\\n  --mlx-path /path/to/Qwen3.6-35B-A3B-MLX-Q4 \\\n  --quantize \\\n  --q-bits 4\n\n# compute and store the reference model data into outputs/\n# mlx_eval.reference \u003creference_model_path\u003e \u003cwindow_count\u003e \u003cmax_tokens\u003e\nuv run mlx_eval.reference /path/to/Qwen3.6-35B-A3B-MLX 16 8192\n\n# and compare the target quantized model against it\n# mlx_eval.compare \u003ctarget_model_path\u003e \u003cwindow_count\u003e\nuv run mlx_eval.compare /path/to/Qwen3.6-35B-A3B-MLX-Q4 16\n```\n\n## Generate chart\n\n```sh\nuv run results/\u003cmodel_name\u003e.py\n```\n\n## Lint and test\n\n```sh\nuv sync --group dev\nuv run ruff check .\nuv run pytest .\n```\n\n## License\n\nMIT.\n\nThe evaluation prompt is derived from [Aes Sedai](https://huggingface.co/AesSedai)'s\n`combined_all_micro.txt`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepsweet%2Fmlx-eval","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeepsweet%2Fmlx-eval","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeepsweet%2Fmlx-eval/lists"}