{"id":15432934,"url":"https://github.com/simonw/llm-mlc","last_synced_at":"2025-04-07T12:10:59.320Z","repository":{"id":187752625,"uuid":"677588287","full_name":"simonw/llm-mlc","owner":"simonw","description":"LLM plugin for running models using MLC","archived":false,"fork":false,"pushed_at":"2024-03-30T21:39:09.000Z","size":38,"stargazers_count":183,"open_issues_count":14,"forks_count":9,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-02-06T09:35:28.437Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/simonw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-12T01:37:35.000Z","updated_at":"2025-02-05T20:07:26.000Z","dependencies_parsed_at":"2023-08-12T02:58:25.288Z","dependency_job_id":"64a6fdc8-60fd-427e-ae82-c86bc5cd795e","html_url":"https://github.com/simonw/llm-mlc","commit_stats":{"total_commits":15,"total_committers":1,"mean_commits":15.0,"dds":0.0,"last_synced_commit":"b05eec9ba008e700ecc422c69e28395fa8e146fd"},"previous_names":["simonw/llm-mlc"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-mlc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-mlc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-mlc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/simonw%2Fllm-mlc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/simonw","download_url":"https://codeload.github.com/simonw/llm-mlc/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247648977,"owners_count":20972945,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T18:29:26.748Z","updated_at":"2025-04-07T12:10:59.296Z","avatar_url":"https://github.com/simonw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# llm-mlc\n\n[![PyPI](https://img.shields.io/pypi/v/llm-mlc.svg)](https://pypi.org/project/llm-mlc/)\n[![Changelog](https://img.shields.io/github/v/release/simonw/llm-mlc?include_prereleases\u0026label=changelog)](https://github.com/simonw/llm-mlc/releases)\n[![Tests](https://github.com/simonw/llm-mlc/workflows/Test/badge.svg)](https://github.com/simonw/llm-mlc/actions?query=workflow%3ATest)\n[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/simonw/llm-mlc/blob/main/LICENSE)\n\n[LLM](https://llm.datasette.io/) plugin for running models using [MLC](https://mlc.ai/mlc-llm/docs/)\n\n## Installation\n\nInstall this plugin in the same environment as `llm`.\n```bash\nllm install llm-mlc\n```\nYou need to install two dependencies manually - `mlc-chat-nightly` and `mlc-ai-nightly` - because the installation process differs from one platform to another in a way that is not yet automated.\n\nThe steps for this are [described in detail on the mlc.ai/package](https://mlc.ai/package/) site.\n\nIf you are on an Apple Silicon M1/M2 Mac you can run this command:\n```bash\nllm mlc pip install --pre --force-reinstall \\\n  mlc-ai-nightly \\\n  mlc-chat-nightly \\\n  -f https://mlc.ai/wheels\n```\nThe `llm mlc pip` command here ensures that `pip` will run in the same virtual environment as `llm` itself.\n\nFor other systems, [follow the instructions here](https://mlc.ai/package/).\n\nFinally, run the `llm mlc setup` command to complete the installation:\n```bash\nllm mlc setup\n```\nThis will setup `git lfs` and use it to download some extra dependencies:\n```\nGit LFS is not installed. Should I run 'git lfs install' for you?\nInstall Git LFS? [y/N]: y\nUpdated Git hooks.\nGit LFS initialized.\nDownloading prebuilt binaries...\nCloning into '/Users/simon/Library/Application Support/io.datasette.llm/mlc/dist/prebuilt/lib'...\nremote: Enumerating objects: 221, done.\nremote: Counting objects: 100% (86/86), done.\nremote: Compressing objects: 100% (54/54), done.\nremote: Total 221 (delta 59), reused 56 (delta 32), pack-reused 135\nReceiving objects: 100% (221/221), 52.06 MiB | 9.13 MiB/s, done.\nResolving deltas: 100% (152/152), done.\nUpdating files: 100% (60/60), done.\nReady to install models in /Users/simon/Library/Application Support/io.datasette.llm/mlc\n```\n\n## Installing models\n\nAfter installation you will need to download a model using the `llm mlc download-model` command.\n\nHere's how to download and install Llama 2:\n\n```bash\nllm mlc download-model Llama-2-7b-chat --alias llama2\n```\nThis will download around 8GB of content.\n\nYou can also use `Llama-2-13b-chat` (about 15.15GB) or `Llama-2-70b-chat` (extremely big), though these files are a lot larger.\n\nThe `-a/--alias` is optional, but can be used to set a shorter alias for the model. This can then be used with `llm -m \u003calias\u003e` instead of the full name.\n\nThe `download-model` command also takes a URL to one of [the MLC repositories on Hugging Face](https://huggingface.co/mlc-ai).\n\nFor example, to install [mlc-chat-WizardLM-13B-V1](https://huggingface.co/mlc-ai/mlc-chat-WizardLM-13B-V1.2-q4f16_1):\n```bash\nllm mlc download-model https://huggingface.co/mlc-ai/mlc-chat-WizardLM-13B-V1.2-q4f16_1\n```\nYou can see a full list of models you have installed this way using:\n```bash\nllm mlc models\n```\nThis will also show the name of the model you should use to activate it, e.g.:\n```\nMlcModel: mlc-chat-Llama-2-7b-chat-hf-q4f16_1 (aliases: llama2, Llama-2-7b-chat)\n```\n## Running a prompt through a model\n\nOnce you have downloaded and added a model, you can run a prompt like this:\n```bash\nllm -m Llama-2-7b-chat 'five names for a cute pet ferret'\n```\n\n\u003e Great! Here are five cute and creative name suggestions for a pet ferret:\n\u003e\n\u003e 1. Ferbie - a playful and affectionate name for a friendly and outgoing ferret.\n\u003e 2. Mr. Whiskers - a suave and sophisticated name for a well-groomed and dignified ferret.\n\u003e 3. Luna - a celestial and dreamy name for a curious and adventurous ferret.\n\u003e 4. Felix - a cheerful and energetic name for a lively and mischievous ferret.\n\u003e 5. Sprinkles - a fun and playful name for a happy and energetic ferret with a sprinkle of mischief.\n\u003e\n\u003e Remember, the most important thing is to choose a name that you and your ferret will love and enjoy!\n\nAnd to send a follow-up prompt to continue the current conversation, use `-c`:\n\n```bash\nllm -c 'two more'\n```\n\n\u003e Of course! Here are two more cute name ideas for a pet ferret:\n\u003e\n\u003e 1. Digger - a fun and playful name that suits a pet that loves to dig and burrow, and is also a nod to the ferret's natural instincts as a burrower.\n\u003e 2. Gizmo - a fun and quirky name that suits a pet with a curious and mischievous personality, and is also a nod to the ferret's playful and inventive nature.\n\n## Model options\n\nThese options are available for all models. They mostly take a floating point value between 0.0 and 1.0.\n\n- `-o temperature`: A higher temperature encourages more diverse outputs, while a lower temperature produces more deterministic outputs.\n- `-o top_p`: At each step, we select tokens from the minimal set that has a cumulative probability exceeding this value.\n- `-o repetition_penalty`: Controls the likelihood of the model generating repeated texts.\n- `-o max_gen_len`: Takes an integer, which controls the maximum length of the generated text.\n\nUse them like this:\n\n```bash\nllm -m Llama-2-7b-chat \\\n  -o temperature 0.5 \\\n  -o top_p 0.9 \\\n  -o repetition_penalty 0.9 \\\n  -o max_gen_len 100 \\\n  'five names for a cute pet ferret'\n```\nThe [MLC documentation](https://mlc.ai/mlc-llm/docs/get_started/mlc_chat_config.html) has more details on these options.\n\n## Development\n\nTo set up this plugin locally, first checkout the code. Then create a new virtual environment:\n```bash\ncd llm-mlc\npython3 -m venv venv\nsource venv/bin/activate\n```\nNow install the dependencies and test dependencies:\n```bash\npip install -e '.[test]'\n```\nTo run the tests:\n```bash\npytest\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-mlc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsimonw%2Fllm-mlc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsimonw%2Fllm-mlc/lists"}