{"id":13471735,"url":"https://github.com/mistralai/mistral-inference","last_synced_at":"2025-05-12T05:29:28.509Z","repository":{"id":196853021,"uuid":"697302510","full_name":"mistralai/mistral-inference","owner":"mistralai","description":"Official inference library for Mistral models","archived":false,"fork":false,"pushed_at":"2025-03-20T15:03:08.000Z","size":563,"stargazers_count":10211,"open_issues_count":150,"forks_count":912,"subscribers_count":128,"default_branch":"main","last_synced_at":"2025-05-11T20:09:14.627Z","etag":null,"topics":["llm","llm-inference","mistralai"],"latest_commit_sha":null,"homepage":"https://mistral.ai/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mistralai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-09-27T13:05:24.000Z","updated_at":"2025-05-11T18:22:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"8653b97e-1391-414c-8ef8-055099b23c98","html_url":"https://github.com/mistralai/mistral-inference","commit_stats":{"total_commits":109,"total_committers":24,"mean_commits":4.541666666666667,"dds":0.6422018348623852,"last_synced_commit":"de6f6464210ca51cbe75399c66125a633202618f"},"previous_names":["mistralai/mistral-src"],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mistralai%2Fmistral-inference","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mistralai%2Fmistral-inference/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mistralai%2Fmistral-inference/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mistralai%2Fmistral-inference/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mistralai","download_url":"https://codeload.github.com/mistralai/mistral-inference/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253633116,"owners_count":21939389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["llm","llm-inference","mistralai"],"created_at":"2024-07-31T16:00:48.801Z","updated_at":"2025-05-12T05:29:28.479Z","avatar_url":"https://github.com/mistralai.png","language":"Jupyter Notebook","funding_links":[],"categories":["Jupyter Notebook","Official Mistral Resources","HarmonyOS","llm","Browse The Shelves","2. Open Foundation Models"],"sub_categories":["Windows Manager","Local LLM developer tools"],"readme":"# Mistral Inference\n\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/mistralai/mistral-inference/blob/main/tutorials/getting_started.ipynb\"\u003e\n  \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/\u003e\n\u003c/a\u003e\n\n\nThis repository contains minimal code to run Mistral models.\n\nBlog 7B: [https://mistral.ai/news/announcing-mistral-7b/](https://mistral.ai/news/announcing-mistral-7b/)\\\nBlog 8x7B: [https://mistral.ai/news/mixtral-of-experts/](https://mistral.ai/news/mixtral-of-experts/)\\\nBlog 8x22B: [https://mistral.ai/news/mixtral-8x22b/](https://mistral.ai/news/mixtral-8x22b/)\\\nBlog Codestral 22B: [https://mistral.ai/news/codestral](https://mistral.ai/news/codestral/) \\\nBlog Codestral Mamba 7B: [https://mistral.ai/news/codestral-mamba/](https://mistral.ai/news/codestral-mamba/) \\\nBlog Mathstral 7B: [https://mistral.ai/news/mathstral/](https://mistral.ai/news/mathstral/) \\\nBlog Nemo: [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/) \\\nBlog Mistral Large 2: [https://mistral.ai/news/mistral-large-2407/](https://mistral.ai/news/mistral-large-2407/) \\\nBlog Pixtral 12B: [https://mistral.ai/news/pixtral-12b/](https://mistral.ai/news/pixtral-12b/)\nBlog Mistral Small 3.1: [https://mistral.ai/news/mistral-small-3-1/](https://mistral.ai/news/mistral-small-3-1/)\n\nDiscord: [https://discord.com/invite/mistralai](https://discord.com/invite/mistralai)\\\nDocumentation: [https://docs.mistral.ai/](https://docs.mistral.ai/)\\\nGuardrailing: [https://docs.mistral.ai/usage/guardrailing](https://docs.mistral.ai/usage/guardrailing)\n\n## Installation\n\nNote: You will use a GPU to install `mistral-inference`, as it currently requires `xformers` to be installed and `xformers` itself needs a GPU for installation.\n\n### PyPI\n\n```\npip install mistral-inference\n```\n\n### Local\n\n```\ncd $HOME \u0026\u0026 git clone https://github.com/mistralai/mistral-inference\ncd $HOME/mistral-inference \u0026\u0026 poetry install .\n```\n\n## Model download\n\n### Direct links\n\n| Name        | Download | md5sum |\n|-------------|-------|-------|\n| 7B Instruct | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar | `80b71fcb6416085bcb4efad86dfb4d52` |\n| 8x7B Instruct | https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar (**Updated model coming soon!**) | `8e2d3930145dc43d3084396f49d38a3f` |\n| 8x22 Instruct | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-Instruct-v0.3.tar | `471a02a6902706a2f1e44a693813855b` |\n| 7B Base | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-v0.3.tar | `0663b293810d7571dad25dae2f2a5806` |\n| 8x7B |     **Updated model coming soon!**       | - |\n| 8x22B | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-v0.3.tar | `a2fa75117174f87d1197e3a4eb50371a` |\n| Codestral 22B | https://models.mistralcdn.com/codestral-22b-v0-1/codestral-22B-v0.1.tar | `1ea95d474a1d374b1d1b20a8e0159de3` |\n| Mathstral 7B | https://models.mistralcdn.com/mathstral-7b-v0-1/mathstral-7B-v0.1.tar | `5f05443e94489c261462794b1016f10b` |\n| Codestral-Mamba 7B | https://models.mistralcdn.com/codestral-mamba-7b-v0-1/codestral-mamba-7B-v0.1.tar | `d3993e4024d1395910c55db0d11db163` |\n| Nemo Base | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-base-2407.tar | `c5d079ac4b55fc1ae35f51f0a3c0eb83` |\n| Nemo Instruct | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar | `296fbdf911cb88e6f0be74cd04827fe7` |\n| Mistral Large 2 | https://models.mistralcdn.com/mistral-large-2407/mistral-large-instruct-2407.tar | `fc602155f9e39151fba81fcaab2fa7c4` |\n\nNote:\n- **Important**:\n  - `mixtral-8x22B-Instruct-v0.3.tar` is exactly the same as [Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), only stored in `.safetensors` format\n  - `mixtral-8x22B-v0.3.tar` is the same as [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1), but has an extended vocabulary of 32768 tokens.\n  - `codestral-22B-v0.1.tar` has a custom non-commercial license, called [Mistral AI Non-Production (MNPL) License](https://mistral.ai/licenses/MNPL-0.1.md)\n  - `mistral-large-instruct-2407.tar` has a custom non-commercial license, called [Mistral AI Research (MRL) License](https://mistral.ai/licenses/MRL-0.1.md)\n- All of the listed models above support function calling. For example, Mistral 7B Base/Instruct v3 is a minor update to Mistral 7B Base/Instruct v2,  with the addition of function calling capabilities.\n- The \"coming soon\" models will include function calling as well.\n- You can download the previous versions of our models from our [docs](https://docs.mistral.ai/getting-started/open_weight_models/#downloading).\n\n### From Hugging Face Hub\n\n| Name        | ID | URL |\n|-------------|-------|-------|\n| Pixtral Large Instruct | mistralai/Pixtral-Large-Instruct-2411 | https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411 |\n| Pixtral 12B Base | mistralai/Pixtral-12B-Base-2409 | https://huggingface.co/mistralai/Pixtral-12B-Base-2409 |\n| Pixtral 12B | mistralai/Pixtral-12B-2409 | https://huggingface.co/mistralai/Pixtral-12B-2409 |\n| Mistral Small 3.1 24B Base | mistralai/Mistral-Small-3.1-24B-Base-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503\n| Mistral Small 3.1 24B Instruct | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 |\n\n\n### Usage\n\n**News!!!**: Mistral Large 2 is out. Read more about its capabilities [here](https://mistral.ai/news/mistral-large-2407/).\n\nCreate a local folder to store models\n```sh\nexport MISTRAL_MODEL=$HOME/mistral_models\nmkdir -p $MISTRAL_MODEL\n```\n\nDownload any of the above links and extract the content, *e.g.*:\n\n```sh\nexport 12B_DIR=$MISTRAL_MODEL/12B_Nemo\nwget https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar\nmkdir -p $12B_DIR\ntar -xf mistral-nemo-instruct-2407.tar -C $12B_DIR\n```\n\nor\n\n```sh\nexport M8x7B_DIR=$MISTRAL_MODEL/8x7b_instruct\nwget https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar\nmkdir -p $M8x7B_DIR\ntar -xf Mixtral-8x7B-v0.1-Instruct.tar -C $M8x7B_DIR\n```\n\nFor Hugging Face models' weights, here is an example to download [Mistral Small 3.1 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503):\n\n```python\nfrom pathlib import Path\nfrom huggingface_hub import snapshot_download\n\n\nmistral_models_path = Path.home().joinpath(\"mistral_models\")\n\nmodel_path = mistral_models_path / \"mistral-small-3.1-instruct\"\nmodel_path.mkdir(parents=True, exist_ok=True)\n\nrepo_id = \"mistralai/Mistral-Small-3.1-24B-Instruct-2503\"\n\nsnapshot_download(\n    repo_id=repo_id,\n    allow_patterns=[\"params.json\", \"consolidated.safetensors\", \"tekken.json\"],\n    local_dir=model_path,\n)\n```\n\n## Usage\n\nThe following sections give an overview of how to run the model from the Command-line interface (CLI) or directly within Python.\n\n### CLI\n\n- **Demo**\n\nTo test that a model works in your setup, you can run the `mistral-demo` command.\n*E.g.* the 12B Mistral-Nemo model can be tested on a single GPU as follows:\n\n```sh\nmistral-demo $12B_DIR\n```\n\nLarge models, such **8x7B** and **8x22B** have to be run in a multi-GPU setup.\nFor these models, you can use the following command:\n\n```sh\ntorchrun --nproc-per-node 2 --no-python mistral-demo $M8x7B_DIR\n```\n\n*Note*: Change `--nproc-per-node` to more GPUs if available.\n\n- **Chat**\n\nTo interactively chat with the models, you can make use of the `mistral-chat` command.\n\n```sh\nmistral-chat $12B_DIR --instruct --max_tokens 1024 --temperature 0.35\n```\n\nFor large models, you can make use of `torchrun`.\n\n```sh\ntorchrun --nproc-per-node 2 --no-python mistral-chat $M8x7B_DIR --instruct\n```\n\n*Note*: Change `--nproc-per-node` to more GPUs if necessary (*e.g.* for 8x22B).\n\n- **Chat with Codestral**\n\nTo use [Codestral](https://mistral.ai/news/codestral/) as a coding assistant you can run the following command using `mistral-chat`.\nMake sure `$M22B_CODESTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/Codestral-22B-v0.1`\n\n```sh\nmistral-chat $M22B_CODESTRAL --instruct --max_tokens 256\n```\n\nIf you prompt it with *\"Write me a function that computes fibonacci in Rust\"*, the model should generate something along the following lines:\n\n```sh\nSure, here's a simple implementation of a function that computes the Fibonacci sequence in Rust. This function takes an integer `n` as an argument and returns the `n`th Fibonacci number.\n\nfn fibonacci(n: u32) -\u003e u32 {\n    match n {\n        0 =\u003e 0,\n        1 =\u003e 1,\n        _ =\u003e fibonacci(n - 1) + fibonacci(n - 2),\n    }\n}\n\nfn main() {\n    let n = 10;\n    println!(\"The {}th Fibonacci number is: {}\", n, fibonacci(n));\n}\n\nThis function uses recursion to calculate the Fibonacci number. However, it's not the most efficient solution because it performs a lot of redundant calculations. A more efficient solution would use a loop to iteratively calculate the Fibonacci numbers.\n```\n\nYou can continue chatting afterwards, *e.g.* with *\"Translate it to Python\"*.\n\n- **Chat with Codestral-Mamba**\n\nTo use [Codestral-Mamba](https://mistral.ai/news/codestral-mamba/) as a coding assistant you can run the following command using `mistral-chat`.\nMake sure `$7B_CODESTRAL_MAMBA` is set to a valid path to the downloaded codestral-mamba folder, e.g. `$HOME/mistral_models/mamba-codestral-7B-v0.1`.\n\nYou then need to additionally install the following packages:\n\n```\npip install packaging mamba-ssm causal-conv1d transformers\n```\n\nbefore you can start chatting:\n\n```sh\nmistral-chat $7B_CODESTRAL_MAMBA --instruct --max_tokens 256\n```\n\n- **Chat with Mathstral**\n\nTo use [Mathstral](https://mistral.ai/news/mathstral/) as an assistant you can run the following command using `mistral-chat`.\nMake sure `$7B_MATHSTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/mathstral-7B-v0.1`\n\n```sh\nmistral-chat $7B_MATHSTRAL --instruct --max_tokens 256\n```\n\nIf you prompt it with *\"Albert likes to surf every week. Each surfing session lasts for 4 hours and costs $20 per hour. How much would Albert spend in 5 weeks?\"*, the model should answer with the correct calculation.\n\nYou can then continue chatting afterwards, *e.g.* with *\"How much would he spend in a year?\"*.\n\n- **Chat with Mistral Small 3.1 24B Instruct**\n\nTo use [Mistral Small 3.1 24B Instruct](https://mistral.ai/news/mistral-small-3-1/) as an assistant you can run the following command using `mistral-chat`.\nMake sure `$MISTRAL_SMALL_3_1_INSTRUCT` is set to a valid path to the downloaded mistral small folder, e.g. `$HOME/mistral_models/mistral-small-3.1-instruct`\n\n```sh\n    mistral-chat $MISTRAL_SMALL_3_1_INSTRUCT --instruct --max_tokens 256\n```\n\nIf you prompt it with *\"The above image presents an image of which park ? Please give the hints to identify the park.\"* with the following image URL *https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png*, the model should answer with the Yosemite park and give hints to identify it.\n\nYou can then continue chatting afterwards, *e.g.* with *\"What is the name of the lake in the image?\"*. The model should respond that it is not a lake but a river.\n\n### Python\n\n- *Instruction Following*:\n\n```py\nfrom mistral_inference.transformer import Transformer\nfrom mistral_inference.generate import generate\n\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.protocol.instruct.messages import UserMessage\nfrom mistral_common.protocol.instruct.request import ChatCompletionRequest\n\n\ntokenizer = MistralTokenizer.from_file(\"./mistral-nemo-instruct-v0.1/tekken.json\")  # change to extracted tokenizer file\nmodel = Transformer.from_folder(\"./mistral-nemo-instruct-v0.1\")  # change to extracted model dir\n\nprompt = \"How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar.\"\n\ncompletion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])\n\ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n- *Multimodal Instruction Following*:\n\n\n```python\nfrom pathlib import Path\n\nfrom huggingface_hub import snapshot_download\nfrom mistral_common.protocol.instruct.messages import ImageURLChunk, TextChunk\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_inference.generate import generate\nfrom mistral_inference.transformer import Transformer\n\nmodel_path = Path.home().joinpath(\"mistral_models\") / \"mistral-small-3.1-instruct\" # change to extracted model\n\ntokenizer = MistralTokenizer.from_file(model_path / \"tekken.json\")\nmodel = Transformer.from_folder(model_path)\n\nurl = \"https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png\"\nprompt = \"The above image presents an image of which park ? Please give the hints to identify the park.\"\n\nuser_content = [ImageURLChunk(image_url=url), TextChunk(text=prompt)]\n\ntokens, images = tokenizer.instruct_tokenizer.encode_user_content(user_content, False)\n\nout_tokens, _ = generate(\n    [tokens],\n    model,\n    images=[images],\n    max_tokens=256,\n    temperature=0.15,\n    eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id,\n)\nresult = tokenizer.decode(out_tokens[0])\n\nprint(\"Prompt:\", prompt)\nprint(\"Completion:\", result)\n```\n\n- *Function Calling*:\n\n```py\nfrom mistral_common.protocol.instruct.tool_calls import Function, Tool\n\ncompletion_request = ChatCompletionRequest(\n    tools=[\n        Tool(\n            function=Function(\n                name=\"get_current_weather\",\n                description=\"Get the current weather\",\n                parameters={\n                    \"type\": \"object\",\n                    \"properties\": {\n                        \"location\": {\n                            \"type\": \"string\",\n                            \"description\": \"The city and state, e.g. San Francisco, CA\",\n                        },\n                        \"format\": {\n                            \"type\": \"string\",\n                            \"enum\": [\"celsius\", \"fahrenheit\"],\n                            \"description\": \"The temperature unit to use. Infer this from the users location.\",\n                        },\n                    },\n                    \"required\": [\"location\", \"format\"],\n                },\n            )\n        )\n    ],\n    messages=[\n        UserMessage(content=\"What's the weather like today in Paris?\"),\n        ],\n)\n\ntokens = tokenizer.encode_chat_completion(completion_request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])\n\nprint(result)\n```\n\n- *Fill-in-the-middle (FIM)*:\n\nMake sure to have `mistral-common \u003e= 1.2.0` installed:\n```\npip install --upgrade mistral-common\n```\n\nYou can simulate a code completion in-filling as follows.\n\n```py\nfrom mistral_inference.transformer import Transformer\nfrom mistral_inference.generate import generate\nfrom mistral_common.tokens.tokenizers.mistral import MistralTokenizer\nfrom mistral_common.tokens.instruct.request import FIMRequest\n\ntokenizer = MistralTokenizer.from_model(\"codestral-22b\")\nmodel = Transformer.from_folder(\"./mistral_22b_codestral\")\n\nprefix = \"\"\"def add(\"\"\"\nsuffix = \"\"\"    return sum\"\"\"\n\nrequest = FIMRequest(prompt=prefix, suffix=suffix)\n\ntokens = tokenizer.encode_fim(request).tokens\n\nout_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)\nresult = tokenizer.decode(out_tokens[0])\n\nmiddle = result.split(suffix)[0].strip()\nprint(middle)\n```\n\n### Test\n\nTo run logits equivalence:\n```\npython -m pytest tests\n```\n\n## Deployment\n\nThe `deploy` folder contains code to build a [vLLM](https://M7B_DIR.com/vllm-project/vllm) image with the required dependencies to serve the Mistral AI model. In the image, the [transformers](https://github.com/huggingface/transformers/) library is used instead of the reference implementation. To build it:\n\n```bash\ndocker build deploy --build-arg MAX_JOBS=8\n```\n\nInstructions to run the image can be found in the [official documentation](https://docs.mistral.ai/quickstart).\n\n\n## Model platforms\n\n- Use Mistral models on [Mistral AI official API](https://console.mistral.ai/) (La Plateforme)\n- Use Mistral models via [cloud providers](https://docs.mistral.ai/deployment/cloud/overview/)\n\n## References\n\n[1]: [LoRA](https://arxiv.org/abs/2106.09685): Low-Rank Adaptation of Large Language Models, Hu et al. 2021\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmistralai%2Fmistral-inference","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmistralai%2Fmistral-inference","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmistralai%2Fmistral-inference/lists"}