{"id":19556790,"url":"https://github.com/deci-ai/inferyllm-docs","last_synced_at":"2025-04-26T22:33:14.004Z","repository":{"id":201239532,"uuid":"705618109","full_name":"Deci-AI/inferyllm-docs","owner":"Deci-AI","description":null,"archived":false,"fork":false,"pushed_at":"2024-04-15T13:37:39.000Z","size":245,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-04T18:12:27.375Z","etag":null,"topics":["dependency-graph"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Deci-AI.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-16T11:17:50.000Z","updated_at":"2025-03-30T12:15:02.000Z","dependencies_parsed_at":"2024-11-11T04:39:21.258Z","dependency_job_id":"76af1c82-ce4c-477c-8d78-fc03b27cfd24","html_url":"https://github.com/Deci-AI/inferyllm-docs","commit_stats":null,"previous_names":["deci-ai/inferyllm-docs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deci-AI%2Finferyllm-docs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deci-AI%2Finferyllm-docs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deci-AI%2Finferyllm-docs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Deci-AI%2Finferyllm-docs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Deci-AI","download_url":"https://codeload.github.com/Deci-AI/inferyllm-docs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251063667,"owners_count":21530837,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dependency-graph"],"created_at":"2024-11-11T04:39:08.527Z","updated_at":"2025-04-26T22:33:10.751Z","avatar_url":"https://github.com/Deci-AI.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# InferyLLM\r\n[![CircleCI](https://circleci.com/gh/Deci-AI/infery-llm/tree/master.svg?style=svg\u0026circle-token=70fc750615364de727b3e72c362e02e18aa8b0d0)](https://dl.circleci.com/status-badge/redirect/gh/Deci-AI/infery-llm/tree/master)\r\n\r\n## Table of Contents\r\n1. [Introduction](#introduction)\r\n2. [Installation](#installation)\r\n3. [Serving](#serving)\r\n4. [Generation](#generation)\r\n5. [Advanced Usage](#advanced-usage):\r\n   * [CLI](#cli)\r\n   * [Lowering loading time](#lowering-loading-time-the-prepare-command)\r\n   * [Benchmarking models and serving configurations](#benchmarking) \r\n\r\n## Introduction\r\n\r\nInferyLLM is a high-performance engine and server for running LLM inference.\r\n\r\n### InferyLLM is fast\r\n- Optimized CUDA kernels for MQA, GQA and MHA\r\n- Continuous batching using a paged KV cache and custom paged attention kernels \r\n- Kernel autotuning capabilities with automatic selection of the optimal kernels and parameters on the given HW\r\n- Support for extremely efficient LLMs, designed to reach SOTA throughput\r\n\r\n### InferyLLM is easy to use\r\n- Containerized OR local entrypoint servers\r\n- Simple, minimal-dependency python client\r\n- Seamless integration with 🤗 model hub\r\n\r\n### Model support\r\n   * [deci/decilm-6b](https://huggingface.co/Deci/DeciLM-6b)\r\n   * [deci/decilm-7b](https://huggingface.co/Deci/DeciLM-7b)\r\n   * [deci/decicoder-1b](https://huggingface.co/Deci/DeciCoder-1b)\r\n   * [meta-llama/llama-2-7b-hf](https://huggingface.co/meta-llama/llama-2-7b-hf)\r\n   * [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1), [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-v0.2)\r\n   * All fine-tuned variants of the above (ex: the instruct variants like [Deci/DeciLM-7B-instruct](https://huggingface.co/Deci/DeciLM-7b-instruct)).\r\n   * More models coming soon... (Mixtral, Falcon, MPT etc)\r\n\r\n### Supported GPUs\r\n* Compute capability \u003e= 8.0 (e.g. A100, A10, L4, ...). See full list [here](https://developer.nvidia.com/cuda-gpus)\u003cbr\u003e\r\n* Memory requirements depends on the model size. For example:\r\n1. DeciLM-7B - at least 24G. \r\n2. DeciCoder-1B - 16G is more than enough.\r\n  \r\n## Installation\r\n### Prerequisites\r\nBefore you begin, in order to use InferyLLM you need some details from Deci.ai:\r\n1. Artifactory credentials (referred to as ARTIFACTORY USER and ARTIFACTORY TOKEN) in order to *download* and *update* the package in the future.\r\n2. Authentication token for running the server. This will be referred to as INFERY_LLM_DECI_TOKEN below.\r\n\r\nThen, ensure you have met the following system requirements:\r\n\r\n- General requirements:\r\n  - Python \u003e= 3.11\r\n  - [CUDA ToolKit \u003e= 12.1](https://developer.nvidia.com/cuda-downloads)\r\n- For local serving:\r\n  - GLIBC \u003e= 2.31\r\n  - GCC, G++ \u003e= 11.3\r\n  - `gcc`, `g++` and `nvcc` in your `$PATH` at the time of installation\r\n- For containerized serving:\r\n  - [nvidia-container-runtime \u003e= 1.13.4](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/release-notes.html)\r\n\r\n### Installing locally\r\n\r\nInferyLLM may be used with a lean (client-only) installation or a full (client+server) installation.\r\n\r\n**Client Installation**\r\n```bash\r\n# Install InferyLLM (along with LLMClient)\r\npip install --extra-index-url=https://[ARTIFACTORY USER]:[ARTIFACTORY TOKEN]@deci.jfrog.io/artifactory/api/pypi/deciExternal/simple infery-llm\r\n```\r\n**Server Installation**\r\n```bash\r\n# Install InferyLLM Server\r\nINFERY_LLM_DECI_TOKEN=[DECI TOKEN] infery-llm install\r\n```\r\n\r\nFor a more thorough explanation, please refer to the [Advanced Usage](#advanced-usage) and check out the `install` CLI command.\r\n\r\n### Pulling the InferyLLM container\r\n\r\nTo pull an InferyLLM container from Deci's container registry:\r\n\r\n```bash\r\n# Log in to Deci's container registry\r\ndocker login --username [ARTIFACTORY USER] --password [ARTIFACTORY TOKEN] deci.jfrog.io\r\n\r\n# Pull the container. You may be specify a version instead of \"latest\" (e.g. 0.0.7)\r\ndocker pull deci.jfrog.io/deci-external-docker-local/infery-llm:latest\r\n```\r\n\r\n## Serving\r\n**Note**: remember you need an authentication token to run the server (INFERY_LLM_DECI_TOKEN)\r\n\r\nThere are two ways to serve an LLM with InferyLLM:\r\n1. Through a local entrypoint\r\n2. By using the InferyLLM container\r\n\r\nBy default, InferyLLM serves at `0.0.0.0:8080` this is configurable through passing the `--host` and `--port` flags.       \r\n\r\n\u003cdetails\u003e\r\n  \u003csummary\u003eServing with a container (suggested)\u003c/summary\u003e\r\n\r\nAssuming you have pulled the container as shown in the [Installation](#pulling-the-inferyllm-container) section,\r\nrunning the server is a simple one-liner. You can also use the container to query the serving CLI `help` for all \r\navailable serving flags and defaults:\r\n\r\n**Note**: In order to mount any local models already downloaded to the local machine add `-v \u003c/path/to/local/model\u003e:\u003c/path/to/container/destination\u003e`.\u003cbr\u003e\r\nIn the next example, the `~/.cache/deci/` directory will be mounted to `/deci/` inside the container.\r\n```bash\r\n# Serve Deci/DeciLM-6b (from HF hub) on port 9000\r\ndocker run --runtime=nvidia -e INFERY_LLM_DECI_TOKEN=[DECI TOKEN] -v ~/.cache/deci/:/deci/ -p 9000:9000 deci.jfrog.io/deci-external-docker-local/infery-llm:[VERSION TAG] --model-name Deci/DeciLM-6b --port 9000\r\n\r\n# Serve Mistral-7B. Set the Max Batch Size to 16 and limit the Maximum Sequence (input + generation) to 2048\r\ndocker run --runtime=nvidia -e INFERY_LLM_DECI_TOKEN=[DECI TOKEN] -v ~/.cache/deci/:/deci/ -p 9000:9000 deci.jfrog.io/deci-external-docker-local/infery-llm:[VERSION TAG] --model-name mistralai/Mistral-7B-Instruct-v0.2 --max-seq-length 2048 --max-batch-size 16 \r\n\r\n# See all serving CLI options and defaults\r\ndocker run --rm --runtime=nvidia -e INFERY_LLM_DECI_TOKEN=[DECI TOKEN] deci.jfrog.io/deci-external-docker-local/infery-llm:[VERSION TAG] --help\r\n```\r\n\r\nNotice that a HuggingFace token may be passed as an environment variable (using the docker `-e` flag) or as a CLI parameter\r\n\u003c/details\u003e\r\n\r\n\u003cdetails\u003e\r\n  \u003csummary\u003eServing with a local entry point\u003c/summary\u003e\r\n\r\nAssuming you have installed the `infery-llm` local serving requirements, you may use the InferyLLM CLI as a server entrypoint:\r\n```bash\r\n# Serve Deci/DeciLM-7b (from HF hub) on port 9000\r\ninfery-llm serve --model-name Deci/DeciLM-7b --port 9000\r\n\r\n# See all serving options\r\ninfery-llm serve --help\r\n```\r\n\u003c/details\u003e\r\n\r\n## Generation\r\n\r\nAssuming you have a running server listening at `0.0.0.0:9000`, you may submit generation requests in two ways:\r\n\r\n1. Through InferyLLM's `LLMClient`:\r\n```python\r\nimport asyncio\r\nfrom infery_llm.client import LLMClient, GenerationParams\r\n\r\nclient = LLMClient(\"http://0.0.0.0:9000\")\r\n\r\n# Prepare GenerationParams (max_new_tokens, temperature, top_p, top_k, stop_tokens, ...)\r\ngen_params = GenerationParams(max_new_tokens=50, top_p=0.95, top_k=100, temperature=0.1)\r\n\r\n# Submit a single prompt and query results (along with metadata in this case)\r\nresult = client.generate(\"Large language models are \", generation_params=gen_params, return_metadata=True)\r\nprint(f\"Output: {result.output}.\\nGenerated Tokens :{result.metadata[0]['generated_token_count']}\")\r\n\r\n# Submit a batch of prompts (a list of results will be returned this time)\r\nprompts = [\"A receipe for making spaghetti: \", \"5 interesting facts about the President of France are: \", \"Write a short story about a dog named Snoopy: \"]\r\nresult = client.generate(prompts, generation_params=gen_params)\r\n[print(f\"Prompt: {output['prompt']}\\nGeneration: {output['output']}\") for output in result.outputs]\r\n\r\n# Use stop tokens\r\ngen_params = GenerationParams(stop_str_tokens=[1524], stop_strs=[\"add tomatoes\"], skip_special_tokens=True)\r\nresult = client.generate(\"A receipe for making spaghetti: \", generation_params=gen_params)\r\n\r\n# Stream results\r\nfor text in client.generate(\"Will the real Slim Shady please \", generation_params=gen_params, stream=True):\r\n    print(text, end=\"\")\r\n\r\n# Async generation is also supported from within async code:\r\nasync def example():\r\n    result = await client.generate_async(\"AsyncIO is fun because \", generation_params=gen_params)\r\n    print(result.output)\r\nasyncio.run(example())\r\n```\r\n\r\n2. Through a `curl` command (assuming you have [cURL](https://curl.se/) installed)\r\n``` bash\r\ncurl -X POST http://0.0.0.0:9000/generate \\\r\n-H 'Content-Type: application/json' \\\r\n-d '{\"prompts\":[\"def factorial(n: int) -\u003e int:\"], \"generation_params\":{\"max_new_tokens\": 500, \"temperature\":0.5, \"top_k\":50, \"top_p\":0.8}, \"stream\":true}'\r\n```\r\n\r\n## Advanced Usage\r\n\r\n### CLI\r\n\r\nInferyLLM and its CLI are rapidly accumulating more features. For example, the `infery-llm` CLI already allows to `benchmark`\r\nwith numerous configurations, to `prepare` model artifacts before serving in order to [cut down loading time](#lowering-loading-time),\r\nand more. To see the available features you may simply pass `--help` to the `infery-llm` CLI or any of its subcommands:\r\n\r\nFor container users:\r\n```bash\r\n# Query infery-llm CLI help menu\r\ndocker run --entrypoint infery-llm --runtime=nvidia deci.jfrog.io/deci-external-docker-local/infery-llm:latest --help\r\n\r\n# Query the infery-llm CLI's `benchmark` subcommand help menu\r\ndocker run --entrypoint infery-llm --runtime=nvidia deci.jfrog.io/deci-external-docker-local/infery-llm:latest benchmark --help\r\n```\r\n\r\nFor local installation users:\r\n```bash\r\n# Query infery-llm CLI help menu\r\ninfery-llm --help\r\n\r\n# Query the infery-llm CLI's `benchmark` subcommand help menu\r\ninfery-llm benchmark --help\r\n```\r\n\r\n### Lowering loading time (the `prepare` command)\r\n\r\nInferyLLM has its own internal format and per-model artifact requirements. While the required artifacts are automatically\r\ngenerated by the `infery-llm serve` command, you can also generate them ahead of time with the `infery-llm prepare`\r\ncommand, thus drastically cutting down server start-time.\r\n\r\nFor container users:\r\n```bash\r\n# Create artifacts for serving a Deci/DeciCoder-1b and place the result in ~ on the host machine\r\ndocker run --rm --entrypoint infery-llm -v ~/:/models --runtime=nvidia deci.jfrog.io/deci-external-docker-local/infery-llm:latest prepare --hf-model Deci/DeciCoder-1b --output-dir /models/infery_llm_model\r\n\r\n# Now serve the created artifact (specifically here on port 9000)\r\ndocker run --runtime=nvidia -e INFERY_LLM_DECI_TOKEN=[DECI TOKEN] -p 9000:9000 -v ~/:/models deci.jfrog.io/deci-external-docker-local/infery-llm:latest --infery-model-dir /models/infery_llm_model --port 9000\r\n```\r\n\r\nFor local installation users:\r\n```bash\r\n# Create artifacts for serving a Deci/DeciCoder-1b and place the result in ~ on the host machine\r\ninfery-llm prepare --hf-model Deci/DeciCoder-1b --output-dir /models/infery_llm_model\r\n\r\n# Now serve the created artifact (specifically here on port 9000)\r\ninfery-llm serve --infery-model-dir /models/infery_llm_model --port 9000\r\n```\r\n\r\n**Important note on caching:** Just like 🤗 caches downloaded model weights in `~/.cache/huggingface`, InferyLLM \r\ncaches the mentioned artifacts in `~/.cache/deci`. It does so for every model served.\r\nThis means that relaunching a local or containerized server (if it is the same container, not just the same image) will\r\nautomatically lower the loading time.\r\n\r\n### Benchmarking\r\nBy using InferyLLM's `benchmark` cli, you can test different combinations of parameters of serving parameters for your different models.\r\nTest different batch sizes, sequence lengths, max generated tokens, and more. Each GPU and memory size will also change the performance. \u003cbr\u003e\r\n\r\nStarting a new project with `benchmark` should help you find the correct combination for you model deployment.\u003cbr\u003e\r\n\r\nThe metrics benchmark will output: \r\n1. **E2E Time** - time it took to process the number of requests that were sent\r\n2. **E2E Throughput** - How many tokens/s were generated\r\n3. **Mean Latency** - the mean latency for 1 request\r\n\r\nBelow are some more examples for benchmarking different models with several serving configurations: \r\n\u003cdetails\u003e\r\n  \u003csummary\u003eBenchmarking on an A10 GPU\u003c/summary\u003e\r\n\r\nIn order to use `benchmark` cli, you need a json file with a list of prompts, in the following format:\u003cbr\u003e\r\n`[{\"prompt\": \"Your prompt here..\", \"max_new_tokens\": 512}, {\"prompt\": \"Your second prompt..\", \"max_new_tokens\": 256}, {\"prompt\": \"Your 3rd prompt\"}]`.\u003cbr\u003e\r\n\r\nIn the next benchmarking example, lets compare Mistral7B and DeciLM7B on a batch size of 16 with sequence lengths that aren't above 1024 tokens (larger prompts will be skipped, and not tested). \r\nThe prompts file will be saved at `/prompts.json`, the local `prepared` models are saved at `/models/Mistral-7B` and `/models/DeciLM-7B`.\r\n\u003cbr\u003eThe test will run on a A10 GPU instance with 24GB of memory:\r\n\r\n```bash\r\n$ docker run --entrypoint infery-llm --runtime=nvidia -v /prompts.json:/prompts.json  -v /models/:/models/  deci.jfrog.io/deci-external-docker-local/infery-llm:0.0.8 benchmark --infery-model-dir /models/DeciLM-7B --num-prompts 100 --prompts-json /prompts.json --verbose --max-batch-size 16 --input-seq-len=1024 --verbose\r\nINFO: 2024-04-11 08:21:46,272 - INFERY_LLM - Found autotune results (/models/DeciLM-7B/autotune_benchmarks.pkl)\r\nCompleted warmup requests: 100%|██████████| 1/1 [00:20\u003c00:00, 20.51s/it]\r\nCompleted Requests: 100%|██████████| 100/100 [03:37\u003c00:00,  2.18s/it]\r\nINFO: 2024-04-11 08:25:45,044 - INFERY_LLM - \r\n============================== BENCHMARK SUMMARY ==============================\r\nPARAMETERS:\r\n\tNum Prompts:       100\r\n\tInput Tokens:      1025\r\n\tGenerated Tokens:  512\r\n\tMax Batch Size:    16\r\n\tBlock Size:        Model Config Default\r\nRESULTS:\r\n\tE2E Time:          163.709 [s]\r\n\tE2E Throughput:    312.751 [Tokens/s]\r\n\tMean Latency:      1.837 [s]\r\n===============================================================================\r\n```\r\nWe can see that the latency mean of 1 request is 1.837 seconds for this combination.\u003cbr\u003e\r\nNow let's test Mistral7B:\r\n\r\n```bash\r\n$ docker run --entrypoint infery-llm --runtime=nvidia -v /prompts.json:/prompts.json  -v /models/:/models/  deci.jfrog.io/deci-external-docker-local/infery-llm:0.0.8 benchmark --infery-model-dir /models/Mistral-7B --num-prompts 100 --prompts-json /prompts.json --verbose --max-batch-size 16 --input-seq-len=1024 --verbose\r\nINFO: 2024-04-11 09:21:46,272 - INFERY_LLM - Found autotune results (/models/Mistral-7B/autotune_benchmarks.pkl)\r\nCompleted warmup requests: 100%|██████████| 1/1 [00:20\u003c00:00, 20.51s/it]\r\nCompleted Requests: 100%|██████████| 100/100 [03:37\u003c00:00,  2.18s/it]\r\nINFO: 2024-04-11 09:25:45,044 - INFERY_LLM - \r\n============================== BENCHMARK SUMMARY ==============================\r\nPARAMETERS:\r\n\tNum Prompts:       100\r\n\tInput Tokens:      1025\r\n\tGenerated Tokens:  512\r\n\tMax Batch Size:    16\r\n\tBlock Size:        Model Config Default\r\nRESULTS:\r\n\tE2E Time:          217.893 [s]\r\n\tE2E Throughput:    234.977 [Tokens/s]\r\n\tMean Latency:      2.706 [s]\r\n===============================================================================\r\n```\r\nThe Tokens per second for batch-size=16 and sequence-len=1024 was 234 TKS.\u003cbr\u003e\r\nWe now know that **DeciLM-7B is 1.5X faster than Mistral-7B** in the exact same setup and has **33% better throughput!**\u003cbr\u003e\r\n\r\nWhat will be the result for Mistral7B if our sequences are much longer (2048), but we use a smaller batch size (8):\r\n```bash\r\n$ docker run --entrypoint infery-llm --runtime=nvidia -v /prompts.json:/prompts.json  -v /models/:/models/  deci.jfrog.io/deci-external-docker-local/infery-llm:0.0.8 benchmark --infery-model-dir /models/Mistral-7B --num-prompts 100 --prompts-json /prompts.json --verbose --max-batch-size 8 --input-seq-len=2048 --verbose\r\nINFO: 2024-04-11 09:16:46,272 - INFERY_LLM - Found autotune results (/models/Mistral-7B/autotune_benchmarks.pkl)\r\nCompleted warmup requests: 100%|██████████| 1/1 [00:20\u003c00:00, 20.51s/it]\r\nCompleted Requests: 100%|██████████| 100/100 [03:37\u003c00:00,  2.18s/it]\r\nINFO: 2024-04-11 09:20:45,044 - INFERY_LLM - \r\n============================== BENCHMARK SUMMARY ==============================\r\nPARAMETERS:\r\n\tNum Prompts:       100\r\n\tInput Tokens:      1501\r\n\tGenerated Tokens:  512\r\n\tMax Batch Size:    8\r\n\tBlock Size:        Model Config Default\r\nRESULTS:\r\n\tE2E Time:          330.644 [s]\r\n\tE2E Throughput:    154.849 [Tokens/s]\r\n\tMean Latency:      3.306 [s]\r\n===============================================================================\r\n```\r\nWe can see that the total time was longer than before, and that the Mean Latency is longer by almost ~1 second.\r\n\u003c/details\u003e","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeci-ai%2Finferyllm-docs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdeci-ai%2Finferyllm-docs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdeci-ai%2Finferyllm-docs/lists"}