{"id":13492662,"url":"https://github.com/huggingface/optimum-nvidia","last_synced_at":"2025-10-14T15:32:22.299Z","repository":{"id":211008094,"uuid":"701226931","full_name":"huggingface/optimum-nvidia","owner":"huggingface","description":null,"archived":false,"fork":false,"pushed_at":"2025-02-06T11:18:27.000Z","size":1233,"stargazers_count":942,"open_issues_count":54,"forks_count":98,"subscribers_count":36,"default_branch":"main","last_synced_at":"2025-03-25T07:38:46.275Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-06T07:40:55.000Z","updated_at":"2025-03-25T04:44:13.000Z","dependencies_parsed_at":"2023-12-18T18:59:38.340Z","dependency_job_id":"a3367290-14d2-46fc-8181-462f0326fffb","html_url":"https://github.com/huggingface/optimum-nvidia","commit_stats":null,"previous_names":["huggingface/optimum-nvidia"],"tags_count":9,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-nvidia","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-nvidia/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-nvidia/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-nvidia/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/optimum-nvidia/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246012742,"owners_count":20709506,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T19:01:08.003Z","updated_at":"2025-10-14T15:32:17.273Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\n\u003cdiv align=\"center\"\u003e\n\nOptimum-NVIDIA\n===========================\n\u003ch4\u003e Optimized inference with NVIDIA and Hugging Face \u003c/h4\u003e\n\n[![Documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat)](https://huggingface.co/docs/optimum/index)\n[![python](https://img.shields.io/badge/python-3.10-green)](https://www.python.org/downloads/release/python-31013/)\n[![cuda](https://img.shields.io/badge/cuda-12.6-green)](https://developer.nvidia.com/cuda-downloads)\n[![trt-llm](https://img.shields.io/badge/TensorRT--LLM-0.15.0-green)](https://github.com/nvidia/tensorrt-llm)\n[![license](https://img.shields.io/badge/license-Apache%202-blue)](./LICENSE)\n\n---\n\u003cdiv align=\"left\"\u003e\n\nOptimum-NVIDIA delivers the best inference performance on the NVIDIA platform through Hugging Face. Run LLaMA 2 at 1,200 tokens/second (up to 28x faster than the framework) by changing just a single line in your existing transformers code.\n\n\u003c/div\u003e\u003c/div\u003e\n\n# Installation\n\n## Pip\n\nPip installation flow has been validated on Ubuntu only at this stage.\n\n```shell\napt-get update \u0026\u0026 apt-get -y install python3.10 python3-pip openmpi-bin libopenmpi-dev\npython -m pip install --pre --extra-index-url https://pypi.nvidia.com optimum-nvidia\n``` \n\nFor developers who want to target the best performances, please look at the installation methods below.\n\n## Docker container\nYou can use a Docker container to try Optimum-NVIDIA today. Images are available on the Hugging Face Docker Hub.\n\n```bash\ndocker pull huggingface/optimum-nvidia\n```\n\n## Building from source\n\u003c!---\nCurrently, TRT LLM is built and run with Docker, so we should wait until pip installation is available;\nIdeally the user doesn't need to use docker at all to build from source, they should be able to run something like\n`git clone [...] \u0026\u0026 pip install -e optimum-nvidia`\n--\u003e\n\nInstead of using the pre-built docker container, you can build Optimum-NVIDIA from source:\n```bash\nTARGET_SM=\"90-real;89-real\"\ngit clone --recursive --depth=1 https://github.com/huggingface/optimum-nvidia.git\ncd optimum-nvidia/third-party/tensorrt-llm\nmake -C docker release_build CUDA_ARCHS=$TARGET_SM\ncd ../.. \u0026\u0026 docker build -t \u003corganisation_name/image_name\u003e:\u003cversion\u003e -f docker/Dockerfile .\n```\n\n\u003c!-- \n```bash\ngit clone git@github.com:huggingface/optimum-nvidia.git\ncd optimum-nvidia\ndocker build Dockerfile\ndocker run optimum-nvidia\n``` --\u003e\n\n# Quickstart Guide\n## Pipelines\n\nHugging Face pipelines provide a simple yet powerful abstraction to quickly set up inference. If you already have a pipeline from transformers, you can unlock the performance benefits of Optimum-NVIDIA by just changing one line.\n\n```diff\n- from transformers.pipelines import pipeline\n+ from optimum.nvidia.pipelines import pipeline\n\npipe = pipeline('text-generation', 'meta-llama/Llama-2-7b-chat-hf', use_fp8=True)\npipe(\"Describe a real-world application of AI in sustainable energy.\")\n```\n\n## Generate\n\nIf you want control over advanced features like quantization and token selection strategies, we recommend using the `generate()` API. Just like with `pipelines`, switching from existing transformers code is super simple.\n\n```diff\n- from transformers import AutoModelForCausalLM\n+ from optimum.nvidia import AutoModelForCausalLM\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-chat-hf\", padding_side=\"left\")\n\nmodel = AutoModelForCausalLM.from_pretrained(\n  \"meta-llama/Llama-2-7b-chat-hf\",\n+ use_fp8=True,\n+ max_prompt_length=1024,\n+ max_output_length=2048, # Must be at least size of max_prompt_length + max_new_tokens\n+ max_batch_size=8,\n)\n\nmodel_inputs = tokenizer([\"How is autonomous vehicle technology transforming the future of transportation and urban planning?\"], return_tensors=\"pt\").to(\"cuda\")\n\ngenerated_ids = model.generate(\n    **model_inputs, \n    top_k=40, \n    top_p=0.7, \n    repetition_penalty=10,\n)\n\ntokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]\n```\n\nTo learn more about text generation with LLMs, check out [this guide](https://huggingface.co/docs/transformers/llm_tutorial)!\n\n\u003c!-- For more details, read our [documentation](https://huggingface.com/docs/optimum/nvidia/index). --\u003e\n\n# Support Matrix\nWe test Optimum-NVIDIA on 4090, L40S, and H100 Tensor Core GPUs, though it is expected to work on any GPU based on the following architectures: \n* Ampere (A100/A30 are supported. Experimental support for A10, A40, RTX Ax000)\n* Hopper\n* Ada-Lovelace\n\nNote that FP8 support is only available on GPUs based on Hopper and Ada-Lovelace architectures.\n\nOptimum-NVIDIA works on Linux will support Windows soon.\n\nOptimum-NVIDIA currently accelerates text-generation with LLaMAForCausalLM, and we are actively working to expand support to include more model architectures and tasks.\n\n\u003c!-- Optimum-NVIDIA supports the following model architectures and tasks:\n\n| Model             | Tasks           |\n| :----             | :----           |\n| Gemma             | TextGeneration  |\n| Llama             | TextGeneration  |\n| Mistral           | TextGeneration  |\n| Additional Models | Coming soon     | --\u003e\n\n# Contributing\n\nCheck out our [Contributing Guide](./CONTRIBUTING.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-nvidia","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Foptimum-nvidia","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-nvidia/lists"}