{"id":14567705,"url":"https://github.com/huggingface/optimum-tpu","last_synced_at":"2025-10-14T15:32:18.678Z","repository":{"id":232178004,"uuid":"756234502","full_name":"huggingface/optimum-tpu","owner":"huggingface","description":"Google TPU optimizations for transformers models","archived":false,"fork":false,"pushed_at":"2025-01-21T13:24:41.000Z","size":520,"stargazers_count":120,"open_issues_count":9,"forks_count":30,"subscribers_count":34,"default_branch":"main","last_synced_at":"2025-09-30T18:02:30.472Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://huggingface.co/docs/optimum-tpu","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/huggingface.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-02-12T08:54:38.000Z","updated_at":"2025-08-25T01:19:03.000Z","dependencies_parsed_at":"2024-12-09T11:25:23.990Z","dependency_job_id":"c78c0413-afd7-486a-a03e-ef7b833a7cc6","html_url":"https://github.com/huggingface/optimum-tpu","commit_stats":null,"previous_names":["huggingface/optimum-tpu"],"tags_count":14,"template":false,"template_full_name":null,"purl":"pkg:github/huggingface/optimum-tpu","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-tpu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-tpu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-tpu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-tpu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/huggingface","download_url":"https://codeload.github.com/huggingface/optimum-tpu/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/huggingface%2Foptimum-tpu/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279019322,"owners_count":26086711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-07T06:00:50.945Z","updated_at":"2025-10-14T15:32:18.642Z","avatar_url":"https://github.com/huggingface.png","language":"Python","funding_links":[],"categories":["Optimizations","Python"],"sub_categories":["Profiling"],"readme":"\u003cdiv align=\"center\"\u003e\n\nOptimum-TPU\n===========================\n\u003ch4\u003eTake the most out of Google Cloud TPUs with the ease of 🤗 transformers\u003c/h4\u003e\n\n[![Documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat)](https://huggingface.co/docs/optimum/index)\n[![license](https://img.shields.io/badge/license-Apache%202-blue)](./LICENSE)\n[![Optimum TPU / Test TGI on TPU](https://github.com/huggingface/optimum-tpu/actions/workflows/test-pytorch-xla-tpu-tgi.yml/badge.svg)](https://github.com/huggingface/optimum-tpu/actions/workflows/test-pytorch-xla-tpu-tgi.yml)\n\u003c/div\u003e\n\n[Tensor Processing Units (TPU)](https://cloud.google.com/tpu) are AI accelerator made by Google to optimize\nperformance and cost from AI training to inference.\n\nThis repository exposes an interface similar to what Hugging Face transformers library provides to interact with\na magnitude of models developed by research labs, institutions and the community.\n\nWe aim at providing our user the best possible performances targeting Google Cloud TPUs for both training and inference\nworking closely with Google and Google Cloud to make this a reality.\n\n\n## Supported Model and Tasks\n\nWe currently support a few LLM models targeting text generation scenarios:\n- 💎 Gemma (2b, 7b)\n- 🦙 Llama2 (7b) and Llama3 (8b). On Text Generation Inference with Jetstream Pytorch, also Llama3.1, Llama3.2 and Llama3.3 (text-only models) are supported, up to 70B parameters.\n- 💨 Mistral (7b)\n\n\n## Installation\n\n`optimum-tpu` comes with an handy PyPi released package compatible with your classical python dependency management tool.\n\n`pip install optimum-tpu -f https://storage.googleapis.com/libtpu-releases/index.html`\n\n`export PJRT_DEVICE=TPU`\n\n\n## Inference\n\n`optimum-tpu` provides a set of dedicated tools and integrations in order to leverage Cloud TPUs for inference, especially\non the latest TPU version `v5e` and `v6e`.\n\nOther TPU versions will be supported along the way.\n\n### Text-Generation-Inference\n\nAs part of the integration, we do support a [text-generation-inference (TGI)](https://github.com/huggingface/optimum-tpu/tree/main/text-generation-inference) backend allowing to deploy and serve\nincoming HTTP requests and execute them on Cloud TPUs.\n\nPlease see the [TGI specific documentation](text-generation-inference) on how to get started.\n\n### JetStream Pytorch Engine\n\n`optimum-tpu` provides an optional support of JetStream Pytorch engine inside of TGI. This support can be installed using the dedicated CLI command:\n\n```shell\noptimum-tpu install-jetstream-pytorch\n```\n\nTo enable the support, export the environment variable `JETSTREAM_PT=1`.\n\n## Training\n\nFine-tuning is supported and tested on the TPU `v5e`. We have tested so far:\n\n- 🦙 Llama-2 7B, Llama-3 8B and newer;\n- 💎 Gemma 2B and 7B.\n\nYou can check the examples:\n\n- [Fine-Tune Gemma on Google TPU](https://github.com/huggingface/optimum-tpu/blob/main/examples/language-modeling/gemma_tuning.ipynb)\n- The [Llama fine-tuning script](https://github.com/huggingface/optimum-tpu/blob/main/examples/language-modeling/llama_tuning.ipynb)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-tpu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhuggingface%2Foptimum-tpu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhuggingface%2Foptimum-tpu/lists"}