{"id":13574373,"url":"https://github.com/intel/intel-extension-for-pytorch","last_synced_at":"2025-05-12T13:18:40.860Z","repository":{"id":37351083,"uuid":"256061008","full_name":"intel/intel-extension-for-pytorch","owner":"intel","description":"A Python package for extending the official PyTorch that can easily obtain performance on Intel platform","archived":false,"fork":false,"pushed_at":"2025-05-07T06:54:52.000Z","size":118860,"stargazers_count":1847,"open_issues_count":195,"forks_count":275,"subscribers_count":33,"default_branch":"main","last_synced_at":"2025-05-12T13:17:59.295Z","etag":null,"topics":["deep-learning","intel","machine-learning","neural-network","pytorch","quantization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/intel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-04-15T23:35:29.000Z","updated_at":"2025-05-10T13:17:34.000Z","dependencies_parsed_at":"2023-10-12T18:25:34.000Z","dependency_job_id":"3e524895-bf8b-4df1-9ea2-d11be80df0ca","html_url":"https://github.com/intel/intel-extension-for-pytorch","commit_stats":{"total_commits":2137,"total_committers":83,"mean_commits":"25.746987951807228","dds":0.8568086102012167,"last_synced_commit":"140f0981850dc14ff57129f342e39953b5c84142"},"previous_names":[],"tags_count":50,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Fintel-extension-for-pytorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Fintel-extension-for-pytorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Fintel-extension-for-pytorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/intel%2Fintel-extension-for-pytorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/intel","download_url":"https://codeload.github.com/intel/intel-extension-for-pytorch/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253745196,"owners_count":21957319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","intel","machine-learning","neural-network","pytorch","quantization"],"created_at":"2024-08-01T15:00:51.009Z","updated_at":"2025-05-12T13:18:40.828Z","avatar_url":"https://github.com/intel.png","language":"Python","funding_links":[],"categories":["Table of Contents","Python","A01_文本生成_文本对话"],"sub_categories":["AI - Frameworks and Toolkits","大语言对话模型及数据"],"readme":"\u003cdiv align=\"center\"\u003e\n  \nIntel® Extension for PyTorch\\*\n===========================\n\n\u003c/div\u003e\n\n**CPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/main)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/tutorials/getting_started.html)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/cpu/latest/)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu\u0026version=v2.7.0%2Bcpu)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/llm) \u003cbr\u003e\n**GPU** [💻main branch](https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[🌱Quick Start](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/getting_started.html)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[📖Documentations](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[🏃Installation](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=gpu)\u0026nbsp;\u0026nbsp;\u0026nbsp;|\u0026nbsp;\u0026nbsp;\u0026nbsp;[💻LLM Example](https://github.com/intel/intel-extension-for-pytorch/tree/xpu-main/examples/gpu/llm)\u003cbr\u003e  \n\nIntel® Extension for PyTorch\\* extends PyTorch\\* with up-to-date features optimizations for an extra performance boost on Intel hardware. Optimizations take advantage of Intel® Advanced Vector Extensions 512 (Intel® AVX-512) Vector Neural Network Instructions (VNNI) and Intel® Advanced Matrix Extensions (Intel® AMX) on Intel CPUs as well as Intel X\u003csup\u003ee\u003c/sup\u003e Matrix Extensions (XMX) AI engines on Intel discrete GPUs. Moreover, Intel® Extension for PyTorch* provides easy GPU acceleration for Intel discrete GPUs through the PyTorch* xpu device.\n\n## ipex.llm - Large Language Models (LLMs) Optimization\n\nIn the current technological landscape, Generative AI (GenAI) workloads and models have gained widespread attention and popularity. Large Language Models (LLMs) have emerged as the dominant models driving these GenAI applications. Starting from 2.1.0, specific optimizations for certain LLM models are introduced in the Intel® Extension for PyTorch\\*. Check [**LLM optimizations**](./examples/cpu/llm) for details.\n\n### Optimized Model List\n\nWe have supported a long list of LLMs, including the most notable open-source models\nlike Llama series, Qwen series, Phi-3/Phi-4 series,\nand the phenomenal high-quality reasoning model DeepSeek-R1.\n\n| MODEL FAMILY | MODEL NAME (Huggingface hub) | FP32 | BF16 | Weight only quantization INT8 | Weight only quantization INT4 |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|LLAMA| meta-llama/Llama-2-7b-hf | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Llama-2-13b-hf | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Llama-2-70b-hf | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Meta-Llama-3-8B | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Meta-Llama-3-70B | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Meta-Llama-3.1-8B-Instruct | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Llama-3.2-3B-Instruct | ✅ | ✅ | ✅ | ✅ |\n|LLAMA| meta-llama/Llama-3.2-11B-Vision-Instruct | ✅ | ✅ | ✅ | ✅ |\n|GPT-J| EleutherAI/gpt-j-6b | ✅ | ✅ | ✅ | ✅ |\n|GPT-NEOX| EleutherAI/gpt-neox-20b | ✅ | ✅ | ✅ | ✅ |\n|DOLLY| databricks/dolly-v2-12b | ✅ | ✅ | ✅ | ✅ |\n|FALCON| tiiuae/falcon-7b  | ✅ | ✅ | ✅ | ✅ |\n|FALCON| tiiuae/falcon-11b | ✅ | ✅ | ✅ | ✅ |\n|FALCON| tiiuae/falcon-40b | ✅ | ✅ | ✅ | ✅ |\n|FALCON| tiiuae/Falcon3-7B-Instruct | ✅ | ✅ | ✅ | ✅ |\n|OPT| facebook/opt-30b | ✅ | ✅ | ✅ | ✅ |\n|OPT| facebook/opt-1.3b | ✅ | ✅ | ✅ | ✅ |\n|Bloom| bigscience/bloom-1b7 | ✅ | ✅ | ✅ | ✅ |\n|CodeGen| Salesforce/codegen-2B-multi | ✅ | ✅ | ✅ | ✅ |\n|Baichuan| baichuan-inc/Baichuan2-7B-Chat | ✅ | ✅ | ✅ | ✅ |\n|Baichuan| baichuan-inc/Baichuan2-13B-Chat | ✅ | ✅ | ✅ | ✅ |\n|Baichuan| baichuan-inc/Baichuan-13B-Chat | ✅ | ✅ | ✅ | ✅ |\n|ChatGLM| THUDM/chatglm3-6b | ✅ | ✅ | ✅ | ✅ |\n|ChatGLM| THUDM/chatglm2-6b | ✅ | ✅ | ✅ | ✅ |\n|GPTBigCode| bigcode/starcoder | ✅ | ✅ | ✅ | ✅ |\n|T5| google/flan-t5-xl | ✅ | ✅ | ✅ | ✅ |\n|MPT| mosaicml/mpt-7b | ✅ | ✅ | ✅ | ✅ |\n|Mistral| mistralai/Mistral-7B-v0.1 | ✅ | ✅ | ✅ | ✅ |\n|Mixtral| mistralai/Mixtral-8x7B-v0.1 | ✅ | ✅ | ✅ | ✅ |\n|Stablelm| stabilityai/stablelm-2-1_6b | ✅ | ✅ | ✅ | ✅ |\n|Qwen| Qwen/Qwen-7B-Chat | ✅ | ✅ | ✅ | ✅ |\n|Qwen| Qwen/Qwen2-7B | ✅ | ✅ | ✅ | ✅ |\n|Qwen| Qwen/Qwen2.5-7B-Instruct | ✅ | ✅ | ✅ | ✅ |\n|LLaVA| liuhaotian/llava-v1.5-7b | ✅ | ✅ | ✅ | ✅ |\n|GIT| microsoft/git-base | ✅ | ✅ | ✅ | ✅ |\n|Yuan| IEITYuan/Yuan2-102B-hf | ✅ | ✅ | ✅ |   |\n|Phi| microsoft/phi-2 | ✅ | ✅ | ✅ | ✅ |\n|Phi| microsoft/Phi-3-mini-4k-instruct | ✅ | ✅ | ✅ | ✅ |\n|Phi| microsoft/Phi-3-mini-128k-instruct | ✅ | ✅ | ✅ | ✅ |\n|Phi| microsoft/Phi-3-medium-4k-instruct | ✅ | ✅ | ✅ | ✅ |\n|Phi| microsoft/Phi-3-medium-128k-instruct | ✅ | ✅ | ✅ | ✅ |\n|Phi| microsoft/Phi-4-mini-instruct | ✅ | ✅ | ✅ |   |\n|Phi| microsoft/Phi-4-multimodal-instruct | ✅ | ✅ | ✅ |   |\n|Whisper| openai/whisper-large-v2 | ✅ | ✅ | ✅ | ✅ |\n|Maira| microsoft/maira-2 | ✅ | ✅ | ✅ | ✅ |\n|Jamba| ai21labs/Jamba-v0.1 | ✅ | ✅ | ✅ | ✅ |\n|DeepSeek| deepseek-ai/DeepSeek-V2.5-1210 | ✅ | ✅ | ✅ | ✅ |\n|DeepSeek| meituan/DeepSeek-R1-Channel-INT8 |   |   | ✅ |   |\n\n*Note*: The above verified models (including other models in the same model family, like \"codellama/CodeLlama-7b-hf\" from LLAMA family) are well supported with all optimizations like indirect access KV cache, fused ROPE, and customized linear kernels.\nWe are working in progress to better support the models in the tables with various data types. In addition, more models will be optimized in the future.\n\nIn addition, Intel® Extension for PyTorch* introduces module level optimization APIs (prototype feature) since release 2.3.0.\nThe feature provides optimized alternatives for several commonly used LLM modules and functionalities for the optimizations of the niche or customized LLMs.\nPlease read [**LLM module level optimization practice**](./examples/cpu/inference/python/llm-modeling) to better understand how to optimize your own LLM and achieve better performance.\n\n## Support\n\nThe team tracks bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues/). Before submitting a suggestion or bug report, search the existing GitHub issues to see if your issue has already been reported.\n\n## License\n\n_Apache License_, Version _2.0_. As found in [LICENSE](https://github.com/intel/intel-extension-for-pytorch/blob/main/LICENSE) file.\n\n## Security\n\nSee Intel's [Security Center](https://www.intel.com/content/www/us/en/security-center/default.html)\nfor information on how to report a potential security issue or vulnerability.\n\nSee also: [Security Policy](SECURITY.md)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintel%2Fintel-extension-for-pytorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fintel%2Fintel-extension-for-pytorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fintel%2Fintel-extension-for-pytorch/lists"}