https://github.com/SJTU-IPADS/PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
https://github.com/SJTU-IPADS/PowerInfer

bamboo-7b falcon large-language-models llama llm llm-inference local-inference

Last synced: about 2 months ago
JSON representation

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

• Host: GitHub
• URL: https://github.com/SJTU-IPADS/PowerInfer
• Owner: SJTU-IPADS
• License: mit
• Created: 2023-12-15T02:24:10.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-09-06T07:38:57.000Z (4 months ago)
• Last Synced: 2024-10-18T23:17:13.769Z (2 months ago)
• Topics: bamboo-7b, falcon, large-language-models, llama, llm, llm-inference, local-inference
• Language: C++
• Homepage:
• Size: 12.1 MB
• Stars: 7,929
• Watchers: 77
• Forks: 409
• Open Issues: 96
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• AiTreasureBox - SJTU-IPADS/PowerInfer - 12-20_8021_2](https://img.shields.io/github/stars/SJTU-IPADS/PowerInfer.svg)|High-speed Large Language Model Serving on PCs with Consumer-grade GPUs| (Repos)
• awesome-generative-ai - PowerInfer - speed inference engine for deploying LLMs locally (Large Language Models (LLMs) / Running LLMs Locally)
• awesome-production-machine-learning - PowerInfer - IPADS/PowerInfer.svg?style=social) - PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device. (Deployment and Serving)
• StarryDivineSky - SJTU-IPADS/PowerInfer
• Awesome-LLMs-on-device - [Github