Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/SJTU-IPADS/PowerInfer
High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
https://github.com/SJTU-IPADS/PowerInfer
bamboo-7b falcon large-language-models llama llm llm-inference local-inference
Last synced: about 1 month ago
JSON representation
High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
- Host: GitHub
- URL: https://github.com/SJTU-IPADS/PowerInfer
- Owner: SJTU-IPADS
- License: mit
- Created: 2023-12-15T02:24:10.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-07-15T15:05:12.000Z (about 2 months ago)
- Last Synced: 2024-07-31T01:24:35.750Z (about 1 month ago)
- Topics: bamboo-7b, falcon, large-language-models, llama, llm, llm-inference, local-inference
- Language: C++
- Homepage:
- Size: 12.1 MB
- Stars: 7,710
- Watchers: 76
- Forks: 395
- Open Issues: 88
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- AiTreasureBox - SJTU-IPADS/PowerInfer - 09-08_7850_0](https://img.shields.io/github/stars/SJTU-IPADS/PowerInfer.svg)|High-speed Large Language Model Serving on PCs with Consumer-grade GPUs| (Repos)
- awesome-generative-ai - PowerInfer - speed inference engine for deploying LLMs locally (Large Language Models (LLMs) / Running LLMs Locally)
- awesome-production-machine-learning - PowerInfer - IPADS/PowerInfer.svg?style=social) - PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device. (Model Serving and Monitoring)
- StarryDivineSky - SJTU-IPADS/PowerInfer