https://github.com/mit-han-lab/TinyChatEngine

TinyChatEngine: On-Device LLM Inference Library
https://github.com/mit-han-lab/TinyChatEngine

arm c cpp cuda-programming deep-learning edge-computing large-language-models on-device-ai quantization x86-64

Last synced: about 2 months ago
JSON representation

TinyChatEngine: On-Device LLM Inference Library

• Host: GitHub
• URL: https://github.com/mit-han-lab/TinyChatEngine
• Owner: mit-han-lab
• License: mit
• Created: 2023-05-24T09:29:41.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-07-04T04:20:45.000Z (6 months ago)
• Last Synced: 2024-08-04T01:17:15.975Z (5 months ago)
• Topics: arm, c, cpp, cuda-programming, deep-learning, edge-computing, large-language-models, on-device-ai, quantization, x86-64
• Language: C++
• Homepage: https://mit-han-lab.github.io/TinyChatEngine/
• Size: 83.3 MB
• Stars: 657
• Watchers: 12
• Forks: 64
• Open Issues: 31
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

• ai-game-devtools - TinyChatEngine - Device LLM Inference Library. | | | Tool | (Project List / <span id="tool">Tool (AI LLM)</span>)
• Awesome-LLM-Compression - [Code
• awesome-llm-and-aigc - TinyChatEngine - han-lab/TinyChatEngine?style=social"/> : TinyChatEngine: On-Device LLM Inference Library. Running large language models (LLMs) and visual language models (VLMs) on the edge is useful: copilot services (coding, office, smart reply) on laptops, cars, robots, and more. Users can get instant responses with better privacy, as the data is local. This is enabled by LLM model compression technique: [SmoothQuant](https://github.com/mit-han-lab/smoothquant) and [AWQ (Activation-aware Weight Quantization)](https://github.com/mit-han-lab/llm-awq), co-designed with TinyChatEngine that implements the compressed low-precision model. Feel free to check out our [slides](https://github.com/mit-han-lab/TinyChatEngine/blob/main/assets/slides.pdf) for more details! (Summary)