https://github.com/efeslab/fiddler

Fast Inference of MoE Models with CPU-GPU Orchestration
https://github.com/efeslab/fiddler

llm llm-inference local-inference mixtral-8x7b mixture-of-experts

Last synced: 3 months ago
JSON representation

Fast Inference of MoE Models with CPU-GPU Orchestration

• Host: GitHub
• URL: https://github.com/efeslab/fiddler
• Owner: efeslab
• License: apache-2.0
• Created: 2024-02-05T01:01:27.000Z (9 months ago)
• Default Branch: main
• Last Pushed: 2024-05-22T08:38:09.000Z (6 months ago)
• Last Synced: 2024-06-10T08:49:07.202Z (5 months ago)
• Topics: llm, llm-inference, local-inference, mixtral-8x7b, mixture-of-experts
• Language: Python
• Homepage: https://arxiv.org/abs/2402.07033
• Size: 1.23 MB
• Stars: 148
• Watchers: 8
• Forks: 16
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project