https://github.com/FMInference/H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.
https://github.com/FMInference/H2O

gpt-3 heavy-hitters high-throughput kv-cache large-language-models sparsity

Last synced: 3 months ago
JSON representation

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

• Host: GitHub
• URL: https://github.com/FMInference/H2O
• Owner: FMInference
• Created: 2023-06-12T06:03:19.000Z (over 1 year ago)
• Default Branch: main
• Last Pushed: 2024-06-18T02:55:47.000Z (5 months ago)
• Last Synced: 2024-07-14T13:34:47.258Z (4 months ago)
• Topics: gpt-3, heavy-hitters, high-throughput, kv-cache, large-language-models, sparsity
• Language: Python
• Homepage:
• Size: 39.1 MB
• Stars: 319
• Watchers: 5
• Forks: 29
• Open Issues: 24
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

• StarryDivineSky - FMInference/H2O - NeoX 在各种任务中验证了算法的准确性。在 OPT-6.7B 和 OPT-30B 上，我们实施了 20% 重击器的 H2O，将吞吐量提高了 29×、29× 和 3× 三个领先的推理系统 DeepSpeed Zero-Inference、Hugging Face Accelerate 和 FlexGen。在相同的批量大小下，H2O 最多可以减少 1.9× 的延迟。 (文本生成、文本对话 / 大语言对话模型及数据)