https://github.com/prophecy-jimpsons/east
Exploring Extreme Adaptive Sparse Training (EAST) on a 25.3B parameter language model
https://github.com/prophecy-jimpsons/east
ai east llm
Last synced: about 2 months ago
JSON representation
Exploring Extreme Adaptive Sparse Training (EAST) on a 25.3B parameter language model
- Host: GitHub
- URL: https://github.com/prophecy-jimpsons/east
- Owner: Prophecy-Jimpsons
- Created: 2025-01-15T20:29:29.000Z (4 months ago)
- Default Branch: main
- Last Pushed: 2025-03-19T21:31:58.000Z (about 2 months ago)
- Last Synced: 2025-03-19T21:38:42.677Z (about 2 months ago)
- Topics: ai, east, llm
- Language: Python
- Homepage: https://jimpsons.org
- Size: 65.4 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Extreme Adaptive Sparse Training (EAST) for Large Language Models
Background
EAST is a sparse learning technique designed to train deep neural networks at extreme sparsity levels without sacrificing accuracy. This repository aims to push the boundaries of EAST by testing its effectiveness on one of the largest language models to date.
Model Details
Model architecture: [Different for different tests]
Parameter count: 25.3 billion
Dataset: TBA
EAST Implementation
This repository implements the EAST method as described in the paper by Mrare Jimmy. The implementation includes:
Dynamic ReLU phasing (DyReLU)
Weight sharing
Cyclic sparsity
Goals and Contributions
The primary goal of this repository is to investigate the effectiveness of EAST on large language models. By contributing to this repository, you can help:
Advance the state-of-the-art in sparse learning for large language models
Improve the computational efficiency of large language models
Explore new applications of EAST in natural language processing
Acknowledgments
(https://arxiv.org/abs/2411.13545)
License
TBA
Licenses will be defined later.