https://github.com/apehex/llaminate
Optimized llama3 using tokun
https://github.com/apehex/llaminate
Last synced: over 1 year ago
JSON representation
Optimized llama3 using tokun
- Host: GitHub
- URL: https://github.com/apehex/llaminate
- Owner: apehex
- Created: 2024-05-15T13:36:08.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-01-31T18:17:20.000Z (over 1 year ago)
- Last Synced: 2025-01-31T19:23:43.966Z (over 1 year ago)
- Language: Python
- Homepage:
- Size: 4.32 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: .github/README.md
Awesome Lists containing this project
README
# llaminate
> Optimized version of [llama3][github-llama3], using [tokun][github-tokun].
This project is a showcase for a neural tokenization technique.
Since the inputs are compressed and have a smaller shape, the LLM is downsized accordingly.
For example, llama3-8b is brought down to 34 million parameters instead of 8 billion.
## Installation
## Usage
## Resources
### Models
### Notebooks
Final model:
- pretraining: [file][notebook-github-pretrain] / [Google Colab][notebook-colab-pretrain]
- fine-tuning: file / Google Colab
## TODO
See [TODO](TODO.md).
## Credits
This project winks at [llama3 from Meta][github-llama3], but doesn't actually its weights nor code.
## License
Licensed under the [aGPLv3](LICENSE.md).
[github-llama3]: https://github.com/meta-llama/llama3
[github-tokun]: https://github.com/apehex/tokun
[notebook-colab-pretrain]: https://colab.research.google.com/github/apehex/llaminate/blob/main/notebooks/llaminate.student.pretrain.ipynb
[notebook-github-pretrain]: ../notebooks/llaminate.student.pretrain.ipynb