https://github.com/brownwang0426/cullama

Customize Your Own Llama Attention! No peft, no lora, only customizing your own llama attention to whatever you want!
https://github.com/brownwang0426/cullama

customization llama

Last synced: 2 months ago
JSON representation

Customize Your Own Llama Attention! No peft, no lora, only customizing your own llama attention to whatever you want!

• Host: GitHub
• URL: https://github.com/brownwang0426/cullama
• Owner: Brownwang0426
• License: mit
• Created: 2024-08-17T03:06:01.000Z (10 months ago)
• Default Branch: main
• Last Pushed: 2024-08-24T10:52:56.000Z (10 months ago)
• Last Synced: 2025-02-14T11:02:27.065Z (4 months ago)
• Topics: customization, llama
• Language: Jupyter Notebook
• Homepage:
• Size: 423 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Cullama

Customize Your Own Llama Attention! No peft, no lora, only customizing your own llama attention to whatever you want!

The purpose of this repo is to demostrate how you can customize your llama and fully pre-train it. It is not meant to compete with other deeper models which are made by large corpo with GPU advantage.