Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/lxe/cerebras-lora-alpaca
LoRA weights for Cerebras-GPT-2.7b finetuned on Alpaca dataset with shorter prompt
https://github.com/lxe/cerebras-lora-alpaca
Last synced: 14 days ago
JSON representation
LoRA weights for Cerebras-GPT-2.7b finetuned on Alpaca dataset with shorter prompt
- Host: GitHub
- URL: https://github.com/lxe/cerebras-lora-alpaca
- Owner: lxe
- Created: 2023-03-31T06:53:42.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-04-06T08:29:53.000Z (almost 2 years ago)
- Last Synced: 2025-01-12T21:22:00.585Z (23 days ago)
- Language: Jupyter Notebook
- Size: 6.69 MB
- Stars: 63
- Watchers: 4
- Forks: 11
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-instruction-datasets - cerebras-lora-alpaca - GPT | 2.7B | [AlpacaDataCleaned](https://github.com/gururise/AlpacaDataCleaned) | 52k | En | (Statistics)
- Awesome-instruction-tuning - cerebras-lora-alpaca - GPT | 2.7B | [AlpacaDataCleaned](https://github.com/gururise/AlpacaDataCleaned) | 52k | En | (Datasets and Models / Generated by LLMs)
README
---
title: Lora Cerebras Gpt2.7b Alpaca Shortprompt
emoji: 🐨
colorFrom: yellow
colorTo: pink
sdk: gradio
sdk_version: 3.23.0
app_file: app.py
pinned: false
license: apache-2.0
---
### 🦙🐕🧠 Cerebras-GPT2.7B LoRA Alpaca ShortPrompt
[](https://colab.research.google.com/github/lxe/cerebras-lora-alpaca/colab.ipynb)
[](https://huggingface.co/spaces/lxe/lxe-Cerebras-GPT-2.7B-Alpaca-SP)
[](https://github.com/lxe/no-bugs)
[](https://github.com/lxe/onehundred/tree/master)
Scripts to finetune [Cerebras GPT2.7B](https://www.cerebras.net/blog/cerebras-gpt-a-family-of-open-compute-efficient-large-language-models/) on the [Alpaca dataset](https://github.com/gururise/AlpacaDataCleaned), as well as inference demos.
- It is the fastest model in the west!
- The model with LoRA weights merged-in available at [HuggingFace/lxe/Cerebras-GPT-2.7B-Alpaca-SP](https://huggingface.co/lxe/Cerebras-GPT-2.7B-Alpaca-SP)
- The LoRA weights also available at [HuggingFace/lxe/lora-cerebras-gpt2.7b-alpaca-shortprompt](https://huggingface.co/lxe/lora-cerebras-gpt2.7b-alpaca-shortprompt)
- [ggml](https://github.com/ggerganov/ggml) version of the model available at [HuggingFace/lxe/ggml-cerebras-gpt2.7b-alpaca-shortprompt](https://huggingface.co/lxe/Cerebras-GPT-2.7B-Alpaca-SP-ggml). You can run this without a GPU and it's much faster than the original model
### 📈 Warnings
The model tends to be pretty coherent, but it also hallucinates a lot of factually incorrect responses. Avoid using it for anything requiring factual correctness.
### 📚 Instructions
0. Be on a machine with an NVIDIA card with 12-24 GB of VRAM.
1. Get the environment ready
```bash
conda create -n cerberas-lora python=3.10
conda activate cerberas-lora
conda install -y cuda -c nvidia/label/cuda-11.7.0
conda install -y pytorch=1.13.1 pytorch-cuda=11.7 -c pytorch
```
2. Clone the repo and install requirements
```
git clone https://github.com/lxe/cerebras-lora-alpaca.git && cd !!
pip install -r requirements.txt
```
3. Run the inference demo
```
python app.py
```
To reproduce the finetuning results, do the following:
3. Install jupyter and run it
```
pip install jupyter
jupyter notebook
```
4. Navigate to the `inference.ipynb` notebook and test out the inference demo.
5. Navigate to the `finetune.ipynb` notebook and reproduce the finetuning results.
- It takes about 5 hours with the default settings
- Adjust the batch size and gradient accumulation steps to fit your GPU
### 📝 License
Apache 2.0