Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/simplifine-llm/Simplifine
Simplifine: 🚀 Super easy LLM finetuning with one-line commands, cutting-edge optimization, and fully open source! ✨
https://github.com/simplifine-llm/Simplifine
Last synced: about 1 month ago
JSON representation
Simplifine: 🚀 Super easy LLM finetuning with one-line commands, cutting-edge optimization, and fully open source! ✨
- Host: GitHub
- URL: https://github.com/simplifine-llm/Simplifine
- Owner: simplifine-llm
- License: gpl-3.0
- Created: 2024-07-05T03:38:41.000Z (3 months ago)
- Default Branch: main
- Last Pushed: 2024-08-12T14:32:21.000Z (about 1 month ago)
- Last Synced: 2024-08-12T15:40:55.339Z (about 1 month ago)
- Language: Python
- Homepage: https://www.simplifine.com
- Size: 792 KB
- Stars: 61
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-LLM-resourses - Simplifine
README
# 🌟 Simplifine 🌟
Simplifine lets you invoke LLM finetuning with just one line of code using any Hugging Face dataset or model.
> The easiest, fully open-source LLM finetuning library!
**Get free Simplifine Cloud Credits to finetune [here](https://www.simplifine.com/api-key-interest)**
## Roadmap
- **COMPREHENSIVE UPDATE of DOCUMENTATIONS on INCOMING (By Aug 9th, 2024) to match new config files.
## 🔄 Updates
**v0.0.8 (2024-08-08)**
- **Bug Fixes:** Code clean up and trainer fixes.
- **New Feature:** Ability to define more complex configuration files for the trainer.
- **Examples:** -New examples on training cloud and training a fake news detector.
**v0.0.71 (2024-07-25)**
- **Bug Fixes:** Resolved issues that prevented the library from loading on certain configurations.
- **New Feature:** Added support for installing directly from git. Added support for Hugging Face API Tokens to access restricted models.
- **Documentation:** Updated examples.
## 🚀 Features
- **Supervised Fine Tuning** 🧑🏫
- **Question-Answer Finetuning** ❓➕
- **Contrastive Loss for Embedding Tasks** 🌌
- **Multi-label Classification Finetuning** 🏷️
- **WandB Logging** 📊
- **In-built Evaluation Tools** 📈
- **Automated Finetuning Parameters** 🤖
- **State-of-the-art Optimization Techniques (DeepSpeed, FDSP)** 🏎️
## 📦 Installation
```bash
pip install simplifine-alpha
```
Or you can install the package from source. To do so, simply download the content of this repository and navigate to the installation folder and run the following command:
```bash
pip install .
```
You can also directly install from github using the following command:
```bash
pip install git+https://github.com/simplifine-llm/Simplifine.git
```
## 🏁 Quickstart
For a more comprehensive example, see this [notebook](https://github.com/simplifine-llm/Simplifine/blob/main/examples/cloud_quickstart.ipynb) in the examples folder:
Further examples on how to use train engine are also located in the examples folder.
## 🤝 Contributing
We are looking for contributors! Please send an email to [[email protected]](mailto:[email protected]) to get onboarded, or add your name to the waitlist on [www.simplifine.com](http://www.simplifine.com).
## 📄 License
Simplifine is licensed under the GNU General Public License Version 3. See the LICENSE file for more details.
## 📚 Documentation
MAJOR OVERHAUL OF DOCUMENTATION IN THE WORKS (Done by 11th Aug 2024). In the meantime, please use this [notebook](https://github.com/simplifine-llm/Simplifine/blob/main/examples/cloud_quickstart.ipynb) here to learn how to use the model.
## 💬 Support
If you have any suggestions for new features you'd like to see implemented, please raise an issue—we will work hard to make it happen ASAP! For any other questions, feel free to contact us at [[email protected]](mailto:[email protected]).
## ⛮ General Compute Considerations
We currently support both DistributedDataParallel (DDP) and ZeRO from DeepSpeed.
**TL;DR**:
- **DDP** is useful when a model can fit in GPU memory (this includes gradients and activation states).
- **ZeRO** is useful when a model requires sharding across multiple GPUs.
**Longer Version**:
- **DDP**: Distributed Data Parallel (DDP) creates a replica of the model on each processor (GPU). For example, imagine 8 GPUs, each being fed a single data point—this would make a batch size of 8. The model replicas are then updated on each device. DDP speeds up training by parallelizing the data-feeding process. However, DDP **fails** if the replica cannot fit in GPU memory. Remember, the memory not only hosts parameters but also gradients and optimizer states.
- **ZeRO**: ZeRO is a powerful optimization developed by DeepSpeed and comes in different stages (1, 2, and 3). Each stage shards different parts of the training process (parameters, gradients, and activation states). This is really useful if a model cannot fit in GPU memory. ZeRO also supports offloading to the CPU, making even more room for training larger models.
### Example Scenarios and Appropriate Optimization Methods:
1. **LLaMA-3-8b model with 16-bit precision**: Use ZeRO Stage 3 on 8 A100s.
2. **LLaMA-3-8b model with LoRA adapters**: Usually fine with DDP on A100s.
3. **GPT-2 with 16-bit precision**: Use DDP.
## 🪲 FAQs and Bugs
**Issue: RuntimeError: Error building extension 'cpu_adam' python dev**
This error occurs when `python-dev` is not installed, and ZeRO is using offload. To resolve this, try:
```bash
# Try sudo apt-get install python3-dev if the following fails.
apt-get install python-dev # for Python 2.x installs
apt-get install python3-dev # for Python 3.x installs
```
See this [link](https://stackoverflow.com/questions/21530577/fatal-error-python-h-no-such-file-or-directory)