https://github.com/sioaeko/hfmodeldownloader
🤗 A Python script for efficiently downloading and reconstructing large Hugging Face model files by splitting them into manageable chunks
https://github.com/sioaeko/hfmodeldownloader
downloader hugging-face huggingface huggingface-models huggingface-transformers
Last synced: about 1 month ago
JSON representation
🤗 A Python script for efficiently downloading and reconstructing large Hugging Face model files by splitting them into manageable chunks
- Host: GitHub
- URL: https://github.com/sioaeko/hfmodeldownloader
- Owner: sioaeko
- License: mit
- Created: 2024-07-11T14:25:30.000Z (10 months ago)
- Default Branch: main
- Last Pushed: 2024-07-23T14:52:51.000Z (10 months ago)
- Last Synced: 2025-03-21T08:51:16.709Z (about 2 months ago)
- Topics: downloader, hugging-face, huggingface, huggingface-models, huggingface-transformers
- Language: Python
- Homepage:
- Size: 14.6 KB
- Stars: 3
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 🤗 Hugging Face Model Downloader
A Python script for efficiently downloading and reconstructing large Hugging Face model files by splitting them into manageable chunks.
## ✨ Features
- 🚀 Downloads large model files from Hugging Face in multiple parts simultaneously
- 🔗 Automatically extracts download links from the model page
- 🔧 Allows customization of the number of parts for splitting files
- 🧩 Combines downloaded parts back into the original file
- 📊 Displays download progress for each file
## 🛠 Requirements
- Python 3.6+
- `requests` library
- `tqdm` library
## 📥 Installation
1. Clone this repository:
```
git clone https://github.com/sioaeko/huggingface-split-downloader.git
cd huggingface-split-downloader
```
2. Install the required dependencies:
```
pip install requests tqdm
```
## 🚀 Usage
Run the script with the following command:
```
python huggingface_split_downloader.py --parts
```
### Arguments:
- ``: The URL of the Hugging Face model page
- ``: The directory where you want to save the downloaded files
- `--parts`: (Optional) The number of parts to split each file into (default is 5)
### Example:
```
python huggingface_split_downloader.py https://huggingface.co/gpt2 ./downloaded_model --parts 10
```
This command will download the GPT-2 model files, splitting each file into 10 parts, and save them in the `./downloaded_model` directory.
## 🔧 How it works
1. 🔍 Access the provided Hugging Face model page and extract download links
2. For each file:
- 📏 Determine the file size
- ✂️ Split the download into the specified number of parts
- 📥 Download each part concurrently
- 📊 Show a progress bar for the download
3. 🧩 Combine parts back into the original file
4. 🗑️ Delete partial files after successful combination
## 📝 Notes
- 💾 Ensure you have sufficient storage space for the model files
- 🌐 Download speed may vary depending on your internet connection
- 🔄 The script may need adjustments if the structure of the Hugging Face website changes
## 🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## 📄 License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## ⚠️ Disclaimer
This tool is for educational and research purposes. Always ensure you have the right to download and use the models as per Hugging Face's terms of service.