https://github.com/fkcptlst/labtasker
Experiment task scheduling made easy.
https://github.com/fkcptlst/labtasker
machine-learning mongodb-database python3 task-manager task-queue task-scheduler workflow-automation
Last synced: 3 months ago
JSON representation
Experiment task scheduling made easy.
- Host: GitHub
- URL: https://github.com/fkcptlst/labtasker
- Owner: fkcptlst
- License: apache-2.0
- Created: 2024-11-01T03:06:40.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2025-03-22T17:42:05.000Z (3 months ago)
- Last Synced: 2025-03-22T18:33:38.454Z (3 months ago)
- Topics: machine-learning, mongodb-database, python3, task-manager, task-queue, task-scheduler, workflow-automation
- Language: Python
- Homepage: https://fkcptlst.github.io/labtasker/dev
- Size: 4.67 MB
- Stars: 20
- Watchers: 2
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Make your ML experiment wrapper scripts smarter with...
Labtasker
Install • Tutorial / Demo • Documentation • FAQs • Releases
** 🌟 Labtasker makes ML experiment wrapper scripts smarter with task prioritization,
failure handling, halfway resume and more: just change 1 line of code.**
If you like our project, please give us a star ⭐ on GitHub for latest update.
## ✨ When and Where to Use
**TLDR**: Replace `for` loops in your experiment *wrapper script* with labtasker to enable features like experiment
parallelization, dynamic task prioritization, failure handling, halfway resume, and more.
🐳 For detailed examples and concepts, check out the [documentation](https://fkcptlst.github.io/labtasker/).
## 🧪️ A Quick Demo
This demo shows how to easily submit task arguments and run jobs in parallel.
It also features an event listener to monitor task execution in real-time and automate workflows,
such as sending emails on task failure.
For more detailed steps, please refer to the content in the [Tutorial / Demo](https://fkcptlst.github.io/labtasker/latest/guide/basic/).
## ⚡️ Features
- ⚙️ Easy configuration and setup.
- 🧩 Versatile and minimalistic design.
- 🔄 Supports both CLI and Python API for task scheduling.
- 🔌 Customizable plugin system.
## 🔮 Supercharge Your ML Experiments with Labtasker
- ⚡️ **Effortless Parallelization:** Distribute tasks across multiple GPU workers with just a few lines of code.
- 🛡️ **Intelligent Failure Management:** Automatically capture exceptions, retry failed tasks, and maintain detailed error logs.
- 🔄 **Seamless Recovery:** Resume failed experiments with a single command - no more scavenging through logs or directories.
- 🎯 **Real-time Prioritization:** Changed your mind about experiment settings? Instantly cancel, add, or reschedule tasks without disrupting existing ones.
- 🤖 **Workflow Automation:** Set up smart event triggers for email notifications or task workflow based on FSM transition events.
- 📊 **Streamlined Logging:** All stdout/stderr automatically organized in `.labtasker/logs` - zero configuration required.
- 🧩 **Extensible Plugin System:** Create custom command combinations or leverage community plugins to extend functionality.
## 🛠️ Installation
> [!NOTE]
> You need a running Labtasker server to use the client tools.
> Simply use the installed Python CLI `labtasker-server serve` or use docker-compose to deploy the server.
> See [deployment instructions](https://fkcptlst.github.io/labtasker/latest/install/deployment/).
### 1. Install via PyPI
```bash
pip install labtasker
```
### 2. Install the Latest Version from GitHub
```bash
pip install git+https://github.com/fkcptlst/labtasker.git
```
## 🚀 Quick Start
Use the following command to launch a labtasker server in the background:
```bash
labtasker-server serve &
```
Use the following command to quickly setup a labtasker queue for your project:
```bash
labtasker init
```
Then, use `labtasker submit` to submit tasks and use `labtasker loop` to run tasks across any number of workers.
## 📚 Documentation
For detailed information on demo, tutorial, deployment, usage, please refer to
the [documentation](https://fkcptlst.github.io/labtasker/).
## 🔒 License
See [LICENSE](LICENSE) for details.