https://github.com/sandy1990418/flask-celery-template
A Flask-Celery integration template for distributed task processing with progress tracking and use MMMLU to evaluation example.
https://github.com/sandy1990418/flask-celery-template
async celery evalution flask jinja2-templates
Last synced: about 2 months ago
JSON representation
A Flask-Celery integration template for distributed task processing with progress tracking and use MMMLU to evaluation example.
- Host: GitHub
- URL: https://github.com/sandy1990418/flask-celery-template
- Owner: sandy1990418
- License: mit
- Created: 2024-11-01T08:36:03.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-11-25T08:40:51.000Z (over 1 year ago)
- Last Synced: 2025-01-26T02:21:08.362Z (over 1 year ago)
- Topics: async, celery, evalution, flask, jinja2-templates
- Language: Python
- Homepage:
- Size: 2.19 MB
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Flask-Celery-System-Template
A project template integrating Flask and Celery for handling asynchronous tasks and background jobs, with a focus on task progress tracking. Currently implemented with MMMLU (Massive Multitask Language Understanding) benchmark evaluation capabilities.
## 🌟 Features
- 🚀 Built with Flask + Celery + RabbitMQ for reliable distributed processing
- 📊 Task execution tracking with detailed status updates
## 📁 Project Structure
```bash
.
├── src/
│ ├── celeryflow/ # Celery task management and processing
│ ├── models/ # Database models and operations
│ ├── routes/ # API routes and endpoints
│ ├── templates/ # HTML templates
│ └── utils/ # Utility functions and helpers
├── db/ # Database initialization and drivers
├── docker/ # Docker and deployment configurations
├── app_run.py # Flask Application python script
├── requirements.txt # Project dependencies
└── config.yaml # Application configuration
```
## 📦 Requirements
- Python 3.8+
## 🛠️ Installation
1. Clone the repository:
```bash
git clone https://github.com/yourusername/flask-celery-template.git
cd flask-celery-template
```
2. Create a virtual environment:
```bash
python3 -m venv venv
source venv/bin/activate # On Windows use `venv\Scripts\activate`
```
3. Install the dependencies:
```bash
pip install -r requirements.txt
```
## 🚀 Usage
### Setting up RabbitMQ
```bash
# Pull RabbitMQ image (first time only)
docker pulll rabbitmq:management
# Start RabbitMQ container
docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:management
```
### Starting the System
1. Start the Flask application:
```bash
python app_run.py
```
2. Start the Celery worker:
```bash
# Start Celery worker with purge option
celery -A app_run.celery_app worker
# Start Flower for Celery monitoring
celery -A app_run.celery_app flower
```
Celery command options explained:
`-A`: Specify the Celery app location
`-Q`: Specify which queues this worker should listen to
`-E`: Enable event tracking (logs task execution events)
`--pool=solo`: Run in single-thread mode
`--purge`: Clear all queued tasks before starting
`--loglevel=info`: Set logging level to info
3. Access the interfaces:
- Web Application: `http://localhost:5000`
- RabbitMQ Management: `http://localhost:15672` (default credentials: guest/guest)
- Flower Dashboard: `http://localhost:5555`
## 🐳 Usage with Docker Compose
### Quick Start
```bash
# Start all services
docker compose -f ./docker/docker-compose.yaml up
# Start in detached mode
docker compose -f ./docker/docker-compose.yaml up -d
```
## 🙏 Acknowledgments
- OpenAI for the MMMLU benchmark
- Flask and Celery communities for their excellent frameworks
- Claude 3.5 Sonnet for assisting with the fronted
## License
This project is licensed under the MIT License.