Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/hanieas/fastapi-file-management-service
This microservice is designed to manage all file-related tasks.
https://github.com/hanieas/fastapi-file-management-service
alembic celery docker docker-compose fastapi minio mysql rabbitmq sqlalchemy
Last synced: 18 days ago
JSON representation
This microservice is designed to manage all file-related tasks.
- Host: GitHub
- URL: https://github.com/hanieas/fastapi-file-management-service
- Owner: hanieas
- Created: 2024-07-24T13:03:18.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-08-16T14:15:19.000Z (6 months ago)
- Last Synced: 2024-11-18T08:17:26.780Z (3 months ago)
- Topics: alembic, celery, docker, docker-compose, fastapi, minio, mysql, rabbitmq, sqlalchemy
- Language: Python
- Homepage:
- Size: 94.7 KB
- Stars: 10
- Watchers: 1
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# File Management Service
[](https://codecov.io/github/hanieas/fastapi-file-management-service)
## Table of Contents
1. [Introduction](#introduction)
2. [Technology Stack and Features](#technology-stack-and-features)
3. [Why a Separate File Management Service?](#why-a-separate-file-management-service)
4. [How to Use it?](#how-to-use-it)
5. [API Endpoints](#api-endpoints)
6. [Contributing](#Contributing)
## Introduction
This microservice is designed to manage all file-related tasks. It uses **MinIO** for object storage and **MySQL** for managing file metadata. We support chunk uploads for handling large files efficiently, with **Celery** running background tasks to ensure smooth performance.
## Technology Stack and Features
- ⚡ [**FastAPI**](https://fastapi.tiangolo.com) for the Python backend API.
- 🧰 [SQLAlchemy](https://www.sqlalchemy.org/) for the Python SQL database interactions (ORM).
- 🔍 [Pydantic](https://docs.pydantic.dev), used by FastAPI, for the data validation and settings management.
- 🗄️ [MYSQL](https://www.mysql.com/) as the SQL database.
- 🔄 [Alembic](https://alembic.sqlalchemy.org/en/latest) for database migrations.
- 🔧 [Celery](https://docs.celeryq.dev/en/stable/) with [RabbitMQ](https://www.rabbitmq.com/) for task queue management and background processing.
- 💾 [MinIO](https://min.io/) for scalable object storage with chunk upload support.
- ✅ [Pytest](https://pytest.org) for testing to ensure code reliability and functionality.
- 🐋 [Docker Compose](https://www.docker.com) for development and production.
## Why a Separate File Management Service?
1. Centralizes file operations, making management and maintenance easier.
2. Enables scaling file handling independently of other services.
3. Simplifies updates and changes to file handling without impacting other parts of the system.
4. Reduces code duplication by keeping file upload and retrieval logic in one place, resulting in cleaner code.
## How to Use it?
1. **Complete the `.env` File**:
- Copy the contents of `.env.example` to a new file named `.env`.
- Fill in the required environment variables based on your setup.
2. **Build the Docker Image**:
- Run the following command to build the Docker image:
```bash
docker compose build
```
3. **Run the Containers**:
- After the build is complete, start the containers in detached mode with:
```bash
docker compose up -d
```
4. **Migrate the Database**:
- Access the running container to perform the database migration:
```bash
docker compose exec filemanager bash
```
- Inside the container, run the migration using Alembic:
```bash
alembic upgrade head
```
5. **Access the Service**:
- The project is now up and running, accessible on port `8000`.
- You can access the project documentation by navigating to `/docs` on your browser.
## API Endpoints
Here’s a quick reference guide to the available API endpoints, their methods, and what they do:
| Method | URL | Description |
|--------|---------------------------------------------|------------------------------------------------------------------|
| POST | `/api/v1/file/upload/init/` | Initialize a new file upload session. |
| POST | `/api/v1/file/upload/chunk/` | Upload a file chunk. |
| POST | `/api/v1/file/upload/complete/` | Complete the file upload process. |
| GET | `/api/v1/file/get/{file_id}` | Retrieve a file by its ID. |
| GET | `/api/v1/file/status/{file_id}` | Check the upload status of a file. |
| POST | `/api/v1/file/upload/retry` | Retry uploading a file. |
A Postman collection export is also available for testing these endpoints. You can import it into Postman to quickly get started with API testing.
## Contributing
We welcome contributions from everyone! If you have ideas for improvements, new features, or bug fixes, feel free to contribute to this project. Here's how you can get involved:
1. **Create an Issue**:
- If you find a bug, have a question, or want to suggest a feature, please open an issue. This helps us track and discuss your ideas.
2. **Send a Pull Request (PR)**:
- Fork the repository, make your changes in a new branch, and then create a pull request.
- Please make sure your code follows the project's coding standards and passes all tests.
We appreciate your contributions and will do our best to review and merge your pull requests promptly. Thank you for helping us improve this project!