{"id":25442265,"url":"https://github.com/arvind-4/image-compressor","last_synced_at":"2026-04-11T10:36:08.822Z","repository":{"id":248686389,"uuid":"829401164","full_name":"arvind-4/image-compressor","owner":"arvind-4","description":"Image Compressor using FastAPI, MongoDB, Minio, Docker and Celery","archived":false,"fork":false,"pushed_at":"2024-07-17T07:02:24.000Z","size":305,"stargazers_count":0,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-01-03T16:10:08.680Z","etag":null,"topics":["async","async-python","background-tasks","celery","docker","docker-compose","fastapi","minio","mongodb","python"],"latest_commit_sha":null,"homepage":"https://github.com/Arvind-4/image-compressor","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/arvind-4.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-16T11:00:31.000Z","updated_at":"2024-07-17T07:02:27.000Z","dependencies_parsed_at":"2025-05-15T19:10:38.112Z","dependency_job_id":"2a96ed1f-db99-472d-b615-4975aacc6aa4","html_url":"https://github.com/arvind-4/image-compressor","commit_stats":null,"previous_names":["arvind-4/image-compressor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/arvind-4/image-compressor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvind-4%2Fimage-compressor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvind-4%2Fimage-compressor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvind-4%2Fimage-compressor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvind-4%2Fimage-compressor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/arvind-4","download_url":"https://codeload.github.com/arvind-4/image-compressor/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/arvind-4%2Fimage-compressor/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31677819,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-11T08:18:19.405Z","status":"ssl_error","status_checked_at":"2026-04-11T08:17:08.892Z","response_time":54,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","async-python","background-tasks","celery","docker","docker-compose","fastapi","minio","mongodb","python"],"created_at":"2025-02-17T13:17:51.309Z","updated_at":"2026-04-11T10:36:08.796Z","avatar_url":"https://github.com/arvind-4.png","language":"Python","readme":"\u003ch2 align=\"center\"\u003eImage Compression Service Overview\u003c/h2\u003e\n\n### Overview:\n\nThis document describes the architecture, components, and usage of an image compression service built with FastAPI, MongoDB, Celery, Docker, and MinIO.\n\nThis document outlines the design and functionality of an image compression service built using FastAPI, MongoDB, Celery, Docker, and MinIO. The service offers a robust platform for uploading CSV files containing image data, initiating asynchronous image compression tasks, checking task statuses, and downloading compressed images.\n\n### Architecture Diagram:\n\n\u003cimage src=\"https://github.com/Arvind-4/image-compressor/blob/main/.gitbook/diagram.png\" alt=\"Architecture Diagram\" /\u003e\n\n### Quick Start:\n\nRun the application using docker. The services running are:\n\n- **API Service**: Exposed on port 8000, accessible via http://0.0.0.0:8000.\n- **MongoDB**: Exposed on port 27017, accessible via mongodb://0.0.0.0:27017.\n- **Mongo Express**: Exposed on port 8081, accessible via http://0.0.0.0:8081 for MongoDB management. (Auth credentials are **username** is **admin** and **password** is **password**)\n- **Redis**: Exposed on port 6379, accessible via redis://0.0.0.0:6379.\n- **MinIO**:\n  - Main server exposed on port 9000, accessible via http://0.0.0.0:9000.\n  - Web-based management console exposed on port 9001, accessible via http://0.0.0.0:9001. (Auth credentials are **username** is **admin** and **password** is **password**)\n\nThese ports and services together form the architecture for the image compression service, providing database storage, task management, object storage, and web interfaces for administration and API access.\n\n```bash\ncd ~/Dev\nmkdir image-compressor\ncd ~/Dev/image-compressor\ngit clone https://github.com/Arvind-4/image-compressor.git .\ndocker compose up --build\n```\n\n### Manual Setup:\n\n- Create a Directory and virtualenv:\n\n```bash\ncd ~/Dev\nmkdir image-compressor\ncd ~/Dev/image-compressor\ngit clone https://github.com/Arvind-4/image-compressor.git .\n```\n\n- Install using `requirements.txt` file or `poetry`:\n\n```bash\npip install -r requirements.txt -r requirements-dev.txt\n```\n\n\u003ccenter\u003e\n\u003cb\u003eor\u003c/b\u003e\n\u003c/center\u003e\n\n```bash\npoetry install\n```\n\n- Create a `.env` file :\n\n```bash\ncp .env.sample .env\n```\n\n**Change values accordingly in env file**\n\n- Run mongodb, mongodb admin and minio:\n\n```bash\ndocker compose -f compose.service.yaml up --build -d\n```\n\n- Set env:\n\n```bash\nset -a\nsource .env.docker\nsource .env.example\nsource .env\nset +a\n```\n\n- Run fastapi:\n\n```bash\nbash commands/run.sh\n```\n\n- Run celery:\n\n```bash\nbash commands/background-task.sh\n```\n\n### File Structure:\n\n```bash\n.\n├── api.Dockerfile # Docker file for fastapi\n├── app\n│   ├── celery_app.py # Celery app\n│   ├── config.py # Config and env variables for the project\n│   ├── db.py # Mongo db connection file\n│   ├── __init__.py # Module file\n│   ├── main.py # Entrypoint file for our application\n│   ├── routes.py # Routes for the application\n│   ├── schemas.py # Schema and validations for data\n│   ├── services.py # Mongo DB CRUD operations\n│   ├── storage.py # Minio client file\n│   ├── tasks.py # Celery worker file\n│   └── utils.py # functions performed on data\n│   └── webhooks.py # webhook url for the repo\n├── celery.Dockerfile # Docker file for celery\n├── commands\n│   ├── background-task.sh # Celery command\n│   ├── entrypoint.sh # Fastapi production entrypoint\n│   ├── export.sh # Export poetry dependencies\n│   └── run.sh # Run dev server\n├── compose.service.yaml # Service only docker file\n├── compose.yaml # Docker file for all services\n├── .env.docker # Env file for docker\n├── .env.sample # Env file for reference\n├── .gitbook\n│   └── diagram.png # Architecture diagram\n├── LICENSE # LICENSE file forr the project\n├── poetry.lock # Poetry lock file\n├── pyproject.toml # Poetry file\n├── .python-version # Python version\n├── pyvenv.cfg # Python venv file\n├── requirements-dev.txt # Python dev  requirements\n├── requirements.txt # Python requirements file\n├── ruff.toml # Format and lint files\n├── sample-data\n│   └── sheet.csv # Sample csv file for testing and reference\n└── set-env.txt # COmmand to set and load env variables\n\n4 directories, 33 files\n```\n\n### Components:\n\n**FastAPI** serves as the core framework for handling API requests, ensuring efficient and scalable endpoints for uploading CSV files (`/upload/`), checking task statuses (`/status/{request_id}`), and downloading compressed images (`/download/{request_id}`).\n\n**MongoDB** is utilized to persist image data and track task statuses. The `image_requests` collection stores details of uploaded image data and their processing statuses.\n\n**Celery** manages asynchronous image compression tasks (`process_image_task`). This task downloads original images, compresses them with user-defined quality, uploads both original and compressed images to MinIO, and updates task statuses in MongoDB upon completion.\n\n**MinIO**, an object storage service, stores both original and compressed images in separate buckets (`original-images` and `compressed-images`).\n\n**Docker** is a containerization platform that simplifies the process of packaging applications and their dependencies into portable, self-sufficient containers. These containers can run consistently across different computing environments, ensuring reliable deployment and scalability of applications.\n\n### Usage:\n\nUsers interact with the service by uploading CSV files containing image data, monitoring task progress through status queries, and downloading compressed images once tasks are completed.\n\n#### Prerequisites:\n\n- Install Docker and Docker Compose.\n- Set up MongoDB instance.\n- Configure MinIO server.\n\n#### Build and Run:\n\n1. Clone the repository.\n2. Set up environment variables. Refer `.env.sample` file.\n3. Build and run Docker containers using `docker-compose up --build`.\n\n### Functionality:\n\nThe service allows users to upload CSV files containing image data, initiate image compression tasks asynchronously, check task status, and download compressed images.\n\n### API Endpoints:\n\n#### Upload CSV File:\n\n- **Endpoint**: `/api/upload/`\n- **Method**: POST\n- **Parameters**:\n  - `file`: CSV file containing image data.\n- **Functionality**:\n  - Parses CSV content.\n  - Validates and stores image data in MongoDB.\n  - Initiates image compression tasks using Celery.\n\n#### Check Task Status:\n\n- **Endpoint**: `/api/status/{request_id}`\n- **Method**: GET\n- **Parameters**:\n  - `request_id`: Unique identifier for the request.\n- **Functionality**:\n  - Retrieves the status of the image compression task from MongoDB.\n\n#### Download Compressed Images:\n\n- **Endpoint**: `/api/download/{request_id}`\n- **Method**: GET\n- **Parameters**:\n  - `request_id`: Unique identifier for the request.\n- **Functionality**:\n  - Checks if image compression task is completed.\n  - Converts compressed image data to CSV format and streams it for download.\n\n#### Docs:\n\n- **Endpoint**: `/docs`\n- **Method**: GET\n- **Functionality**:\n  - Automatically generated documentation using Swagger UI.\n  - Provides detailed API endpoint descriptions, request/response formats, and example usage scenarios.\n\nThis addition includes the `/docs` endpoint, which is automatically generated by FastAPI's built-in Swagger UI. It enhances the usability of the service by providing interactive API documentation directly accessible through a web browser, facilitating easier integration and usage for developers.\n\n### Celery Tasks:\n\n**`process_image_task`**\n\n- **Description**: Asynchronous task for image compression.\n- **Functionality**:\n  - Downloads original images from provided URLs.\n  - Compresses images using specified quality.\n  - Uploads original and compressed images to MinIO.\n  - Updates task status and Updated URL's in MongoDB upon completion.\n\n### Deployment and Integration:\n\nThe service is containerized using **Docker**, facilitating easy deployment and scalability. The Docker setup includes a Dockerfile defining dependencies and a Docker Compose configuration for orchestrating multiple containers (FastAPI, Celery, MongoDB, and MinIO).\n\n#### Docker Deployment:\n\n- **api.Dockerfile**: Defines the environment and dependencies required for the application for fastapi service.\n- **celery.Dockerfile**: Defines the environment and dependencies required for the application for background service.\n- **Docker Compose**: Orchestrates multiple Docker containers (FastAPI, Celery, MongoDB, MinIO).\n- **Environment Variables**: Configures service endpoints, credentials, and other settings. Refer `.env.sample` file.\n\n### Security Considerations:\n\nTo enhance security, the service implements authentication mechanisms for API endpoints, controls access to sensitive operations, and ensures data encryption for transmission and storage.\n\n- **Authentication**: Implement authentication mechanisms to secure API endpoints.\n- **Authorization**: Control access to sensitive operations like uploading and downloading images.\n- **Data Encryption**: Ensure data transmission and storage are encrypted.\n\n### Conclusion:\n\nThis document provides a comprehensive overview of the image compression service architecture, including its components, API functionality, Celery task management, integration with MongoDB and MinIO for data storage, Docker deployment setup, security measures, and user interaction scenarios. It serves as a guide for deploying, utilizing, and securing the image compression service effectively in various operational environments. Additionally, it covers API endpoints, usage instructions, and more.\n\n---\n\n\u003ch4 align=\"center\"\u003eEnd of Document\u003c/h4\u003e\n\n---\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farvind-4%2Fimage-compressor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Farvind-4%2Fimage-compressor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Farvind-4%2Fimage-compressor/lists"}