Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/rschand01/file-upload-microservice

backend bullmq expressjs firebase javascript microservice multer nodejs redis

Last synced: 15 days ago
JSON representation

Host: GitHub
URL: https://github.com/rschand01/file-upload-microservice
Owner: rschand01
License: gpl-3.0
Created: 2025-01-20T10:09:32.000Z (22 days ago)
Default Branch: main
Last Pushed: 2025-01-20T10:13:20.000Z (22 days ago)
Last Synced: 2025-01-20T11:24:57.337Z (22 days ago)
Topics: backend, bullmq, expressjs, firebase, javascript, microservice, multer, nodejs, redis
Language: JavaScript
Homepage: https://github.com/rschand01/file-upload-microservice
Size: 132 KB
Stars: 1
Watchers: 1
Forks: 0
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md

Awesome Lists containing this project

README

# File Upload Microservice - Backend

## Architecture: Microservice

This project is a robust File Upload Microservice backend built with Node.js, Express, Redis, Bull, and Firebase. It includes comprehensive features for managing file uploads, queuing, and rate limiting, and is designed to handle large-scale file processing efficiently. The system integrates Firebase for cloud storage, Bull for job queuing, and Redis for rate limiting, demonstrating its scalability and production readiness. Additionally, it incorporates advanced error handling and logging to ensure reliability and maintainability.

## Table of Contents

- [Features](#features)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
- [Project Structure](#project-structure)
- [API Endpoints](#api-endpoints)
- [Contributing](#contributing)
- [License](#license)

## Features

- **File Upload Management**: Handles file uploads with support for various file types and integrates with Firebase for cloud storage.
- **Job Queuing**: Utilizes Bull for efficient job queuing and processing of file uploads.
- **Rate Limiting**: Implements rate limiting using Express Rate Limit and Redis to protect API endpoints.
- **Robust Error Handling**: Advanced error handling mechanisms with Winston for comprehensive logging and monitoring.
- **Directory Management**: Automatic directory creation and management for handling file storage.
- **Scalability**: Designed for scalability with integration options for message brokers like KafkaJS for future enhancements.
- JOI for error handling

## Installation

1. **Clone the repository:**

```bash
git clone https://github.com/rschand-dev/file-upload-microservice.git
cd file-upload-microservice
```

2. **Install dependencies:**

```bash
npm install
```

3. **Set up environment variables:**

Create a `.env` file in the root directory and configure the following variables:

```env
# Example: EXPRESS_PORT=3001
EXPRESS_PORT=PORT_NUMBER

# Example: WINSTON_SERVICE="nms-backend-microservice"
WINSTON_SERVICE=SERVICE_NAME

# Example: NODE_ENV=development
# Example: NODE_ENV=production
NODE_ENV=ENVIRONMENT

# Example: REDIS_CLIENT_PASSWORD=enteryourredispasswordhere
REDIS_CLIENT_PASSWORD=PASSWORD

# Example: REDIS_CLIENT_HOST=enteryourredishosthere
REDIS_CLIENT_HOST=HOST

# Example: REDIS_CLIENT_PORT=30982
REDIS_CLIENT_PORT=PORT_NUMBER

# Example: CORS_ORIGIN=http://localhost:3001
CORS_ORIGIN=ALLOWED_ORIGIN

# Example: FIREBASE_API_KEY=yoursecretapikey
FIREBASE_API_KEY=API_KEY
```

4. **Setup Redis**

- **Create a Redis Account & connect to the redis cloud.**

5. **Start the server:**

```bash
npm run start
```

## Configuration

The project uses environment variables for configuration. Refer to the [`.env.example`](.env.example) file for the required variables. Make sure to set these variables in your `.env` file.

## Usage

### Starting the Server

To start the server in development mode:

```bash
npm run serve
```

If all goes well, then you should have something similar in your terminal. Refer Below:

```powershell
npm run serve

> [email protected] preserve
> npx eslint .

> [email protected] serve
> nodemon ./server.mjs

[nodemon] 3.1.4
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,cjs,json
[nodemon] starting `node ./server.mjs`
info: Successfully initialized the Firebase App. {"additional":"projectId: file-upload-microservice","service":"fms","timestamp":"2024-08-17 02:31:47"}
heartbeats undefined
info: Connection to Redis Data Store have been established! (Bull.js) {"service":"fms","timestamp":"2024-08-17 02:31:47"}
info: Express Server is successfully listening! {"additional":"port: 3001","service":"fms","timestamp":"2024-08-17 02:31:47"}
info: Connection to Redis Data Store have been established! {"additional":"port: 19314","service":"fms","timestamp":"2024-08-17 02:31:48"}
```

## API Documentation

Use a tool like Postman, Thunder Client extension in Visual Studio Code or Insomnia to test the API endpoints. Refer to the API Endpoints section for a list of available routes.

## Project Structure

Please click on [PROJECT_STRUCTURE](PROJECT_STRUCTURE) to view the project structure.

Alternatively you can find the `PROJECT_STRUCTURE` file in the root directory of this repository and or project folder.

## API Endpoints

### File Upload

- **POST** /upload - Allows for file upload to the device

## Contributing

Contributions are welcome! Please follow these steps to contribute:

1. Fork the repository
2. Create a new branch (git checkout -b feature-branch)
3. Make your changes
4. Commit your changes (git commit -m 'Add some feature')
5. Push to the branch (git push origin feature-branch)
6. Open a pull request

## License

This project is licensed under the [`GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007`](LICENSE)

See the [License](LICENSE) for details.

## Copyright

**Repository:** [`file-upload-microservice`](https://github.com/rschand-dev/file-upload-microservice)

---

Designed & Developed with 😍 💝🌺

**Kind Regards**

Reginald Chand