Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/hoangsonww/mongo-redis-rabbitmq-elk-stack

🚀 Dive into the world of Node.js, MongoDB, Redis, Postgres, RabbitMQ, and the ELK stack with this comprehensive sample backend project! Explore how these powerful technologies can be integrated to build robust, scalable, and efficient backend solutions.
https://github.com/hoangsonww/mongo-redis-rabbitmq-elk-stack

elasticsearch elk elk-stack kibana logstash mongo mongodb mongodb-database mongoose node nodejs postgres postgresql rabbitmq redis redis-cache

Last synced: about 1 month ago
JSON representation

Host: GitHub
URL: https://github.com/hoangsonww/mongo-redis-rabbitmq-elk-stack
Owner: hoangsonww
License: mit
Created: 2024-06-23T13:18:43.000Z (5 months ago)
Default Branch: master
Last Pushed: 2024-10-07T00:31:41.000Z (about 1 month ago)
Last Synced: 2024-10-10T05:40:39.376Z (about 1 month ago)
Topics: elasticsearch, elk, elk-stack, kibana, logstash, mongo, mongodb, mongodb-database, mongoose, node, nodejs, postgres, postgresql, rabbitmq, redis, redis-cache
Language: JavaScript
Homepage: https://hoangsonww.github.io/Mongo-Redis-RabbitMQ-ELK-Stack/
Size: 425 KB
Stars: 14
Watchers: 12
Forks: 11
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

# Node.js, MongoDB, Redis, Postgres, Kafka, ELK, and RabbitMQ Backend Project

## Table of Contents
- [Introduction](#introduction)
- [Features](#features)
- [MongoDB](#mongodb)
- [Redis-Mongo-Flow](#redis-mongo-flow)
- [RabbitMQ](#rabbitmq)
- [Apache Kafka](#apache-kafka)
- [Round-Robin Load Balancing Algorithm](#round-robin-load-balancing-algorithm)
- [ELK Stack](#elk-stack)
- [Project Structure](#project-structure)
- [Getting Started](#getting-started)
- [Special Notes](#special-notes)
- [Test Statuses](#test-statuses)
- [Recommended GUI Tools](#recommended-gui-tools)
- [License](#license)
- [Author](#author)

## Introduction
This project is a sample Node.js Backend project that demonstrates how to connect to **MongoDB, Redis, PostgreSQL, Kafka, and RabbitMQ.** It demonstrates how these services can be used and interact with each other in a Node.js application.

Additionally, the project demonstrates how to use the ELK stack (Elasticsearch, Logstash, Kibana) for logging, as well as how to implement a round-robin load balancing algorithm using Redis.

And we call this the **NMRPKER-Stack**! 🚀

## Features

### MongoDB
- **MongoDB**: The project connects to a MongoDB database and performs CRUD operations.
- **Aggregation**: The project demonstrates how to use MongoDB's aggregation framework to perform complex queries, such as:
- Grouping data by a field.
- Filtering data based on a condition (lookup, match, project, etc.)
- Sorting data.
- Unwinding arrays.

### Redis-Mongo-Flow
- **Redis-Mongo-Flow**: The project also demonstrates how to use Redis as a cache layer for MongoDB.
- This is done by first checking if the data exists in Redis.
- If it does, the data is retrieved from Redis.
- If it doesn't, the data is retrieved from MongoDB and stored in Redis.
- This is done to reduce the number of queries to the database, thereby reducing the load on the database.

### RabbitMQ
- **RabbitMQ**: The project also demonstrates how to connect to RabbitMQ and publish and consume messages.
- The project uses RabbitMQ to publish a message and consume it.
- The message is published to a queue and consumed by a consumer.
- This is done to demonstrate how to use RabbitMQ for asynchronous communication between services.

### Apache Kafka
- **Apache Kafka**: The project also demonstrates how to connect to Apache Kafka and produce and consume messages.
- The project uses Apache Kafka to produce a message and consume it.
- The message is produced to a topic and consumed by a consumer.
- This is done to demonstrate how to use Apache Kafka for real-time data streaming.

### Round-Robin Load Balancing Algorithm
- **Round-Robin Load Balancing Algorithm**: The project also demonstrates how to use Redis to implement round-robin load balancing.
- The project has two routes, `/api/test/route1` and `/api/test/route2`.
- The project uses Redis to store the number of requests to each route.
- The project uses round-robin load balancing to distribute the requests evenly between the two routes.

### ELK Stack
- **ELK Stack**: The project also demonstrates how to use the ELK stack (Elasticsearch, Logstash, Kibana) for logging.
- The project uses Logstash to parse the log messages and send them to Elasticsearch.
- Elasticsearch will then index the log messages for searching and analysis.
- (Optional) You can also enhance the project by using Kibana to visualize the log data.

Here is a table of the features demonstrated in this project:

| Feature | Description |
|--------------------------------------|------------------------------------------------------------------|
| MongoDB | Connect to MongoDB and perform CRUD operations. |
| Redis-Mongo-Flow | Use Redis as a cache layer for MongoDB. |
| RabbitMQ | Connect to RabbitMQ and publish and consume messages. |
| Apache Kafka | Connect to Apache Kafka and produce and consume messages. |
| Round-Robin Load Balancing Algorithm | Implement round-robin load balancing using Redis. |
| ELK Stack | Use the ELK stack (Elasticsearch, Logstash, Kibana) for logging. |
| PostgreSQL | Connect to PostgreSQL and perform CRUD operations. |
| Express.js | Use Express.js as the web server framework. |
| Node.js | Use Node.js as the runtime environment. |
| JavaScript | Use JavaScript as the programming language. |
| NPM | Use NPM as the package manager. |
| REST API | Use REST API for communication. |

## Project Structure

The project has the following structure:

```
node-mongo-redis-project
├── index.js # Main entry point for the project for testing the connections
├── config.js # Configuration file for the project
├── package.json # NPM package file
├── publish.js # Script to publish a message to RabbitMQ
├── README.md
├── apache-kafka
│ └── kafkaService.js # Core logic for Apache Kafka
├── round-robin
│ ├── index.js # Core logic for round-robin load balancing & for testing the algorithm
│ └── config.js # Configuration file for Redis
├── routes
│ └── test.js # Sample routes for the project
├── redis-mongo-flow
│ ├── app.js # Core logic of the Redis-Mongo flow
│ ├── config.js # Configuration file for Redis and MongoDB
│ ├── seed.js # Script to populate MongoDB with sample data
│ └── test.js # Script to test the Redis-Mongo flow
├── elk-stack
│ └── index.js # Core logic for logging using the ELK stack & testing the stack
└── postgresql
└── app.js # Core logic for PostgreSQL
└── config.js # Configuration file for PostgreSQL
```

## Getting Started
To **get started**, run the following commands:

1. Start the MongoDB service:
```bash
brew services start mongodb-community
```

2. Start the Redis service:
```bash
redis-server
```

3. Start the RabbitMQ service:
```bash
brew services start rabbitmq
```

4. Start the Apache Kafka service:
- First, navigate to the Kafka directory:
```bash
cd kafka_2.13-3.8.0
```
- Start the Zookeeper service:
```bash
bin/zookeeper-server-start.sh config/zookeeper.properties
```
- Start the Kafka service:
```bash
bin/kafka-server-start.sh config/server.properties
```

5. `cd` back into the project directory (fix the path if necessary):
```bash
cd node-mongo-redis-project
```

6. Start the Node.js server:
```bash
node index.js
```

7. (Optional) Test the RabbitMQ service by publishing a message (go to another terminal window):
```bash
node publish.js
```

Verify that the message is received by the consumer by going to the terminal that is running the Node.js server (the one you started the index.js script).
You should receive the following messages in the terminal:
```bash
Server running on port 5000
Visit http://localhost:5000/ to test the connection.
Redis Connected
MongoDB Connected
Redis Test: Hello from Redis
Kafka Producer and Consumer Connected
Aggregation result: [
{ _id: 'Frank', totalOrderValue: 60 },
{ _id: 'David', totalOrderValue: 100 },
{ _id: 'Charlie', totalOrderValue: 35 },
{ _id: 'Grace', totalOrderValue: 75 },
{ _id: 'Alice', totalOrderValue: 50 },
{ _id: 'Bob', totalOrderValue: 80 },
{ _id: 'Ivy', totalOrderValue: 55 },
{ _id: 'Eve', totalOrderValue: 45 },
{ _id: 'Helen', totalOrderValue: 90 }
]
Sent message to Kafka: Hello Kafka from Express!
{
topic: 'test-topic',
partition: 0,
value: 'Hello Kafka from Express!'
}
RabbitMQ Connected
[*] Waiting for messages in task_queue. To exit press CTRL+C
[x] Received 'This is a test message!'
```

8. Visit [http://localhost:5000/](http://localhost:5000/) to test the connections.
- You should see the message: **"Server is running! MongoDB, RabbitMQ, Kafka, and Redis connections established."** on the webpage.
- Also test the API routes by visiting the following URLs: [http://localhost:5000/api/test/route1](http://localhost:5000/api/test/route1) and [http://localhost:5000/api/test/route2](http://localhost:5000/api/test/route2).

9. (Optional) Test the Round-Robin Load Balancing Algorithm:
```bash
cd round-robin
node index.js
```

10. (Optional) Test the ELK Stack:
```bash
cd elk-stack
node index.js
```

## Special Notes
**Note:** Before you get started, be sure to have the following installed on your machine by running the following commands (the following instructions are for MacOS):

1. Install Homebrew:
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```

2. Install MongoDB:
```bash
brew tap mongodb/brew
brew install mongodb-community
```

3. Install Redis:
```bash
brew install redis
```

4. Install RabbitMQ:
```bash
brew install rabbitmq
```

5. Install Apache Kafka:
- Go to the [Apache Kafka website](https://kafka.apache.org/downloads) and download the latest version of Apache Kafka.
- Currently, the most stable version is: **Scala 2.13 version: kafka_2.13-3.8.0.tgz**.
- Extract the downloaded file to wherever you want to install Kafka on your machine.
- Alternatively, you can use this direct link in your terminal to install Kafka:
```bash
wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
```

6. Ensure Kafka is running:
- First, navigate to the Kafka directory:
```bash
cd kafka_2.13-3.8.0
```
- Start the Zookeeper service:
```bash
bin/zookeeper-server-start.sh config/zookeeper.properties
```
- Start the Kafka service:
```bash
bin/kafka-server-start.sh config/server.properties
```

7. Install Postgres:
```bash
brew install postgresql
```

Start the Postgres service:
```bash
brew services start postgresql
```

Create a new database named `davidnguyen` (or any name you prefer - as long as you change the database credentials in the `postgresql/config.js` file):
```bash
createdb davidnguyen
```

Connect to the database:
```bash
psql davidnguyen
```

9. Install Node.js:
```bash
brew install node
```

10. Install NPM:
```bash
brew install npm
```
Then `cd` into the project directory and run:
```bash
npm init -y
```

11. Install all the required packages:
```bash
npm install
```

12. For Windows, refer to the official installation guides for [MongoDB](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-windows/), [Redis](https://redis.io/download), [RabbitMQ](https://www.rabbitmq.com/download.html), [Apache Kafka](https://kafka.apache.org/quickstart), and [PostgreSQL](https://www.postgresql.org/download/windows/).

## Test Statuses
[![MongoDB](https://img.shields.io/badge/MongoDB-Passed-brightgreen)](https://www.mongodb.com/)
[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-Passed-brightgreen)](https://www.postgresql.org/)
[![Redis](https://img.shields.io/badge/Redis-Passed-brightgreen)](https://redis.io/)
[![RabbitMQ](https://img.shields.io/badge/RabbitMQ-Passed-brightgreen)](https://www.rabbitmq.com/)
[![Apache Kafka](https://img.shields.io/badge/Apache%20Kafka-Passed-brightgreen)](https://kafka.apache.org/)
[![ELK Stack](https://img.shields.io/badge/ELK%20Stack-Passed-brightgreen)](https://www.elastic.co/)
[![REST API](https://img.shields.io/badge/REST%20API-Passed-brightgreen)](https://restfulapi.net/)
[![Round-Robin Load Balancing Algorithm](https://img.shields.io/badge/Round--Robin%20Load%20Balancing%20Algorithm-Passed-brightgreen)](https://redis.io/)

## Recommended GUI Tools
- **MongoDB Compass**: A GUI tool for MongoDB that allows you to interact with your MongoDB databases.
- **RedisInsight**: A GUI tool for Redis that allows you to interact with your Redis databases.
- **RabbitMQ Management Plugin**: A plugin for RabbitMQ that provides a web-based management interface for RabbitMQ.
- **Kibana**: A GUI tool for Elasticsearch that allows you to visualize and analyze your data.
- **Postico**: A GUI tool for PostgreSQL that allows you to interact with your PostgreSQL databases.

## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

Feel free to use this project for your own learning purposes or as a reference for your own projects!

## Author
- **Son Nguyen** - [GitHub](https://github.com/hoangsonww)
- **Email**: [[email protected]](mailto:[email protected])

---

Thank you for checking out this demo backend project! 🚀 Feel free to reach out if you have any questions or feedback. 😊