An open API service indexing awesome lists of open source software.

https://github.com/wednesday-solutions/node-mongo-express

An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.
https://github.com/wednesday-solutions/node-mongo-express

authorization circuitbreaker express mongodb mongoose node-api node-i18n node-mongo-api node-mongo-express node-mongo-express-api nodejs oauth2 rbac rest-api sharding

Last synced: about 2 months ago
JSON representation

An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.

Awesome Lists containing this project

README

        







Node Mongo Express



An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.

---




Expert teams of digital product strategists, developers, and designers.









---

We’re always looking for people who value their work, so come and join us. We are hiring!

[![Node Mongo Express CI](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/ci.yml/badge.svg)](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/ci.yml)

[![Node Mongo Express CD](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/cd.yml/badge.svg)](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/cd.yml)

---








---

## Pre-requisites

- yarn
- docker

## Features

- Mongo support
- Docker support
- Rate limited APIs
- RBAC middleware using Auth0
- Sharding mongoDB collection support
- Paginated APIs
- Autogenerated APIs from mongoose models
- Built in slack alerting mechanism
- Suport for redis cache
- Support for aggregate caching
- Support for batch jobs in multi-container environment
- Support for circuit breakers
- Autogenerated swagger documentation
- Load testing using k6
- Support for i18n

## Running Load tests

- [Install](https://k6.io/docs/getting-started/installation/) k6
- Execute the following command: `k6 run __tests__/__load__/script.js`
## Build and run docker container locally

- docker-compose down
- docker-compose build
- docker-compose up

# Shard setup

Run the following script

```
./setup-shards/scripts/setup/base.sh
```

Take a look at [this](./setup-shards/README.md) to create shards and replica sets.

## Seeders

Run the following command to begin seeding

```
./seeders/seed.sh
```

## How to start

- cd `node-mongo-express`
- yarn
- ./setup-shards/scripts/setup/base.sh
- cp .env.example .env.local
- ./seeders/seed.sh
- yarn start
- open browser to `localhost:9000` (port default to 9000)

## API Documentation

Once you've to the server started check out the api documentation at [/api-docs](http://localhost:9000/api-docs)

## Navigating the code base

- The entry point of the application is the [server/index.js](./server/index.js)
- The server/app.js imports the APIs from [server/api/index.js](./server/api/index.js)
- All the different APIs in the [server/api](./server/api) are registered [here](./server/api/index.js)
- MongoDB is used to store data & mongoose is used as the ORM
- [mongo](./server/database/mongo.js)
- [models](./server/database/models/)
- The template has support for the following middlewares
- [auth](./server/middlewares/auth/)
- [injectRequestId](./server/middlewares/injectRequestId)
- [rateLimiter](./server/middlewares/rateLimiter)
- The template has inbuilt support for
- [redis](./server/services/redis.js)
- [circuitBreakers](./server/services/circuitBreaker.js)
- [slack alerts](./server/utils/slackNotify.js)
- [docker](./Dockerfile)
- [docker-compose](./docker-compose.yml)
- [auto generated apis](./server/api/requestGenerators.js)
- [sharding of collections](./setup-shards)
- [aggregate caching](./server/api/aggregate/)

## Philosophy

When using NoSQLs you are optimising for read performance. We're doing this by denormalising data. There are multiple copies of the same data. For example

- Orders contains purchasedProducts which contains Products. Instead of referencing here we embed
- SupplierProducts contains embedded objects for both Suppliers and Products
- StoreProducts contains embedded objects for both Stores and Products

This makes our application write heavy. Every time there is a change to a product we need to make a change to

- SupplierProducts
- StoreProducts
- Products

Orders is not impacted since a change in the product after purchase will not affect the order.

However the application is able to perform extremely fast reads. 2 reasons for better performance is

- shards
- document embedding

NoSQLs are also good for handling large volumes of data. This is supported due to its ability to have shards. In this application we create 4 shards and the data is distributed amongst these shards.

These are the shard keys that we use

- \_id
- Order
- name
- Products
- Suppliers
- Stores

We got really good distribution across shards(24-26%) per shard after seeding 4 million records. It's possible to get a hot shard due to this but we're yet to see that.
- productId
- SupplierProducts
- StoreProducts

productId is chosen as the shard key since we anticipate that the queries for fetching all suppliers/stores that sell a particular product will be much more than fetching all products of a supplier/store.