Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/wednesday-solutions/node-mongo-express

An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.
https://github.com/wednesday-solutions/node-mongo-express

authorization circuitbreaker express mongodb mongoose node-api node-i18n node-mongo-api node-mongo-express node-mongo-express-api nodejs oauth2 rbac rest-api sharding

Last synced: 3 months ago
JSON representation

An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.

Awesome Lists containing this project

README 

        





  

    

  

  


    


Node Mongo Express

    


  



  


An enterprise Mongo-Express REST API built using nodejs showcasing - Testing Strategy, mongoDB sharding, models, a REST API Interface, support for Redis, aggregation queries, aggregation caching, circuit-breakers, slack integration, RBAC, rate limited APIs and multi-container queues and schedulers.

  



---



  


    




      Expert teams of digital product strategists, developers, and designers.

    


  



  


    

      

    

    

      

    

  



---



We’re always looking for people who value their work, so come and join us. We are hiring!






[![Node Mongo Express CI](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/ci.yml/badge.svg)](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/ci.yml)



[![Node Mongo Express CD](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/cd.yml/badge.svg)](https://github.com/wednesday-solutions/node-mongo-express/actions/workflows/cd.yml)



---




















---



## Pre-requisites



    - yarn

    - docker



## Features



    - Mongo support

    - Docker support

    - Rate limited APIs

    - RBAC middleware using Auth0

    - Sharding mongoDB collection support

    - Paginated APIs

    - Autogenerated APIs from mongoose models

    - Built in slack alerting mechanism

    - Suport for redis cache

    - Support for aggregate caching

    - Support for batch jobs in multi-container environment

    - Support for circuit breakers

    - Autogenerated swagger documentation

    - Load testing using k6

    - Support for i18n



## Running Load tests



- [Install](https://k6.io/docs/getting-started/installation/) k6

- Execute the following command: `k6 run __tests__/__load__/script.js`

## Build and run docker container locally



    - docker-compose down

    - docker-compose build

    - docker-compose up



# Shard setup



Run the following script



```

./setup-shards/scripts/setup/base.sh

```



Take a look at [this](./setup-shards/README.md) to create shards and replica sets.



## Seeders



Run the following command to begin seeding



```

./seeders/seed.sh

```



## How to start



    - cd `node-mongo-express`

    - yarn

    - ./setup-shards/scripts/setup/base.sh

    - cp .env.example .env.local

    - ./seeders/seed.sh

    - yarn start

    - open browser to `localhost:9000` (port default to 9000)



## API Documentation



Once you've to the server started check out the api documentation at [/api-docs](http://localhost:9000/api-docs)



## Navigating the code base



-   The entry point of the application is the [server/index.js](./server/index.js)

-   The server/app.js imports the APIs from [server/api/index.js](./server/api/index.js)

-   All the different APIs in the [server/api](./server/api) are registered [here](./server/api/index.js)

-   MongoDB is used to store data & mongoose is used as the ORM

    -   [mongo](./server/database/mongo.js)

    -   [models](./server/database/models/)

-   The template has support for the following middlewares

    -   [auth](./server/middlewares/auth/)

    -   [injectRequestId](./server/middlewares/injectRequestId)

    -   [rateLimiter](./server/middlewares/rateLimiter)

-   The template has inbuilt support for

    -   [redis](./server/services/redis.js)

    -   [circuitBreakers](./server/services/circuitBreaker.js)

    -   [slack alerts](./server/utils/slackNotify.js)

    -   [docker](./Dockerfile)

    -   [docker-compose](./docker-compose.yml)

    -   [auto generated apis](./server/api/requestGenerators.js)

    -   [sharding of collections](./setup-shards)

    -   [aggregate caching](./server/api/aggregate/)



## Philosophy



When using NoSQLs you are optimising for read performance. We're doing this by denormalising data. There are multiple copies of the same data. For example



-   Orders contains purchasedProducts which contains Products. Instead of referencing here we embed

-   SupplierProducts contains embedded objects for both Suppliers and Products

-   StoreProducts contains embedded objects for both Stores and Products



This makes our application write heavy. Every time there is a change to a product we need to make a change to



-   SupplierProducts

-   StoreProducts

-   Products



Orders is not impacted since a change in the product after purchase will not affect the order.



However the application is able to perform extremely fast reads. 2 reasons for better performance is



-   shards

-   document embedding



NoSQLs are also good for handling large volumes of data. This is supported due to its ability to have shards. In this application we create 4 shards and the data is distributed amongst these shards.



These are the shard keys that we use



-   \_id

    -   Order

-   name

    -   Products

    -   Suppliers

    -   Stores

        
We got really good distribution across shards(24-26%) per shard after seeding 4 million records. It's possible to get a hot shard due to this but we're yet to see that.

-   productId

    -   SupplierProducts

    -   StoreProducts

        
productId is chosen as the shard key since we anticipate that the queries for fetching all suppliers/stores that sell a particular product will be much more than fetching all products of a supplier/store.