Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify

Config based Node.js and PostgreSQL Web App Boilerplate/Scaffolding with custom RBAC and stream-based CSV upload and download mechanics. Auto generated Modules through superAdmin panel with features like for API documentation - swagger , for Automation test stack - Mocha and more. Please refer Solution statement from README below.
https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify

api boilerplate docker fastify jenkins-pipeline mocha-chai mocha-tests multi-column-selection-predicates nodejs nodejs-api pivot-tables postgresql redis streaming-csv swagger

Last synced: 4 months ago
JSON representation

Config based Node.js and PostgreSQL Web App Boilerplate/Scaffolding with custom RBAC and stream-based CSV upload and download mechanics. Auto generated Modules through superAdmin panel with features like for API documentation - swagger , for Automation test stack - Mocha and more. Please refer Solution statement from README below.

Awesome Lists containing this project

README 

        
### Badges

[![Node.js Postgres Vanilla Fastify CI](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/actions/workflows/node.js.yml/badge.svg)](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/actions/workflows/node.js.yml)



[![CodeQL](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/actions/workflows/codeql-analysis.yml)



[![GitHub Clones ](https://img.shields.io/badge/dynamic/json?color=success&label=Clone&query=count&url=https://gist.githubusercontent.com/matt212/fff906f983d73f42eadc74ab93701a83/raw/clone.json&logo=github)](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/actions/workflows/cloneCount.yml)



 [![contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/issues)



### Tech stacks

![NodeJS](https://img.shields.io/badge/node.js-6DA55F?style=for-the-badge&logo=node.js&logoColor=white)

![Postgres](https://img.shields.io/badge/postgres-%23316192.svg?style=for-the-badge&logo=postgresql&logoColor=white)

![JavaScript](https://img.shields.io/badge/javascript-%23323330.svg?style=for-the-badge&logo=javascript&logoColor=%23F7DF1E)

![Redis](https://img.shields.io/badge/redis-%23DD0031.svg?style=for-the-badge&logo=redis&logoColor=white)

![Sequelize](https://img.shields.io/badge/Sequelize-52B0E7?style=for-the-badge&logo=Sequelize&logoColor=white)

![Socket.io](https://img.shields.io/badge/Socket.io-black?style=for-the-badge&logo=socket.io&badgeColor=010101)

![JWT](https://img.shields.io/badge/JWT-black?style=for-the-badge&logo=JSON%20web%20tokens)

![Mocha](https://img.shields.io/badge/-mocha-%238D6748?style=for-the-badge&logo=mocha&logoColor=white)

![Swagger](https://img.shields.io/badge/-Swagger-%23Clojure?style=for-the-badge&logo=swagger&logoColor=white)



 🚄 Problem Statement 



 ![redlime](app/video/img/problem-statement.png)



 



🚀 Solution Statement- Key Outcomes



   

  

 🍨 Objective 



![redlime](app/video/img/objective.png)





   



 🍰 Mandates 



![redlime](app/video/img/mandates.png)





    



 🍦 Features/Functionalities : Conventional Levels



![redlime](app/video/img/conventional.png)






 🍧 Features/Functionalities : Advanced Levels



![redlime](app/video/img/advanced.png)








 🎫 

 Scope Statement

 



![redlime](app/video/img/Scope.png)






📡  Platform Setup 





   



🍕sh file Setup



> *`chmod +x ./app/utils/serverSetup/primarySetup.sh `*  



> *`sed -i -e 's/\r$//' ./app/utils/serverSetup/> primarySetup.sh`*



>*`chmod +x ./app/utils/serverSetup/secondarySetup.sh`*



>*`sed -i -e 's/\r$//' ./app/utils/serverSetup/secondarySetup.sh`*



>*`bash ./app/utils/serverSetup/primarySetup.sh`*



   



> _`Please note if you have node/postgres/redis server installed in your instance/machine. Please comment the install information in primarySetup.sh file and then run this file`_



  

  



🍤app Setup



  

> **node,postgres, redis and app dependencies  installation**



>*`bash ./app/utils/serverSetup/primarySetup.sh`*



>*`bash ./app/utils/serverSetup/secondarySetup.sh`*



🍔 mocha chai tests against 20,00,000 recordsets



> _`npm run LoginEval or yarn LoginEval`_



> _`npm run ModuleEval or yarn ModuleEval`_



🍟 app Run



>*`npm run app or yarn app `*



🍩 app Usage



>*`goto localhost:3012/employees`*



>*`username : krennic and  password:orson`*



>*`select birthdate from dropdown and select date range any from 1982 to till date`*



 🍜 app SuperAdmin Usage



>*`goto localhost:3012/black-squadron`*



>*`username : krennic and password:orson`*



>*`for any other controls other than textbox it is mandatory`*



>*`to enter PIPE "|" separator in textbox column for e.g genderid|gendername`*



 🍘 API Documentation Usage(Swagger)



>*`goto localhost:3012/getAccessToken`*



>*`Click on Get AccessToken button and copy the newly generated Access Token`*



>*`goto localhost:3013/api-docs/`*



>*`Click on authorize and paste the access Token`*






 💈 Walkthroughs : Features/Functionalities 

 


 



    🍇 Dynamic pivot operation from Postgres with highchart implementation (Redis Cached)



 


 

 https://user-images.githubusercontent.com/1698139/182132755-e19fdbb3-23f3-422e-ac54-4f2e78d3a2c3.mp4

 







 🍊 Node.js stream based CSV download for more than 2 million recordsets from PostgresSQL with socket.io as progress feature





https://user-images.githubusercontent.com/1698139/182214015-847cced0-95a7-46ad-a9c2-c359e5dcb4de.mp4 







 🍑 Automated Swagger for each Module created By build in scaffoding accessed at 

 
 http://localhost:3013/api-docs/ 




  

  `yarn enable-swagger`

  

  https://user-images.githubusercontent.com/1698139/182231396-1a483aed-7856-4302-beba-2fd879ee87d0.mp4 







 🍐 Basic dashboard operation





https://user-images.githubusercontent.com/1698139/182218607-456a32d1-1239-4a98-b0a5-30ea2dc4da4a.mp4







 🍓 Boilerplate walkthrough with multi control(Radio and Checkbox-Data from respective db tables) with Automated mocha test Cases





![redlime](app/video/gif/f.gif)







 🍏 Boilerplate walkthrough with multi control(Radio-Data from respective db table) with Automated mocha test Cases





![redlime](app/video/gif/e.gif)









🤖 Deliverables Timeline



   

  

 🥑 2022 



![redlime](app/video/img/ph4.png)





   



 🌽 2021 



![redlime](app/video/img/ph3.png)





    



 🍄 2020



![redlime](app/video/img/ph1.png)

![redlime](app/video/img/ph2.png)









#### 

 

 🍫 Releases



- ### `Release notes 23 July 2022`



  >Optimized Dynamic Multi Column , Multi Select  filter from 5 secs to 16 ms for 2   million rows . 

  >[My stack overflow solution](https://stackoverflow.com/a/73085784/1333794)

   



- ### `Release notes 21 July 2022`



  >base and pivot Operation optimized

  

  >incremental load (pagination section loaded after report rendered) for base report

  

  >incremental load (pagination section loaded after report rendered) for pivot report   



- ### `Release notes 19 July 2022`



  >github actions

  >Integrated CI-workflows with redis, postgres for automation mocha test cases

  >

  >[Github Action - Automation mocha test cases](https://github.com/matt212/Nodejs_Postgresql_VanillaJS_Fastify/runs/7475512509?check_suite_focus=true )

   

- ### `Release notes 18 July 2022`



  >Implemented basic dark theme   



- ### `Release notes 17 July 2022`



  >Implemented Redis based Server-Side Caching for Pivot Operations  



- ### `Release notes 1 July 2022`



> **Superadmin scaffolding/ boilerplate Access URL**

 > *http://localhost:3012/black-squadron*

 

 > _`username : krennic and password:orson`_



 >`for any other controls other than textbox it is mandatory `



 >` to enter PIPE "|" separator in textbox column for e.g genderid|gendername` 



- ### `Release notes 11 Feb 2021`



  >boilerplate for multiselect integration with radio and checkboxes completed 



- ### `Release notes 09 Feb 2021`



  >Applied in  boilerplate multi and single select Dynamic SQLConstruct with 360 degree coverage 



- ### `Release notes 06 Feb 2021`



  >for boilerplate multi and single select static mapping for edit and default rendering case scenarios completed

  

- ### `Release notes 29 Jan 2021`



  >validations applied for bolierplate dynamic radio button and checkboxes with 



- ### `Release notes 27 Jan 2021`



  >applied multi control for radio scaffolding 

 

- ### `Release notes 20 Jan 2021`



  >For rbac test cases modularized



  > `yarn mroleEval`



  > `yarn userroleEval`



- ### `Release notes 9 Jan 2021`



  >For rbac user-Role Mapping  module migrated to fastify with its 35 mocha chai test cases respectively 



  > `yarn mroleEval`



  > `yarn userroleEval`



- ### `Release notes 22 Dec 2020`

  >Dynamic Bolierplate generation completed for dataType Strings , Integers and BigInt data types config based

  

  >For rbac modname,muser modules migrated to fastify with their respective 43 and 63 mocha chai test cases

  

  >For rbac role module migrated to fastify with its 43 mocha chai test cases

  > `yarn ModEval`

  > `yarn roleEval`

 

- ### `Release notes 21 Dec 2020`

  > 73 **Modular** Mocha test cases against 20,00,000 recordset with 360 coverage including schema,NaN,Undefined,Multi Column Search,Multi Select,Pivot X and Y for pageSize and pageNo  payload Validations

  

  > `yarn ModuleEval`



- ### `Release notes 20 Dec 2020`

  > 73 **Modular** Mocha test cases against 20,00,000 recordset with 360 coverage including schema,NaN,Undefined,Multi Column Search,Multi Select,Pivot X and Y for pageSize and pageNo  payload Validations

  > `yarn ModuleEval`



- ### `Release notes 11 Dec 2020`

  > basic login tests are written and can view and evaluated by running below command

  > `yarn LoginEval`



- ### `Release notes 10 Dec 2020`

  > run `yarn perfgraph`

  > in new shell run autocannon `yarn customAutocannon`

  > when autocannon bench is completed go back to perfgraph shell and stop the shell (ctlrl+C)

  > it will generate graph html link in same perfgraph shell



- ### `Release notes 7 Dec 2020`

  > made date filter optional with code refractoring by introducing `disableDate=true` arguement in API payload.




  



 

 

 🚁 Project Statement 



![redlime](app/video/img/merge_from_ofoct.jpg)






 

 🛰️ Architecture 




 🌯 Server Side 




 🥐 Route Structure Module Operations 



![redlime](app/video/img/route-structure-module-operations.png) 



 🥗 Module View (EJS) Render 



![redlime](app/video/img/module-view-ejs.png) 



 🍲 Module Load Operations 



![redlime](app/video/img/module-load-operations.png) 



 🍟 Multipurpose Features Operations 



![redlime](app/video/img/multipurpose-features-operations.png) 



 🧆 Groupby Operations 



![redlime](app/video/img/groupby-operations.png) 



 🥞 Create Operation 



![redlime](app/video/img/create-operations.png) 



 🥨 Update Operations 



![redlime](app/video/img/update-operations.png) 



 🥖 ETL Upload Operations 



![redlime](app/video/img/upload-operations.png) 



 🥐 ETL Download Operations 



![redlime](app/video/img/download-operations.png) 



 🍳 Api Cache Operations



![redlime](app/video/img/api-cache-operations.png) 



 🥙 Api Cache Operations-Drilldown



![redlime](app/video/img/api-cache-drilldown.png) 






 



 🛸 Client Side 




 ✈️ View Structure and Page Operations 



![redlime](app/video/img/view_structure_and_page_operations.png) 



 🛩️ Page Load and Date Range Operations 



![redlime](app/video/img/page_load_and_date_range_operations.png) 



 

🛶 Sort and Pagination Operations 



![redlime](app/video/img/sort_and_pagination_operations.png) 



 🪂 Filter Operations 



![redlime](app/video/img/Filter_operations.png) 



 ⛵ Create and Update Operations 



![redlime](app/video/img/create_and_update_Operations.png) 



 ⛴️ Field Validations 



![redlime](app/video/img/Field_validations.png) 



 🚤 Field Validations-Event listeners 



![redlime](app/video/img/event_listners.png) 



 🤿 App Module-ActionFunctions 



![redlime](app/video/img/app_module-actionFunctions.png) 



 🛷 Pivot Operations 



![redlime](app/video/img/pivotoperations.png) 



 🏝️ ETL upload Operations 



![redlime](app/video/img/etl-upload-operations.png) 



 🍱 ETL download user Actions



![redlime](app/video/img/etl-download-user-actions.png) 



 🍣 ETL download Operations



![redlime](app/video/img/etl-download-operations.png) 






 🌯 Database Side




 🥐 RBAC-ERD 



![redlime](app/video/img/RBAC-ERD.png)



 🍄  RBAC-Case Scenarios 



![redlime](app/video/img/rbac-case-scenarios.jpg)







 

 🍍 Release Flow 




 🥥 Nodejs Web App Env Release Mechanism Pt-1 



![redlime](app/video/img/nodejs-web-app-Env-pt1.png) 



 🍒 Nodejs Web App Env Release Mechanism Pt-2 



![redlime](app/video/img/nodejs-web-app-Env-pt2.png) 



 

🍌 How to Release Node App 



![redlime](app/video/img/how-to-release-node-app.png) 






 

 🌶️ Performance  




 🧄 Normal Mode (API against 2 Million Rows )[requests:2000, concurrent:100] 



https://user-images.githubusercontent.com/1698139/182044366-3edb4b47-a7f0-4330-83fd-b2a3ea773117.mp4



 🥑 Cluster Mode (API against 2 Million Rows )[requests:2000, concurrent:100]



https://user-images.githubusercontent.com/1698139/182044831-87c0a025-0168-413b-a96f-b278ade5489f.mp4






  





#### 🍱Reach out to me 🍝 🍤



[![Linkedin Profile](https://img.shields.io/badge/LinkedIn-0077B5?style=for-the-badge&logo=linkedin&logoColor=white

)](https://www.linkedin.com/in/rizwan-patel/)