https://github.com/tommygrammar/distributed-scalable-data-architecture

Built a Distributed Data Architecture capable of supporting several platforms and multi-organizations
https://github.com/tommygrammar/distributed-scalable-data-architecture

data data-architecture distributed-systems fault- nginx

Last synced: 6 days ago
JSON representation

Built a Distributed Data Architecture capable of supporting several platforms and multi-organizations

• Host: GitHub
• URL: https://github.com/tommygrammar/distributed-scalable-data-architecture
• Owner: tommygrammar
• Created: 2024-08-10T15:09:49.000Z (6 months ago)
• Default Branch: scalable-data-architecture-systems
• Last Pushed: 2024-10-17T19:51:11.000Z (3 months ago)
• Last Synced: 2024-11-24T03:12:16.107Z (2 months ago)
• Topics: data, data-architecture, distributed-systems, fault-, nginx
• Language: Python
• Homepage:
• Size: 48.8 KB
• Stars: 1
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

        
# Distributed-Scalable-Data-Architecture

Used redis for this particualar project but do feel free to use any other database technology, you just have to change the addresses in the mds(Main Distribution Service) and also the write and read logic in wds and rds respectively.

You can easily add new data sources to read service and write service which house the Restful apis

Scale horizontally by adding more instances of rds and wds

Should serve as a foundation for a more complex data architecture.

What we have:

  -read service - houses the read endpoint which receives a read request thru an api, sends the data as a message using rabbitmq

  -read distribution service(rds) - responsible for the reading of the redis instance. receives the request from rabbitmq, reads it and sends the response to read service which then updates the frontend thru the api. periodically checks the  instances addresses.

  -write-service - same as read service but now for writing

  -write distribution service - same as rds, but for writing. should add tho that in wds, it writes to two redis instances, a main and a backup, the redis instances adresses are provided by the main distribution service. periodically checks the instances addresses

  -main distribution service(mds) - houses the addresses of the redis instances. dynamically reassigns main and backup roles based on availability of backup instances and connection health.

  -rabbitmq message queues for processing reading and writing requests

But yes, understand how everything works and you are going to find it easy. I should be rolling out new updates really soon in another branch. The goal is a high frequency data architecture with extremely high performance.

will rolout the docker compose or kubernetes soon.

the terminal.sh is a bash automated script for running all of tgem in different terminals using one script.