{"id":16158770,"url":"https://github.com/omar-besbes/database-synchronization-app","last_synced_at":"2025-05-06T01:02:54.200Z","repository":{"id":163278964,"uuid":"637432270","full_name":"omar-besbes/database-synchronization-app","owner":"omar-besbes","description":"Database sync across multi-tier architecture via RabbitMQ, resilient to network disruptions.","archived":false,"fork":false,"pushed_at":"2024-03-05T22:02:09.000Z","size":350,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-06T01:02:44.733Z","etag":null,"topics":["database-synchronization","docker","docker-compose","kubernetes","lamport-clock","nestjs","rabbitmq"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/omar-besbes.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-07T14:40:07.000Z","updated_at":"2024-11-26T14:21:39.000Z","dependencies_parsed_at":"2024-11-02T05:22:05.192Z","dependency_job_id":"8f89e9fd-bcb3-4400-8a3a-04626a0a8d76","html_url":"https://github.com/omar-besbes/database-synchronization-app","commit_stats":null,"previous_names":["omar-besbes/database-synchronization-app"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omar-besbes%2Fdatabase-synchronization-app","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omar-besbes%2Fdatabase-synchronization-app/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omar-besbes%2Fdatabase-synchronization-app/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/omar-besbes%2Fdatabase-synchronization-app/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/omar-besbes","download_url":"https://codeload.github.com/omar-besbes/database-synchronization-app/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252601876,"owners_count":21774663,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database-synchronization","docker","docker-compose","kubernetes","lamport-clock","nestjs","rabbitmq"],"created_at":"2024-10-10T01:55:00.632Z","updated_at":"2025-05-06T01:02:54.182Z","avatar_url":"https://github.com/omar-besbes.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Database Synchronization System\n\n[![RabbitMQ](https://img.shields.io/badge/Rabbitmq-FF6600?style=for-the-badge\u0026logo=rabbitmq\u0026logoColor=white)](https://www.rabbitmq.com)\n[![NestJS](https://img.shields.io/badge/nestjs-%23E0234E.svg?style=for-the-badge\u0026logo=nestjs\u0026logoColor=white)](https://nestjs.com)\n[![Docker](https://img.shields.io/badge/docker-%230db7ed.svg?style=for-the-badge\u0026logo=docker\u0026logoColor=white)](https://www.docker.com)\n[![Kubernetes](https://img.shields.io/badge/kubernetes-%23326ce5.svg?style=for-the-badge\u0026logo=kubernetes\u0026logoColor=white)](https://kubernetes.io)\n\n## Overview\n\nThis database synchronization system addresses the challenge of synchronizing distributed databases in a scenario with unique constraints. In a setup comprising a Head Office (HO) and multiple Branch Offices (BOs) spread across different locations, including areas with limited internet connectivity, traditional synchronization methods prove inadequate. To overcome these challenges, this solution leverages RabbitMQ message queues to facilitate efficient data exchange between offices.\n\n## Features Roadmap\n\n-  [x] Abstraction over the number of branch offices (BOs), allowing for dynamic scalability.\n-  [x] Support for any kind of operation on the database (Create, Update, Delete).\n-  [x] Database-agnostic (SQL databases for now).\n-  [x] Manual synchronization of databases across multiple offices.\n-  [ ] Automatic synchronization upon internet outage recovery.\n-  [ ] Real-time synchronization.\n-  [ ] Support for dynamic schema evolution.\n\n## Project Structure\n\nThe project follows a monorepo style structure, organized into the following directories:\n\n-  `apps/`: Contains the source code for both the head office and branch office applications.\n\n   -  `apps/head/`: Source code for the head office application.\n   -  `apps/branch/`: Source code for the branch office application.\n\n-  `docs/`: Contains other relevant documentation apart from the README.md.\n\n-  `libs/`: Contains modules that are shared between the two applications.\n\n-  `orchestration/`: Contains Kubernetes configuraton files that describe how the app deployment and management should be orchestrated.\n\n-  `scripts/`: Contains various scripts required to start the entire system.\n\n## How it works ?\n\nWe will talk about how RabbitMQ makes it easy to manage communications between the different offices \nand then we will tackle how synchronization works.\n\nIf you don't know about RabbitMQ, you can read more about it [here](docs/rabbitmq.md) or you can google it and come back later.\n\n### RabbitMQ\n\nThere are 3 main entities envolved:\n- RabbitMQ broker\n- Head Office\n- Branch Office\n\nEach office has its own queue to consume messages from.\nThis way, even if the recipient is not available, the sender can still send its messages and the recipient will consume them when it's back.\n\nCommunications happen only between HO and BOs, so no inter-communications between BOs.\n\nWe will have 2 `fanout` exchanges:\n- One that recieves messages from the BOs and forwards them to the HO queue.\n- One that recieves messages from the HO and forwards them to each BO queue.\n\nThe concept of exchanges is what will enable adding or removing BOs without the need to change any office's configuration or RabbitMQ's.\n\n![](docs/RabbitMQ_synchronization_app_diagram.png)\n\n### Synchronization\n\nTODO\n\n## Installation and Setup\n\n\u003e [!NOTE]\n\u003e The installation process has been thoroughly tested on a Debian machine and is expected to work smoothly on all Unix-based systems. \u003cbr\u003e\n\u003e However, it is important to note that compatibility with other operating systems, such as Windows, cannot be guaranteed.\n\nFirst, clone the repository.\n\n```sh\ngit clone https://github.com/omar-besbes/database-synchronization-app.git\n```\n\n\u003cdetails\u003e \n\u003csummary\u003e Using \u003ccode\u003ekubectl\u003c/code\u003e (Recommended) \u003c/summary\u003e\n\u003cbr\u003e\n\nIf you want the full experience, you have to use kubernetes.\nThis includes the following:\n- having multiple BOs\n- on-demand scaling of BOs\n\nYou have to make sure that you have a cluster in your system.\nThis can be achieved by installing a tool like [minikube](https://minikube.sigs.k8s.io/docs/start/).\n\nTo verify that you have a cluster, run:\n\n```sh\nkubectl get po -A\n```\n\nYou should get an output like this:\n\n```sh\nNAMESPACE              NAME                                         READY   STATUS   RESTARTS        AGE\nkube-system            coredns-5dd5756b68-cqwk6                     1/1     Running   1 (11h ago)     18h\nkube-system            etcd-minikube                                1/1     Running   1 (11h ago)     18h\nkube-system            kube-apiserver-minikube                      1/1     Running   1 (3h20m ago)   18h\nkube-system            kube-controller-manager-minikube             1/1     Running   1 (11h ago)     18h\nkube-system            kube-proxy-mcjpb                             1/1     Running   1 (11h ago)     18h\nkube-system            kube-scheduler-minikube                      1/1     Running   1 (11h ago)     18h\nkube-system            storage-provisioner                          1/1     Running   3 (3h19m ago)   18h\nkubernetes-dashboard   dashboard-metrics-scraper-7fd5cb4ddc-sg5k5   1/1     Running   1 (11h ago)     18h\nkubernetes-dashboard   kubernetes-dashboard-8694d4445c-jnjwm        1/1     Running   2 (3h19m ago)   18h\n```\n\nPlease, either:\n\n-  copy the scripts folder under `orchestration/shared` using: `cp -r scripts/ orchestration/shared/`\n-  or (I personally prefer this) create a hard link for each file under `scripts` in `orchestration/shared/scripts`\n   using: `ln  scripts/* orchestration/shared/scripts/`\n\nTo get our system up and running, please run this:\n\n```sh\nkubectl apply -k orchestration/head-office/\nkubectl apply -k orchestration/branch-office/\n```\n\nThis will generate the necessary config maps for the 3 deployments (rabbitmq broker, head office and branch office) to start.\n\nIf you are using `minikube`, you can visualize your cluster by running:\n\n```sh\nminikube dashboard\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e \n\u003csummary\u003e Using \u003ccode\u003edocker compose\u003c/code\u003e (Recommended) \u003c/summary\u003e\n\u003cbr\u003e\n\nIf you just want to quickly this system working without needing multiple branch offices,\nthis methods suits your need. This will run a single HO and a single BO connected through RabbitMQ broker.\n\nTo get our system up and running, please run this:\n\n```sh\n./scripts/generate_secrets.sh \u003e .env\n./scripts/generate_ssl_certificates.sh \u003e\u003e .env\ndocker compose up\n```\n\nThis will generate a `.env` file containing the necessary credentials and ssl certificates in order for the databases to start.\n\n\u003c/details\u003e\n\n\u003cdetails\u003e \n\u003csummary\u003e Manually \u003c/summary\u003e\n\u003cbr\u003e\n\nI don't have any idea why someone would want to start the whole system manually.\nFor the sake of completeness, here it is.\n\nWe will first show how to set up a HO and a single BO and then how to add other BO.\n\n### Starting the system with a single branch office\n\n1. Install dependencies\n\n```sh\nyarn\n```\n\n2. Configure the application settings\n\n-  make sure you have a relational database up and running for each office you plan on running.\n-  make sure you have a RabbitMQ instance up and running. You can find out how to install and set up a RabbitMQ instance\n   locally [here](https://rabbitmq.com), or you can use managed RabbitMQ instance [here](https://www.cloudamqp.com).\n-  use the `.env.example` files (there is one in `apps/head` for the head office and one in `apps/branch` for branch\n   offices) to generate `.env` for each office. Put them besides the corresponding `.env.example` files.\n\n3. Build \u0026 start the applications\n\n```sh\n# build step\n# you only have to run this once for all offices\nyarn build head \u0026\u0026 yarn build branch\n# start step\n# you have to run these in separate shell sessions\nyarn start head # start the HO\nyarn start branch # start a BO\n```\n\n### Adding branch offices\n\nThis is the tricky part, we can not use the same `.env` for all branch offices. The environment variables are crucial\nfor how the whole system will work. So, to use multiple branch offices, we have to start them one by one. For each one,\nwe change the `.env` file and start a new branch office. Notice that changing the `.env` will not affect the already\nrunning instances as environment variables are loaded at startup and are never checked afterward. This is also why it is\nimportant to wait for the branch office app to start before moving on to another.\n\n\u003c/details\u003e\n\n## Why Kubernetes ?\n\nIn the setup section, Kubernetes is utilized for orchestrating the deployment. However, the system itself does not inherently leverage Kubernetes features such as load balancing or auto-scaling. Instead, Kubernetes is chosen primarily for simplifying testing and demonstrations.\n\nIn real-world deployments, where offices are geographically dispersed and may operate on separate physical servers, Kubernetes may not be necessary.\n\n\n## Forking and Customization\n\nIf you intend to fork this project and customize it to your needs, here are some considerations:\n\n### Database Agnosticism\n\nThe codebase is designed to be database agnostic for **relational** databases. This means that\ntheoretically, each office could use a different SQL database without encountering major compatibility issues.\nHowever, it's important to note that this feature has not been tested.\n\nTo minimize risk and ensure proper functionality, it's recommended to conduct thorough testing when deploying the system\nwith different RDBMS systems.\n\nAdditionally, any specific database-related configurations or queries should be carefully reviewed and adjusted as\nneeded to ensure compatibility with the chosen database platform. Although, in my opinion, if changing to different\nRDBMS systems will require having changes to queries, making this change defeats the whole purpose of having a single\ncode for multiple instances.\n\n## Contributing\n\nContributions are welcome! Please clone the repository, make your changes, and submit a pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomar-besbes%2Fdatabase-synchronization-app","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fomar-besbes%2Fdatabase-synchronization-app","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fomar-besbes%2Fdatabase-synchronization-app/lists"}