{"id":15056667,"url":"https://github.com/odennav/large-scale-system-microservices-gcp","last_synced_at":"2025-04-10T04:33:54.618Z","repository":{"id":228835483,"uuid":"775045979","full_name":"odennav/large-scale-system-microservices-gcp","owner":"odennav","description":"Deploy a Large-scale system with microservices using Google Kubernetes Engine","archived":false,"fork":false,"pushed_at":"2024-05-02T22:08:34.000Z","size":7769,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-02T19:51:25.638Z","etag":null,"topics":["auto-scaling","cassandra","django-application","docker","elasticsearch","fluentd","gke-cluster","jaeger","kibana","kubernetes","logstash","netflix-eureka","netflix-ribbon","netflix-zuul","nginx-proxy","postgresql-database","rabbitmq","redis","rolling-upgrade","spring-boot"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/odennav.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-03-20T16:57:25.000Z","updated_at":"2024-05-23T10:10:12.000Z","dependencies_parsed_at":"2024-05-03T06:56:48.297Z","dependency_job_id":"86650534-df67-433a-8c86-5849b5945597","html_url":"https://github.com/odennav/large-scale-system-microservices-gcp","commit_stats":null,"previous_names":["odennav/large-webapp-micros-gcp","odennav/large-scale-app-micros-gcp","odennav/large-scale-system-microservices-gcp"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Flarge-scale-system-microservices-gcp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Flarge-scale-system-microservices-gcp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Flarge-scale-system-microservices-gcp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/odennav%2Flarge-scale-system-microservices-gcp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/odennav","download_url":"https://codeload.github.com/odennav/large-scale-system-microservices-gcp/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239140090,"owners_count":19588333,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["auto-scaling","cassandra","django-application","docker","elasticsearch","fluentd","gke-cluster","jaeger","kibana","kubernetes","logstash","netflix-eureka","netflix-ribbon","netflix-zuul","nginx-proxy","postgresql-database","rabbitmq","redis","rolling-upgrade","spring-boot"],"created_at":"2024-09-24T21:54:55.597Z","updated_at":"2025-02-16T14:31:10.923Z","avatar_url":"https://github.com/odennav.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n### Deployment of Large Scale System with Kubernetes on Google Cloud ###\nDeploy a Large-scale system that utilizes Python Django for the Webapp, Java Spring Boot for the RESTful services, PostgreSQL and Cassandra for the database, ReactJS for the SPA(Single Page Application).\n\nContainerization is achieved with Docker containers and Docker-compose, Service discovery handled by Netflix-Eureka and Gateway provided by Netflix-Zuul.\n\n\n![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/41.PNG)\n\nClient-side load balancing is managed using Ribbon, while Server-side load balancing is facilitated by Nginx.\n\nLogging is handled by Elasticsearch, Fluent, and Kibana, with tracing supported by Uber Jaeger, and Monitoring and alerts are provided by Prometheus.\n\nCaching is managed using Redis, RabbitMQ facilitates asynchronous processing, and Cassandra handles horizontal data partitioning.\n\nDeployment is on Google Cloud kubernetes engine, enabling Auto-scaling, High availability, and Rolling upgrades.\n\n\n\n### Getting Started ###\n\n1. **Create New Project**\n   \n   Choose name of your project.\n   \n   Confirm your billing account or use 90-day trial to help cover costs of resources for this project.\n\n   Search for 'Compute Engine' and select 'CREATE INSTANCE'.\n\n   Enter name of VM instance, select Region, Zone and Machine Configuration(Low cost, day-to-day computing).\n   \n   **Select Machine type** \n\n   - 4 vCPUs, 2 core, 16 GB memory\n   \n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/39.png)\n\n\n   Change availability policies to 'Spot' VM provisioning model.\n   This reduces monthly estimate cost by more than 50%\n\n   Each instance requires a disk to boot from. Select an image to create a new boot disk or attach an existing disk to the instance.\n   \n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/40.png)\n\n   Then select 'CREATE' at bottom to provision new VM instance.\n   Note the External IP assigned.\n\n\n2. **Create a New User on VM Instance**\n   \n   Click on 'SSH' and allow SSH-in-browser to connect to devbuild-1 VM\n   Second option for SSH connection as shown [here](https://www.youtube.com/watch?v=fmh94mNQHQc):\n   \n   - Generate your own private/public key pair, copy public key to settings of devbuild-1 instance and save private key on your local machine.\n   - Connect to devbuild-1 instance using putty\n   \n   Change password for root user\n   ```bash\n   sudo passwd\n   ```\n   Switch to root user.\n   Add new user to sudo group. In this case new user is 'odennav-admin'\n\n   ```bash\n   sudo adduser odennav-admin\n   sudo usermod -aG sudo odennav-admin\n   ```\n   You'll be prompted to set a password and provide additional information about the new    \n   user, such as full name, work phone, etc. This information is optional. Press 'Enter'   \n   to skip each prompt.\n    \n   ```bash\n   Test sudo privileges by switching to new user\n   su - odennav-admin\n   sudo ls /root\n   ```\n\n   You'll notice prompt to enter your user password.\n   To disable this prompt for every sudo command, implement the following:\n\n   Add sudoers file for odennav-admin\n   ```bash\n   cd /etc/sudoers.d/\n   echo \"odennav-admin ALL=(ALL) NOPASSWD: ALL\" \u003e odennav-admin\n   ```\n   Set permissions for sudoers file\n   ```bash\n   chmod 0440 odennav-admin\n    ```\n\n3. **Confirm Git is installed**\n   ```bash\n   git version\n   ```\n\n   If not available, install the package\n   ```bash\n   sudo apt update\n   sudo apt install git -y\n   git version\n   ```\n\n   Clone git repo codebase\n   ```bash\n   git clone https://github.com/odennav/large-scale-app-micros-gcp.git\n   ```\n\n4. **Complete Image Build of System Components**\n   \n   Enable execution of all bash scripts in large-scale-app-micros-gcp/ directory\n   ```bash\n   cd large-scale-app-micros-gcp/\n   sudo find ~/large-scale-app-micros-gcp/ -type f -name \"*.sh\" -exec sudo chmod +x {} \\;\n   ```\n\n   Install docker and docker-compose\n   ```bash\n   cd large-scale-app-micros-gcp/bin\n   ./install-docker.sh\n   ```\n   \n   Confirm docker is installed\n   ```bash\n   docker version\n   ```\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/4.png)\n\n   Trigger a complete build and build the entire codebase.\n   -Building codebase\n   -Staging them to a stage directory\n   -Building docker images required\n   ```bash\n   ./build-system.sh\n   ```\n\n\n5. **Setup Container Registry and Push images to Google Cloud Registry**\n   \n   Before we can upload created images to google cloud registry, we have to authorize host docker in devbuild-1 VM instance.\n   ```bash\n   cd ~/large-scale-app-micros-gcp/bin\n   ./gcp-authorize-docker.sh\n   ```\n   During remote authentication to gcloud CLI, you'll be asked to copy link in your browser because Google SDK wants to access your google account.     Allow access, copy authorization code and paste in SSH-in-browser.\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/2.png)\n   It locates all images starting with 'ntw', then tags them before pushing to registry\n   Ensure registry zone in script is the same region as your devbuild-1 instance and is also a region in either US, Europe or Asia.\n\n   **REGISTRY_HOST**=eu.gcr.io due to VM instance and Kubernetes cluster provisioned in European region.\n\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes\n   ./gcp-push-images.sh\n   ```\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/5.png)\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/6.png)\n\n6. **Create Kubernetes Cluster on Google Cloud**\n\n   Go to 'Compute' section and click 'Kubernetes Engine'.\n   Enable Kubernetes API\n   Click on 'CREATE' under section of Kubernetes clusters.\n   Choose 'GKE Standard' or switch to 'STANDARD CLUSTER'.\n   Name your kubernetes cluster and select 'Regional' for Location type, to have our system in multiple zones within a region.\n   Select same region for devbuild-1 VM instance\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/7.png)\n-----\n   \n   **Change number of nodes per zone to 1 in 'Node pool details', so we have 3 machines in different zones.**\n   \n   \n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/8.png)\n\n-----\n   **For each node in pool:**\n   - Select E2 machine type\n   - 6 vCPU, 2 cores each\n   - 12GB memory total\n   - Boot disk size of 50GB\n   - Enable nodes on spot VMs(reduces monthly cost)\n   \n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/9.png)\n\n\n\n-----\n   **Click 'CREATE' at bottom to start process of creating kubernetes cluster.**\n\n   Note cluster created with total of 18 vCPUs, 36GB memory and nodes from each zone.\n\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/11.png)\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/10.png)\n\n\n-----\n\n7. **Kubernetes Environment Configuration for System**\n   \n   To host our system on this gcp k8s cluster, we'll have to create k8s config and services.\n\n   Install kubectl\n   kubectl enables devbuild-1 VM instance to communicate with the kubernetes cluster\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes\n   ./gcp-install-kubectl.sh\n   ```\n\n\n\n   **Configure kubectl command line access to k8s cluster**\n\n   Locate your cluster created in 'Kubernetes Engine' section on GCP and click on ':'  to view options, then click 'Connect'.\n   Copy and paste gcloud command to devbuild-1 terminal.\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/12.png)\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/13.png) \n   \n\n-----\n\n   Confirm kubectl has access to nodes in pool\n   ```bash\n   kubectl get nodes\n   ```\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/14.png)\n\n   \n\n   Create namespaces\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env\n   kubectl apply -f 0-namespaces.yaml\n   ```\n\n   Apply config map\n   Set environment variables for each component in system\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env\n   kubectl apply -f 1-config-map.yaml\n   ```\n\n   Apply secrets config\n   Set environment variables for secrets to postgresql databases\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env\n   kubectl apply -f 2-secrets-map.yaml\n   ```\n   You can delete 2-secrets.yaml file after secrets resource is created and stored in k8s cluster.\n\n   Apply resources limit\n   Set CPU and Memory resource limits to pods created in services namespace\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env\n   kubectl apply -f 3-resources.yaml\n   ```\n\n\n8. **Kubernetes Volume Configuration**\n   \n   Configure persistent volumes for each node in node pool\n\n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes/volume\n   ./kube-volumes.sh\n   ```\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/15.png)\n\n   Set up all System Workloads on Kubernetes Cluster\n   Identify all config yaml files and create all resources required to start the app on kubernetes cluster\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/37.png)\n\n   kube-deploy.sh script creates all services and deployments configured in the yaml files identified.\n   Ensure variable **REGISTRY_HOST**=eu.gcr.io is set in bash script due to regional setup of clusters and registry \n   ```bash\n   cd ~/large-scale-app-micros-gcp/kubernetes\n   ./kube-deploy.sh\n   ```\n\n   Go to 'Workloads' section under 'Kubernetes Engine' and view pods created and running.\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/18.png)\n\n   Confirm all namespaces have been created\n   ```bash\n   kubectl get ns\n   ```\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/25.png)\n\n   Check for webapp and spa services in ui namespace\n\n   ```bash\n   kubectl get svc -n ui\n   ```\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/26.png)\n\n9. **Access both Webapp and SPA**\n   \n   Create 'allow-ssh' firewall rule in your vpc network settings.\n   Search for 'VPC Network' in Google cloud platform and click on 'Firewall' on left-side bar. Select 'CREATE FIREWALL RULE'.\n   - Enter name of firewall rule - 'allow-sys'\n   - Direction of traffic is 'Ingress'\n   - Use '0.0.0.0/0' as source of IPv4 ranges\n   - Under section for 'Protocol and ports' select 'TCP' and insert Nodeports configured for both webapp and spa, 32100 and 32105.\n   \n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/28.png)\n\n   - **Click on 'CREATE' at bottom to create VPC firewall rule.**\n\n   Access Login page from your browser on local machine.\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/24.png)\n\n   Please note this option opens ports 32100 and 32105 on all VM instances in your VPC network.\n   \n   Effectively, you can select the firewall rules initially created for each of the kubernetes cluster nodes and edit firewall settings by including    Nodeports among TCP ports allowed.\n\n\n10. **Access System Components for Centralized Logging, Tracing and Resource Monitoring**\n   \n    Add the following Nodeports as TCP ports in firewall rule created above\n    - **32101 for Kibana/Elasticsearch** \n   \n    ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/30.png)\n\n    - **32102 for Uber Jaeger**\n\n    ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/31.png)\n\n    - **32103 for Prometheus**\n   \n    ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/33.png)\n\n    - **32104 for RabbitMQ**\n\n    ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/36.png)\n\n\n\n## Shutdown Kubernetes Cluster\n   To remove all resources for kubernetes cluster and stop incurring charges. \n   \n    ```bash\n    cd ~/large-scale-app-micros-gcp/kubernetes\n    ./kube-deploy.sh ./config/ delete\n    ```\n    Delete persistent volumes created\n    \n    ```bash\n    cd ~/large-scale-app-micros-gcp/kubernetes\n    ./kube-volumes.sh delete\n    ```\n\n   Go to 'Clusters' section in 'Kubernetes Engine' product and delete cluster.\n   Click on Actions ':' menu and select 'Delete'\n\n   ![](https://github.com/odennav/large-scale-system-micros-gcp/blob/main/docs/38.png)\n\n\n## Special Credits\n\nSpecial thanks to [Anurag Yadav](https://www.newtechways.com/).\n\n\n\n### Contributions ###\n\n* Writing tests\n* Code review\n\nEnjoy!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodennav%2Flarge-scale-system-microservices-gcp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fodennav%2Flarge-scale-system-microservices-gcp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fodennav%2Flarge-scale-system-microservices-gcp/lists"}