Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/odennav/large-scale-system-microservices-gcp
Deploy a Large-scale system with microservices using Google Kubernetes Engine
https://github.com/odennav/large-scale-system-microservices-gcp
auto-scaling cassandra django-application docker elasticsearch fluentd gke-cluster jaeger kibana kubernetes logstash netflix-eureka netflix-ribbon netflix-zuul nginx-proxy postgresql-database rabbitmq redis rolling-upgrade spring-boot
Last synced: 3 days ago
JSON representation
Deploy a Large-scale system with microservices using Google Kubernetes Engine
- Host: GitHub
- URL: https://github.com/odennav/large-scale-system-microservices-gcp
- Owner: odennav
- License: apache-2.0
- Created: 2024-03-20T16:57:25.000Z (6 months ago)
- Default Branch: main
- Last Pushed: 2024-05-02T22:08:34.000Z (5 months ago)
- Last Synced: 2024-09-24T21:54:21.871Z (3 days ago)
- Topics: auto-scaling, cassandra, django-application, docker, elasticsearch, fluentd, gke-cluster, jaeger, kibana, kubernetes, logstash, netflix-eureka, netflix-ribbon, netflix-zuul, nginx-proxy, postgresql-database, rabbitmq, redis, rolling-upgrade, spring-boot
- Language: Java
- Homepage:
- Size: 7.41 MB
- Stars: 2
- Watchers: 1
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
### Deployment of Large Scale System with Kubernetes on Google Cloud ###
Deploy a Large-scale system that utilizes Python Django for the Webapp, Java Spring Boot for the RESTful services, PostgreSQL and Cassandra for the database, ReactJS for the SPA(Single Page Application).
Containerization is achieved with Docker containers and Docker-compose, Service discovery handled by Netflix-Eureka and Gateway provided by Netflix-Zuul.
Client-side load balancing is managed using Ribbon, while Server-side load balancing is facilitated by Nginx.
Logging is handled by Elasticsearch, Fluent, and Kibana, with tracing supported by Uber Jaeger, and Monitoring and alerts are provided by Prometheus.
Caching is managed using Redis, RabbitMQ facilitates asynchronous processing, and Cassandra handles horizontal data partitioning.
Deployment is on Google Cloud kubernetes engine, enabling Auto-scaling, High availability, and Rolling upgrades.
### Getting Started ###
1. **Create New Project**
Choose name of your project.
Confirm your billing account or use 90-day trial to help cover costs of resources for this project.
Search for 'Compute Engine' and select 'CREATE INSTANCE'.
Enter name of VM instance, select Region, Zone and Machine Configuration(Low cost, day-to-day computing).
**Select Machine type**
- 4 vCPUs, 2 core, 16 GB memory
Change availability policies to 'Spot' VM provisioning model.
This reduces monthly estimate cost by more than 50%
Each instance requires a disk to boot from. Select an image to create a new boot disk or attach an existing disk to the instance.
Then select 'CREATE' at bottom to provision new VM instance.
Note the External IP assigned.
2. **Create a New User on VM Instance**
Click on 'SSH' and allow SSH-in-browser to connect to devbuild-1 VM
Second option for SSH connection as shown [here](https://www.youtube.com/watch?v=fmh94mNQHQc):
- Generate your own private/public key pair, copy public key to settings of devbuild-1 instance and save private key on your local machine.
- Connect to devbuild-1 instance using putty
Change password for root user
```bash
sudo passwd
```
Switch to root user.
Add new user to sudo group. In this case new user is 'odennav-admin'
```bash
sudo adduser odennav-admin
sudo usermod -aG sudo odennav-admin
```
You'll be prompted to set a password and provide additional information about the new
user, such as full name, work phone, etc. This information is optional. Press 'Enter'
to skip each prompt.
```bash
Test sudo privileges by switching to new user
su - odennav-admin
sudo ls /root
```
You'll notice prompt to enter your user password.
To disable this prompt for every sudo command, implement the following:
Add sudoers file for odennav-admin
```bash
cd /etc/sudoers.d/
echo "odennav-admin ALL=(ALL) NOPASSWD: ALL" > odennav-admin
```
Set permissions for sudoers file
```bash
chmod 0440 odennav-admin
```
3. **Confirm Git is installed**
```bash
git version
```
If not available, install the package
```bash
sudo apt update
sudo apt install git -y
git version
```
Clone git repo codebase
```bash
git clone https://github.com/odennav/large-scale-app-micros-gcp.git
```
4. **Complete Image Build of System Components**
Enable execution of all bash scripts in large-scale-app-micros-gcp/ directory
```bash
cd large-scale-app-micros-gcp/
sudo find ~/large-scale-app-micros-gcp/ -type f -name "*.sh" -exec sudo chmod +x {} \;
```
Install docker and docker-compose
```bash
cd large-scale-app-micros-gcp/bin
./install-docker.sh
```
Confirm docker is installed
```bash
docker version
```
Trigger a complete build and build the entire codebase.
-Building codebase
-Staging them to a stage directory
-Building docker images required
```bash
./build-system.sh
```
5. **Setup Container Registry and Push images to Google Cloud Registry**
Before we can upload created images to google cloud registry, we have to authorize host docker in devbuild-1 VM instance.
```bash
cd ~/large-scale-app-micros-gcp/bin
./gcp-authorize-docker.sh
```
During remote authentication to gcloud CLI, you'll be asked to copy link in your browser because Google SDK wants to access your google account. Allow access, copy authorization code and paste in SSH-in-browser.
It locates all images starting with 'ntw', then tags them before pushing to registry
Ensure registry zone in script is the same region as your devbuild-1 instance and is also a region in either US, Europe or Asia.
**REGISTRY_HOST**=eu.gcr.io due to VM instance and Kubernetes cluster provisioned in European region.
```bash
cd ~/large-scale-app-micros-gcp/kubernetes
./gcp-push-images.sh
```
6. **Create Kubernetes Cluster on Google Cloud**
Go to 'Compute' section and click 'Kubernetes Engine'.
Enable Kubernetes API
Click on 'CREATE' under section of Kubernetes clusters.
Choose 'GKE Standard' or switch to 'STANDARD CLUSTER'.
Name your kubernetes cluster and select 'Regional' for Location type, to have our system in multiple zones within a region.
Select same region for devbuild-1 VM instance
-----
**Change number of nodes per zone to 1 in 'Node pool details', so we have 3 machines in different zones.**
-----
**For each node in pool:**
- Select E2 machine type
- 6 vCPU, 2 cores each
- 12GB memory total
- Boot disk size of 50GB
- Enable nodes on spot VMs(reduces monthly cost)
-----
**Click 'CREATE' at bottom to start process of creating kubernetes cluster.**
Note cluster created with total of 18 vCPUs, 36GB memory and nodes from each zone.
-----
7. **Kubernetes Environment Configuration for System**
To host our system on this gcp k8s cluster, we'll have to create k8s config and services.
Install kubectl
kubectl enables devbuild-1 VM instance to communicate with the kubernetes cluster
```bash
cd ~/large-scale-app-micros-gcp/kubernetes
./gcp-install-kubectl.sh
```
**Configure kubectl command line access to k8s cluster**
Locate your cluster created in 'Kubernetes Engine' section on GCP and click on ':' to view options, then click 'Connect'.
Copy and paste gcloud command to devbuild-1 terminal.
-----
Confirm kubectl has access to nodes in pool
```bash
kubectl get nodes
```
Create namespaces
```bash
cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env
kubectl apply -f 0-namespaces.yaml
```
Apply config map
Set environment variables for each component in system
```bash
cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env
kubectl apply -f 1-config-map.yaml
```
Apply secrets config
Set environment variables for secrets to postgresql databases
```bash
cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env
kubectl apply -f 2-secrets-map.yaml
```
You can delete 2-secrets.yaml file after secrets resource is created and stored in k8s cluster.
Apply resources limit
Set CPU and Memory resource limits to pods created in services namespace
```bash
cd ~/large-scale-app-micros-gcp/kubernetes/config/0-env
kubectl apply -f 3-resources.yaml
```
8. **Kubernetes Volume Configuration**
Configure persistent volumes for each node in node pool
```bash
cd ~/large-scale-app-micros-gcp/kubernetes/volume
./kube-volumes.sh
```
Set up all System Workloads on Kubernetes Cluster
Identify all config yaml files and create all resources required to start the app on kubernetes cluster
kube-deploy.sh script creates all services and deployments configured in the yaml files identified.
Ensure variable **REGISTRY_HOST**=eu.gcr.io is set in bash script due to regional setup of clusters and registry
```bash
cd ~/large-scale-app-micros-gcp/kubernetes
./kube-deploy.sh
```
Go to 'Workloads' section under 'Kubernetes Engine' and view pods created and running.
Confirm all namespaces have been created
```bash
kubectl get ns
```
Check for webapp and spa services in ui namespace
```bash
kubectl get svc -n ui
```
9. **Access both Webapp and SPA**
Create 'allow-ssh' firewall rule in your vpc network settings.
Search for 'VPC Network' in Google cloud platform and click on 'Firewall' on left-side bar. Select 'CREATE FIREWALL RULE'.
- Enter name of firewall rule - 'allow-sys'
- Direction of traffic is 'Ingress'
- Use '0.0.0.0/0' as source of IPv4 ranges
- Under section for 'Protocol and ports' select 'TCP' and insert Nodeports configured for both webapp and spa, 32100 and 32105.
- **Click on 'CREATE' at bottom to create VPC firewall rule.**
Access Login page from your browser on local machine.
Please note this option opens ports 32100 and 32105 on all VM instances in your VPC network.
Effectively, you can select the firewall rules initially created for each of the kubernetes cluster nodes and edit firewall settings by including Nodeports among TCP ports allowed.
10. **Access System Components for Centralized Logging, Tracing and Resource Monitoring**
Add the following Nodeports as TCP ports in firewall rule created above
- **32101 for Kibana/Elasticsearch**
- **32102 for Uber Jaeger**
- **32103 for Prometheus**
- **32104 for RabbitMQ**
## Shutdown Kubernetes Cluster
To remove all resources for kubernetes cluster and stop incurring charges.
```bash
cd ~/large-scale-app-micros-gcp/kubernetes
./kube-deploy.sh ./config/ delete
```
Delete persistent volumes created
```bash
cd ~/large-scale-app-micros-gcp/kubernetes
./kube-volumes.sh delete
```
Go to 'Clusters' section in 'Kubernetes Engine' product and delete cluster.
Click on Actions ':' menu and select 'Delete'
## Special Credits
Special thanks to [Anurag Yadav](https://www.newtechways.com/).
### Contributions ###
* Writing tests
* Code review
Enjoy!