https://github.com/deeptiman/go-cache-kubernetes

The web application is a Data Caching Service designed and implemented using microservices architecture.
https://github.com/deeptiman/go-cache-kubernetes

docker docker-compose golang golang-library gomongo kafka kafka-consumer kafka-producer kubernetes kubernetes-cluster kubernetes-deployment microservices-architecture mongodb redis redis-cache redis-client

Last synced: 7 days ago
JSON representation

The web application is a Data Caching Service designed and implemented using microservices architecture.

• Host: GitHub
• URL: https://github.com/deeptiman/go-cache-kubernetes
• Owner: Deeptiman
• License: mit
• Created: 2020-08-23T07:01:35.000Z (about 4 years ago)
• Default Branch: master
• Last Pushed: 2021-06-14T21:48:57.000Z (over 3 years ago)
• Last Synced: 2024-06-21T19:08:30.950Z (5 months ago)
• Topics: docker, docker-compose, golang, golang-library, gomongo, kafka, kafka-consumer, kafka-producer, kubernetes, kubernetes-cluster, kubernetes-deployment, microservices-architecture, mongodb, redis, redis-cache, redis-client
• Language: Go
• Homepage: https://github.com/Deeptiman/go-cache-kubernetes
• Size: 239 KB
• Stars: 13
• Watchers: 3
• Forks: 10
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# go-cache-kubernetes

     

	

	 

	 



The web application is a Data Caching service designed and implemented using microservices architecture. The cloud deployment environments are used Kubernetes, Docker, and written in Go programming language. The application also uses a MongoDB as NoSQL database with Redis in-memory database for the caching services.


## Features

 - **MongoDB** is implemented to perform several database operations. The installation can be done using the go dependency module.

		

		go get go.mongodb.org/mongo-driver/mongo

	  https://github.com/mongodb/mongo-go-driver

 - **Redis Cache** is implemented to integrate the data caching in the application. So, the go-redis will cache the second GET request while reading the user details.

 

		go get github.com/go-redis/redis/v8

	  https://github.com/go-redis/redis

 - **Kafka Message Broker**: The confluent-kafka-go is used as a Go client library for Kafka message broker. The library will provide **Producer** and **Consumer** architecture to stream messages to the user for a subscribed topic. So, there will be two REST APIs that the user can use for Producing the messages reading from MongoDB and Consuming or Reading messages from the message broker.

		go get github.com/confluentinc/confluent-kafka-go/kafka

	https://github.com/confluentinc/confluent-kafka-go

	Note: It's recommended to install **confluent-kafka-go v1.4.0**, as the **librdkafka** will come with the bundle and no need to install separately.

## Kubernetes tools

Kubernetes provides several tools that can be useful to setup Kubernetes in the local environment.

 - **minikube** tool will run a single-node Kubernetes cluster running inside a Virtual Machine. Virtualization has to be supported in the computer and Hypervisor needed to be enabled.

	    

	 **Installation**

	    follows with the Hypervisor installation and [Hyperkit](https://minikube.sigs.k8s.io/docs/drivers/hyperkit/) is the recommended virtualization toolkit.   

		$ sudo install minikube

		

		$ minikube start

	https://kubernetes.io/docs/setup/learning-environment/minikube/

	 

	

		

 - **kubectl** command-line tool will work to manage a Kubernetes cluster. The tool will be used to deploy, create, analyze, inspect pods that are running under a Kubernetes cluster.

	**Installation**

	

	```curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" ```

	https://kubernetes.io/docs/tasks/tools/install-kubectl/

## Build the Docker images

The application uses Docker for container-based development. The docker image gets published as a public repository at Docker Hub.

 - **Build the image**

		$ docker build -t go-cache-kubernetes-v1 .

 - **Tag the image**

		$ docker tag go-cache-kubernetes-v1 deeptiman1991/go-cache-kubernetes-v1:1.0.0

 - **Login to docker hub**

		$ docker login

		Type Username and Password to complete the authentication

 - **Push the image to docker hub**

		$ docker push deeptiman1991/go-cache-kubernetes-v1:1.0.0

## Kubernetes Deployment

There will be several deployments, services that need to be running in the cluster as a Pod. The creation of a Pod requires a YAML file that will specify the kind, spec, containerPort, metadata, volume, and more. So, these parameters will be used to provide resources to the Kubernetes cluster.

**Start minikube** to begin the deployment process start the minikube 

	

	$ minikube start

Kubernetes Secret Management
 

The application will be using few MongoDB credentials for database connection. So the username and password will be secure using HashiCorp Vault as static secrets.





 
More details explanation on HashiCorp Vault, please check my Medium article : Secrets in Kubernetes and HashiCorp Vault 

 


 
  Implement the Vault Envs in the pod deployment  

		

	    spec:

	      serviceAccountName: vault

	      containers:

	      - name: go-cache-kubernetes-container-poc

		image: deeptiman1991/go-cache-kubernetes-v1:1.0.0

		env:

		- name: VAULT_ADDR

		  value: "http://vault:8200"

		- name: JWT_PATH

		  value: "/var/run/secrets/kubernetes.io/serviceaccount/token"

		- name: SERVICE_PORT

		  value: "8080"		

  

	

	

		YAML

		go-cache-poc-app.yaml

	

	

	

	

So, now SECRET_USERNAME & SECRET_PASSWORD environment variables can be used to connect to the MongoDB database from the application. 

    

Deploy PersistentVolumeClaim
 


This will allocate a volume of 1GB storage in the cluster




	

	

	

		Name

		go-cache-poc-pvc

	

	

		Kind

		PersistentVolumeClaim

	

	

		YAML

		go-cache-poc-pvc.yaml

	

	

	

    $ kubectl apply -f go-cache-poc-pvc.yaml

Deploy Go Web App
 


This will load the web app Docker image in the cluster.	




	

	

	

		Name

		go-cache-poc

	

	

		Kind

		Deployment

	

	

		YAML

		go-cache-poc-app.yaml

	

	

	

    $ kubectl apply -f go-cache-poc-app.yaml	

**Verify**

	$ kubectl get deployments

	NAME           		      READY   UP-TO-DATE   AVAILABLE   AGE

	go-cache-kubernetes-app-poc   1/1     1            1           14s	

	  

	There is only one pod is running under this deployment.

Deploy Go Web App Service


This service will create an external endpoint using a LoadBalancer.




	

	

	

		Name

		go-cache-poc-service

	

	

		Kind

		Service

	

	

		YAML

		go-cache-poc-svc.yaml

	

	

	

	$ kubectl apply -f go-cache-poc-svc.yaml

	

**Verify**

	$ kubectl get services

 
Deploying MongoDB ReplicaSet as a Kubernetes StatefulSet


Kubernetes provides a feature that will allow us to create a stateful application in the cluster. There will be a storage class and services running under the cluster that will allow the databases to connect with services and store records in their persistent database.




More details explanation on MongoDB StatefulSet, please check my Medium article : MongoDB StatefulSet in Kubernetes 




 MongoDB service will create the Mongo services in the cluster.

	

	

	

		Name

		mongodb-service

	

	

		Kind

		Service

	

	

		YAML

		mongodb-service.yaml

	

	

	


	   $ kubectl apply -f mongodb-service.yaml

	 

**MongoDB StatefulSet** will create the StatefulSet app in the cluster.

	

	

	

		Name

		mongod

	

	

		Kind

		StatefulSet

	

	

		YAML

		mongodb-statefulset.yaml

	

	

	

	   $ kubectl apply -f mongodb-statefulset.yaml

#### Define the Administrator

There will be three mongo containers in the cluster. We need to connect to anyone of them to define the administrator.

Command to exec

	 $ kubectl exec -it mongod-0 -c mongod-container bash

	-it: mongo app name

	-c:  mongo container name

Bash

  	$ hostname -f

	     

	 mongod-0.mongodb-service.default.svc.cluster.local

Mongo Shell

	 $ mongo

Type to the following query to generate the replica set

	> rs.initiate({_id: "MainRepSet", version: 1, members: [

		{ _id: 0, host : "mongod-0.mongodb-service.default.svc.cluster.local:27017" }

	]}); 	  	

	 	 

then verify	

			    	

	> rs.status();

Now create the Admin user

		> db.getSiblingDB("admin").createUser({

		      user : "admin",

		      pwd  : "admin123",

		      roles: [ { role: "root", db: "admin" } ]

		 });

		 

So, now the MongoDB is complete setup with ReplicaSet and with an Administrator for the database.

Deploy Redis in Kubernetes

There will be several steps to follow for deploying Redis into the Kubernete cluster. 





	
Download Docker images for Redis

	
 

   	   $ docker run -p 6379:6379 redislabs/redismod	

	
	

	 
Redis Deployment

		

		  

			

				Name

				redismod

			

			

				Kind

				Deployment

			

			

				YAML

				redis-deployment.yaml

			

		  

		

	 	
$ kubectl apply -f redis-deployment.yaml

	

   
Redis Service

   		

		  

			

				Name

				redis-service

			

			

				Kind

				Service

			

			

				YAML

				redis-service.yaml

			

		  

		

	
$ kubectl apply -f redis-service.yaml
	

	
Deploy the redismod image

	


	  $ kubectl run redismod --image=redislabs/redismod --port=6379

	

	
Expose the deployment

	


	  $ kubectl expose deployment redismod --type=NodePort

	

	
Now, check for the Redis Connection

	


	  $ redis-cli -u $(minikube service --format "redis://{{.IP}}:{{.Port}}" --url redismod)

	

	 You must be getting an ip address with a port that can be used as a connection string for Redis


Deploy Kafka in Kubernetes

There will be a deployment of ZooKeeper, Kafka Service, and running kafka/zookeeper server script. Please install Apache Kafka in your local machine and gcloud.




Zookeeper

There will be deployment and service similar to the other Pods running in the cluster.

	 
zookeeper-deployment

	 

		  

			

				Name

				zookeeper-app

			

			

				Kind

				Deployment

			

			

				YAML

				zookeeper-deployment.yaml

			

		  

		

	 	

	$ kubectl apply -f zookeeper-deployment.yaml

		    

 
zookeeper-service

 

		  

			

				Name

				zookeeper-service

			

			

				Kind

				Service

			

			

				YAML

				zookeeper-service.yaml

			

		  

		

	 	

    $ kubectl apply -f zookeeper-service.yaml

		    

Kafka

kafka-service

	

		  

			

				Name

				kafka-service

			

			

				Kind

				Service

			

			

				YAML

				kafka-service.yaml

			

		  

		

		   

	         

	 	

 	$ kubectl apply -f kafka-service.yaml


kafka-replication-controller

	

		  

			

				Name

				kafka-repcon

			

			

				Kind

				Deployment

			

			

				YAML

				kafka-repcon.yaml

			

		  

		


	 	

	$ kubectl apply -f kafka-repcon.yaml

		

Start zookeeper/kafka server

   
zookeeper server

		  

	$ cd kafka/

	$~/kafka/ bin/zookeeper-server-start.sh config/zookeeper.properties	


   
kafka server

		 

	$ cd kafka/

	$~/kafka/ bin/kafka-server-start.sh config/server.properties

				


## Troubleshoot with kubectl

The kubectl is a very handy tool while troubleshooting application into the Kubernetes.

**Few useful commands**



	
 kubectl get pods 

	
 kubectl describe pods   

	
 kubectl logs   

	
 kubectl exec -ti  --bash 



## Swagger API documentation

The go-swagger toolkit is integrated for the REST APIs documentation. The API doc can be accessible via http://localhost:5000/docs

https://github.com/go-swagger/go-swagger

swagger.yaml


## More Info



	
 Getting started Kubernetes 

	
 Kubernetes Secret Managements 

	
 Hello Minikube



	
 Docker Documentation



	
 Swagger ReDoc





License

This project is licensed under the MIT License