{"id":22929949,"url":"https://github.com/genocs/enterprise-containers","last_synced_at":"2025-06-11T07:09:18.076Z","repository":{"id":44393490,"uuid":"509998638","full_name":"Genocs/enterprise-containers","owner":"Genocs","description":"End to end setup containers application using Docker Containers, Kuberneters orcherstrators","archived":false,"fork":false,"pushed_at":"2025-02-24T13:56:43.000Z","size":37606,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-01T16:54:15.654Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"PowerShell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Genocs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"open_collective":"genocs","custom":["https://www.buymeacoffee.com/genocs"]}},"created_at":"2022-07-03T11:04:37.000Z","updated_at":"2025-02-24T13:56:47.000Z","dependencies_parsed_at":"2025-04-01T16:58:18.146Z","dependency_job_id":null,"html_url":"https://github.com/Genocs/enterprise-containers","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fenterprise-containers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fenterprise-containers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fenterprise-containers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fenterprise-containers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Genocs","download_url":"https://codeload.github.com/Genocs/enterprise-containers/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Genocs%2Fenterprise-containers/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259219697,"owners_count":22823576,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-14T10:19:48.110Z","updated_at":"2025-06-11T07:09:18.044Z","avatar_url":"https://github.com/Genocs.png","language":"PowerShell","funding_links":["https://opencollective.com/genocs","https://www.buymeacoffee.com/genocs","https://www.buymeacoffee.com/genocs)!"],"categories":[],"sub_categories":[],"readme":"# Enterprise Kubernetes Cluster Setup\n\n[![GitHub](https://img.shields.io/github/license/Genocs/genocs-library?color=2da44e\u0026style=flat-square)](https://github.com/Genocs/enterprice-containers/blob/main/LICENSE)\n[![Discord](https://img.shields.io/discord/1106846706512953385?color=%237289da\u0026label=Discord\u0026logo=discord\u0026logoColor=%237289da\u0026style=flat-square)](https://discord.com/invite/fWwArnkV)\n[![Twitter](https://img.shields.io/twitter/follow/genocs?color=1DA1F2\u0026label=Twitter\u0026logo=Twitter\u0026style=flat-square)](https://twitter.com/genocs)\n\n---\n\n![Azure-KEDA](/images/Azure-KEDA.drawio.svg)\n\nSetup Kubernetes cluster to be production ready isn't a simple task. It requires to take in consideration many topics.\n\nIn this walkthrough we will setup the steps required to have a Kubernetes cluster up and running.\n\nThis proposal is thought as to be used on Azure, even though most of the options could be reused for any cloud provider like Google Cloud or AWS.\n\n## Prerequisites\n\n- **Azure Subscription** to create AKS cluster\n- **kubectl** logged into kubernetes cluster\n- **Powershell**\n- **Postman**\n- **Helm**\n- **DockerHub account** (optional)\n\n## Introduction\n\nThe setup process can be spitted into different steps:\n\n- Bare components\n- Security\n- Scaling\n- Monitoring\n- Application\n\n## Setup - Overview\n\nDuring this step we are going to setup Kubernetes cluster tackling the following components:\n\n- Setup private Docker images repository\n- Setup the Kubernetes cluster\n- Setup the vault for secrets\n- Secure secrets\n- Setup networking\n- Setup autoscaler\n\n### Monitoring\n\nThe implementation will monitor both infrastructure and application side.\n\nThe monitoring will be implemented using different approach. Some are open source, other don't.\n\nOpen-source Components uses in this demo:\n\n- [Grafana](https://grafana.com/)\n- [Jaeger](https://www.jaegertracing.io/)\n- [Promotheus](https://prometheus.io/)\n\n**NOTE:** The open-source components have the entreprise version. It requires to have active subscription if you plan to use them.  \n\n### Security and Networking\n\n\u003cimg src=\"./images/k8s-architecture.png\" /\u003e\n\nSecurity and Networking context will implement all the components that allows to handle:\n\n- routing (forward requests to the services throughout a reverse proxy)\n- termination (secure APIs calls by SSL or TLS)\n- throttling\n\nMain components are:\n\n- Public IP\n- Vnet and Subnet\n- Api Gateway Ingress Controller\n\nThere are various products out of the box that can be used to implement the solution. All of them have PROS and CONS.\n\nThe most used ones are:\n\n- [Azure AGIC](https://learn.microsoft.com/en-us/azure/application-gateway/ingress-controller-install-new)\n- [NGNIX](https://www.nginx.com/)\n- [Kong](https://konghq.com/)\n\nIn this solution the first choice is based on `Kong`.\n\nAuthentication and authorization are implemented using `OAuth2`. `Kong` provides plugin that streamline the implementation.\n\n- [OAuth2](https://oauth.net/2/)\n- [OpenId](https://openid.net/connect/)\n\n## Autoscaler\n\nThere are multiple options for scaling Kubernetes and containers in general.\n\nHere `(KEDA) Kubernetes-based Event Driven Autoscaling` will be used.\n\nRabbitMQ is used as the event source.\n\nIf you wish to use Kubernetes cluster apart from AKS, you can skip the `Step 2.1` of provisioning the cluster and [install KEDA](https://github.com/kedacore/keda#setup) on your own Kubernetes cluster.\n\nSimilarly, if you do not wish to execute the PowerShell scripts, you can execute the commands which are part of those scripts manually.\n\n## Code organization\n\n### [docker-compose](docker-compose)\n\nContains the docker compose files to install a bunch of enterprise level tools.\n\nPlease refer to this [README](./docker-compose/README.md) for details.\n\n### [docs](docs)\n\nContains some useful documentation like:\n\n- [aks-preview](./docs/aks-preview.md)\n- [azure-developer-community](./docs/azure-developer-community.md)\n- [minikube-wsl2](./docs/minikube-wsl2.md)\n- [monitoring-and-observability](./docs/monitoring-and-observability.md)\n\n### [src](src)\n\nContains the source code for a hypothetical application.\n\nIt contains the source code for a model class used by two services:\n\n- The **Producers**: a WebApi that send messages to RabbitMQ cluster\n- The **Internal Service**: a WebApi that can be used only inside the cluster\n- The **Consumers**: a Worker listening RabbitMQ messages.\n\n`Genocs.KubernetesCourse.WebApi` contains the code for generating the messages which are published to a RabbitMQ queue, as well as to call an internal WebApi.\n\n`Genocs.KubernetesCourse.Internal.WebApi` contains the code for the WebApi used only inside the cluster.\n\n`Genocs.KubernetesCourse.Worker` contains the consumer code processing RabbitMQ messages.\n\nBoth the Producer and Consumer uses the common data model.\n\nIn order to build these using Dockerfile:\n\n- [Genocs.KubernetesCourse.WebApi](./src/Genocs.KubernetesCourse.WebApi.dockerfile)\n- [Genocs.KubernetesCourse.Internal.WebApi](./src/Genocs.KubernetesCourse.Internal.WebApi.dockerfile)\n- [Genocs.KubernetesCourse.Worker](./src/Genocs.KubernetesCourse.Worker.dockerfile)\n\nYou can build them and they are ready to be pushed to:\n\n- Azure Container Registry [docker-compose ACR](./src/docker-compose-acr.yaml)\n- DockerHub [docker-compose Dockerhub](./src/docker-compose-dockerhub.yaml)\n\nThe docker images can be built and push to dockerhub using the following commands:\n\n``` ps\n# Build\ndocker-compose -f .\\src\\docker-compose-dockerhub.yaml build\n\n# Push\ndocker-compose -f .\\src\\docker-compose-dockerhub.yaml push\n```\n\n**NOTE:** Please update the yaml files with the correct image version before use them.\n\n### [Powershell](Powersehll)\n\nContains the helper PowerShell scripts to:\n\n- Provisioning the AKS cluster\n- To proxy into the Kubernetes control plane\n- To deploy the application\n- To delete the application\n- To delete the resource group\n\n**NOTE: AKS is expensive to keep alive**\n\n### [k8s](k8s)\n\nContains Kubernetes manifest files for deploying the Producer, Consumer and the Autoscaler components to the Kubernetes cluster.\n\n### [helm](helm)\n\nContains the Helm RBAC enabling yaml file which add the Cluster Role Binding for RBAC enabled Kubernetes cluster.\n\nThis was required before Helm 3.0 for the Tiller service. With helm 3.0, Tiller is no longer required.\n\n### [terraform](terraform)\n\nContains the terraform Kubernetes cluster setup. The setup is primary meant to be used on Azure. Anyway, deploy on other cloud Provider is quite straightforward.\n\n### [skaffold](skaffold)\n\nThe Skaffold and Kaniko Kubernetes cluster setup.\n\nThis allows to setup Kubernetes cluster on Google Cloud.\n\n## **SECTION 2 - Setup**\n\n1. Install Azure Container Registry ACR\n2. Install Kubernetes Cluster AKS\n3. Install Azure Key Vault AKV\n4. Setup Network Infrastructure\n5. Deploy Azure Key Vault Secret\n6. Deploy RabbitMQ node\n7. Deploy MongoDB node\n8. Deploy Prometheus \u0026 Grafana\n9. Deploy Jaeger\n10. Deploy KEDA Autoscaler\n11. Deploy Application\n12. Deploy Application AutoScaler\n13. Get list of the resources\n\n### 2.1 Install Azure Container Registry ACR\n\nThe **ACR** (Azure Container Registry) allows to store Docker images inside a private repositoy.\n\nRun [initializeACR](/Powershell/initializeACR.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\initializeACR.ps1\n```\n\n### 2.2 Install Kubernetes Cluster AKS\n\nRun [initializeAKS](/Powershell/initializeAKS.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\initializeAKS.ps1\n```\n\n### 2.3 Install Azure Key Vault AKV\n\nThe **AKV** Azure Key Vault is used to store every secret used by the application in a safe place.\n\nSecret data are:\n\n- connection strings\n- API Key\n- certificates\n- public/private keys\n\nRun [initializeAKV](/Powershell/initializeAKV.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\initializeAKV.ps1\n```\n\n### 2.4 Setup Network Infrastructure\n\nThe **AKS** will use a network infrastructure composed by:\n\n- Public IP\n- VNET\n- AGIC (Application Gateway Ingress Controller)\n\nThe Poweshell script will *'Create peering beetween AGIC and AKS and viceversa'* as well.\n\nRun [initializeNetwork](/Powershell/initializeNetwork.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\initializeNetwork.ps1\n```\n\n### 2.5 Deploy Azure Key Vault Secret\n\nThis step shows how to setup a script file to push Secrets inside AKV.\n\nRun [deployAKV-secrets](/Powershell/deployAKV-secrets.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\deployAKV-secrets.ps1\n```\n\n### 2.6 Deploy RabbitMQ node\n\nThis step shows how to setup a RabbitMQ node inside AKS.\n\n***Not reccomanded for Production***\n\nRun [deployRabbitMQ](/Powershell/deployRabbitMQ.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\deployRabbitMQ.ps1\n```\n\n### 2.7 Deploy MongoDB node\n\nThis step shows how to setup a MongoDb instance inside AKS.\n\n***Not reccomanded for Production***\n\nRun [deployMongoDB](/Powershell/deployMongoDB.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\deployMongoDB.ps1\n```\n\n### 2.8 Deploy Prometheus \u0026 Grafana\n\nThis step shows how to setup a Prometheus and Grafana inside AKS.\n\nRun [deployPrometheus](/Powershell/deployPrometheus.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\deployPrometheus.ps1\n```\n\n### 2.9 Deploy Jaeger\n\nThis step shows how to setup a Jaeger distributed Tracing inside AKS.\n\nRun [deployJaeger](/Powershell/deployJaeger.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\deployJaeger.ps1\n```\n\n### 2.10 Deploy KEDA Autoscaler\n\nThis step shall install the KEDA autoscaler inside AKS.\n\nRun [deployKEDA](/Powershell/deployKEDA.ps1) powershell script with default values from root directory.\n\n``` PS\n.\\Powershell\\deployKEDA.ps1\n```\n\nVerify KEDA is installed correctly on the Kubernetes cluster.\n\n``` bash\nkubectl get all -n keda\n```\n\n### 2.11 Deploy Application\n\nDeploy External WebAPI (Producer), Internal WebAPI and Backgroud Worker Consumer.\n\nExecute the powershell script.\n\n``` PS\n# Use this to setup the application with Secret coming from file as Opaque secret\ncd Powershell \n.\\deployApplications-AKS-SecretFile.ps1\ncd ..\n```\n\n``` PS\n# Use this to deploy the application using secret coming from Key Vault\ncd Powershell \n.\\Powershell\\deployApplications-AKS.ps1\ncd ..\n```\n\nThe `deployApplications-AKS` powershell script deploys the RabbitMQConsumer and RabbitMQProducer in the correct order.\n\nAlternately, all the components can also be deployed directly using the `kubectl` apply command recursively on the k8s directory as shown below:\n\n``` bash\n# Run the kubectl apply recursively on k8s directory\nkubectl apply -R -f .\n```\n\n### 2.12 Deploy Application AutoScaler\n\nExecute the `deployAutoScaler.ps1` powershell script.\n\n``` PS\ncd Powershell \n.\\deployAutoScaler.ps1\ncd ..\n```\n\n**Note:** The default options can be overwritten by passing arguments to the initializeAKS script. In the below example, we are overriding the number of nodes in the AKS cluster to 4 instead of 3 and resource group name as `kedaresgrp`.\n\n``` PS\n.\\Powershell\\initilaizeAKS `\n    -workerNodeCount 4 `\n    -resourceGroupName \"kedaresgrp\"\n```\n\nIf you do not wish to run the individual PowerShell scripts, you can run one single script which will deploy all the necessary things by running the above scripts in correct order.\n\n``` PS\n.\\Powershell\\deployAll.ps1\n```\n\n### 2.13 Get list of the resources\n\n General purpose command to get info about K8S resources\n\n``` bash\n# General purpose command to get all resources\n# -A to get all namespaces\n# -w to watch for changes\n\n# Get services\nkubectl get svc\n\n# Get secrets\nkubectl get ingress\n\n# Get pod\nkubectl get pod\n\n# Get deployments\nkubectl get deployment\n\n# Get secrets\nkubectl get secret\n\n# Get custom resource definition\nkubectl get crd\n```\n\n![List of all Kubernetes services](/images/all-services.png)\n\nAs we can see above, RabbitMQ service is available within the Kubernetes cluster and it exposes `4369`, `5672`, `25672` and `15672` ports. We will be using `15672` port to map to a local port.\n\nAlso note the public `LoadBalancer` IP for the Producer Service. In this case the IP is **`52.139.237.252`**.\n\n**Note:** This IP will be different when the services are redeployed on a different Kubernetes cluster.\n\n### 2.11 Watch for deployments\n\nThe rabbitmq `ScaledObject` will be deployed as part of the deployment. Watch out for the deployments to see the changes in the scaling as the number of messages increases\n\n``` bash\nkubectl get deployment -w\nkubectl get deploy -w\n```\n\n![List of all Kubernetes services](/images/initial-deploy-state.png)\n\nInitially there is 1 instance of rabbitmq-consumer and 2 replicas of the rabbitmq-producer (Producer) deployed in the cluster.\n\n### 2.12 Browse RabbitMQ Management UI\n\nRabbitMQ Management UI is enabled by port forwarding.\n\nIn order to do this, open a bash shell and run the command.\n\nPlease keep the shell open to keep open the port forwarding.\n\n``` bash\nkubectl port-forward svc/rabbitmq 15672:15672\n```\n\nOpen the web browser:\n[rabbitmq](http://localhost:15672/)\n\nLogin to the management UI using credentials as `guest` and `guest`.\n\nRemember that these were set during the installation of RabbitMQ services using Helm. If you are using any other user, please update the username and password accordingly.\n\n### 2.13 Generate load using `Visual Studio code Extension`\n\nTBV\n\n### 2.14 Generate load using `Postman`\n\nI am using [Postman](https://www.getpostman.com/) to submit a POST request to the API which generates 2000 messages onto a RabbitMQ queue named `hello`. You can use any other command line tool like CURL to submit a GET request.\n\nUse the `EXTERNAL-IP -52.139.237.252` with port `8080` to submit a GET request to the API. http://52.139.237.252:8080/api/TechTalks/Generate?numberOfMessages=2000\n\n![postman success](/images/postman-get-request.png)\n\nNote that we are setting the number of messages to be produced by Producer as 2000 in this case. You can change the number to any other integer value.\n\nAfter building the GET query, hit the blue `Send` button on the top right. If everything goes fine, you should receive a `200 OK` as status code.\n\n![postman success](/images/postman-success.png)\n\nThe Producer will produce 2000 messages on the queue named `hello`. The consumer is configured to process `10` messages in a batch. The Consumer take the message, deserialize it and send the ACK to the message broker.\n\n### 2.15 Auto-scaling in action\n\nSee the number of containers for consumer grow to adjust the messages and also the drop when messages are processed.\n\n![autoscaling consumers](/images/pods-and-deployments-autoscaled.png)\n\nOn the left-hand side of the screen you can see the pods auto scaled and on the right we see the deploymnets autoscaled progressively to 2, 4, 8, 16 and 30.\n\nWhile the messages are being processed, we can also observe the RabbitMQ management UI.\n\n![autoscaling consumers](/images/RabbitMQ-managementUI.PNG)\n\nOur consumer processes 50 messages in a batch by prefetching them together. This can be verified by looking at the details of the consumers.\n\n![Prefetch messages](/images/rabbitMQ-prefetch.PNG)\n\nOnce all the messages are processed, KEDA will scale down the pods and the deployments.\n\n![autoscaled down consumers](/images/pods-and-deployments-scaled-down.png)\n\nList Custom Resource Definition\n\n``` bash\nkubectl get crd\n```\n\n![autoscaled down consumers](/images/KEDA-CRD.PNG)\n\nAs part of the KEDA installation, ScaledObject and TriggerAuthentications are deployed on the Kubernetes cluster.\n\n### 2.16 Delete all resources\n\n``` PS\n.\\Powershell\\deleteRG.ps1\n```\n\n## Resource Groups\n\nThis demo will create all the Azure Resource inside:\n\n- rg-aks-genocs\n- rg-agic-genocs\n- MC_rg-aks-genocs_aks-genocs_westeurope\n- DefaultResourceGroup-WEU\n\n## 1. rg-aks-genocs\n\n- aks-genocs - Kubernetes service\n- genocscontainer - Container registry\n- kv-genocsakst - Key vault\n\n## 2. rg-agic-genocs\n\n- agic-genocs - Application gateway\n- agic-pip - Public IP address\n- agic-vnet - Virtual network\n\n## 3. MC_rg-aks-genocs_aks-genocs_westeurope\n\n- 4c0f07f9-b962-478a-a875-be77a79462dc      Public IP address           West Europe\n- aks-agentpool-33168902-nsg                Network security group      West Europe\n- aks-agentpool-33168902-routetable         Route table                 West Europe\n- aks-genocs-agentpool                      Managed Identity            West Europe\n- aks-nodepool1-12518635-vmss               Virtual machine scale set   West Europe\n- aks-vnet-33168902                         Virtual network             West Europe\n- kubernetes                                Load balancer               West Europe\n- omsagent-aks-genocs                       Managed Identity            West Europe\n- pvc-49df2627-0ef7-4264-8441-f0e36b629bf2  Disk                        West Europe\n\n## 4. DefaultResourceGroup-WEU\n\n## Community\n\n- Discord [@genocs](https://discord.com/invite/fWwArnkV)\n- Facebook Page [@genocs](https://facebook.com/Genocs)\n- Youtube Channel [@genocs](https://youtube.com/c/genocs)\n\n## Contributors\n\nSubmit your PR and join the list!\n\n\u003ca href=\"https://github.com/Genocs/enterprise-containers/graphs/contributors\"\u003e\n  \u003cimg src=\"https://contrib.rocks/image?repo=Genocs/enterprise-containers\" /\u003e\n\u003c/a\u003e\n\n## License\n\nThis project is licensed with the [MIT license](LICENSE).\n\n## Support :star:\n\nHas this project helped you learn something New? or Helped you at work?\nHere are a few ways by which you can support.\n\n- Leave a star! :star:\n- Recommend this awesome project to your colleagues. 🥇\n- Do consider endorsing me on LinkedIn for ASP.NET Core - [Connect via LinkedIn](https://www.linkedin.com/in/giovanni-emanuele-nocco-b31a5169/) 🦸\n- Or, If you want to support this project in the long run, [consider buying me a coffee](https://www.buymeacoffee.com/genocs)! ☕\n\n\u003cbr\u003e\n\u003ca href=\"https://www.buymeacoffee.com/genocs\"\u003e\u003cimg width=\"250\" alt=\"black-button\" src=\"https://user-images.githubusercontent.com/31455818/138557309-27587d91-7b82-4cab-96bb-90f4f4e600f1.png\"\u003e\u003c/a\u003e\n\n## Acknowledgements\n\nA lot of people inspired me and provided unevaluable amount of information:\n\nHere some of them:\n\n- [NileshGule](https://github.com/NileshGule/)\n- [DevMentors](https://github.com/devmentors/)\n\n## References\n\n- [Hands-on-Kubernetes-on-Azure-Third-Edition](https://github.com/PacktPublishing/Hands-on-Kubernetes-on-Azure-Third-Edition)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenocs%2Fenterprise-containers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgenocs%2Fenterprise-containers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgenocs%2Fenterprise-containers/lists"}