{"id":26279566,"url":"https://github.com/utwo/k8s-playground","last_synced_at":"2025-03-14T14:15:28.118Z","repository":{"id":272334483,"uuid":"916225150","full_name":"Utwo/k8s-playground","owner":"Utwo","description":"Gitops repo with k8s tools simulating a real enterprise project","archived":false,"fork":false,"pushed_at":"2025-03-13T19:51:39.000Z","size":357,"stargazers_count":0,"open_issues_count":11,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-13T20:36:55.276Z","etag":null,"topics":["argocd","gitops","k8s"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Utwo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-13T17:31:24.000Z","updated_at":"2025-03-12T09:07:31.000Z","dependencies_parsed_at":"2025-01-13T19:32:16.367Z","dependency_job_id":"d044664c-27bd-44d9-849e-1fed3566a0e6","html_url":"https://github.com/Utwo/k8s-playground","commit_stats":null,"previous_names":["utwo/k8s-playground"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Utwo%2Fk8s-playground","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Utwo%2Fk8s-playground/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Utwo%2Fk8s-playground/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Utwo%2Fk8s-playground/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Utwo","download_url":"https://codeload.github.com/Utwo/k8s-playground/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243589330,"owners_count":20315471,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["argocd","gitops","k8s"],"created_at":"2025-03-14T14:15:27.318Z","updated_at":"2025-03-14T14:15:28.087Z","avatar_url":"https://github.com/Utwo.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# 🚀 K8s Sandbox\n\nA GitOps repository with Kubernetes tools simulating a real enterprise project.\n\n---\n\n## 🔥 Features\n\n✅ ***Implemented:***\n- **Development \u0026 Cluster Setup:**\n  - Devbox\n  - K3d with two clusters (control and apps)\n  - Sops secrets for secrets management\n  - Ingress Nginx\n- **GitOps \u0026 Deployment:**\n  - Argocd with SSO\n  - ApplicationSet for preview PR\n  - Argo Rollouts (Blue-Green \u0026 Canary Deployments)\n  - Kargo for progressive rollouts between environments\n- **Monitoring \u0026 Logging:**\n  - VM Metrics\n  - Alert Manager\n  - Grafana (Dashboards \u0026 Alerts as Code)\n  - Metrics Collectors:\n      - Grafana Alloy (k8s-monitoring-stack)\n      - kube-prometheus-stack\n      - Opentelemetry kube stack\n  - Grafana Tempo (Tracing)\n  - Logs:\n      - Grafana Loki\n      - VM Logs\n  - Kubernetes Events\n- **Automation \u0026 Security:**\n  - Renovate\n  - Keda\n  - Windmill just for test\n  - Trivy\n  - Teleport\n\n⏳ ***To-do:***\n- [vmalertmanagerconfig](https://docs.victoriametrics.com/operator/resources/vmalertmanagerconfig/)\n- [victoria-metrics-anomaly](https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-anomaly)\n- Gvizor\n- Postgres DB operator\n- Atlas schema migration\n- Service mesh\n- Crossplane\n- Github action runner\n\n---\n\n## 🏗️How to set this beauty on the local machine\n\n### Install Dependencies\n\nEnsure Nix and Devbox are installed. If you prefer not to install Nix, manually install the dependencies listed in [./devbox.json](./devbox.json)\n```sh\n$ devbox shell\n```\n\n### Create a K3d Cluster\n\n```sh\n$ k3d cluster create --config k3d-control-config.yaml\n```\n\n### Configure SOPS Locally\n\n1. Generate an encryption key pair:\n  ```sh\n  age-keygen -o ./ignore/key.txt\n  ```\n2. Copy the **public key** into `.sops.yaml` under the `age:` attribute.\n3. Create a K8s secret with the generated key for the sops-operator to be able to decrypt the secrets:\n  ```sh\n  kubectl create namespace sops-operator\n  kubectl create secret generic sops-age-key -n sops-operator --from-file=./ignore/key.txt\n  ```\n\n### Deploy Infrastructure Resources\n\n1. Create a **GitHub OAuth Application** if you want to use [SSO with ArgoCD](https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#dex). Paste the client ID and secret into the [argocd-secrets.yaml](infra/argocd/argocd-secrets.yaml) encryptedfile.\n\n2. Deploy ArgoCD:\n  ```sh\n  kubectl apply -k infra/argocd\n  kubectl apply -f infra/init-argo-apps.yaml\n  ```\n\nAccess the ArgoCD UI at [https://argocd.127.0.0.1.nip.io](https://argocd.127.0.0.1.nip.io)\n\n### Add the Second Application Cluster\n\n```sh\nk3d cluster create --config k3d-apps-config.yaml --volume $(pwd)/tmp:/tmp/k3dvol\n```\n\nCreate a K8s secret with the generated key for the sops-operator to be able to decrypt the secrets:\n\n```sh\nkubectl create namespace sops-operator\nkubectl create secret generic sops-age-key -n sops-operator --from-file=./ignore/key.txt\n```\n\nAdd the SA, Role and, RB to the target cluster\n```sh\nargocd cluster add k3d-sandbox-apps --name k3d-sandbox-apps --kube-context k3d-sandbox-control\n```\n\n\nGet the token from the argocd-manager-token secret:\n```sh\nkubectl get secret argocd-manager-token-* -n kube-system -o yaml --context k3d-sandbox-apps\n```\n\nCreate a new secret in [infra/argocd/clusters/k3d-apps-secret.yaml](infra/argocd/clusters/k3d-apps-secret.yaml) following this [example](infra/argocd/clusters/example-secret.yaml):\n```\nbearerToken ⇒ token from argocd-manager-token secret. You must base64 decode it. `echo \u003ctoken\u003e | base64 -d`\ncaData ⇒ ca.crt from argocd-manager-token secret.\n\n# not needed for this example\ncertData ⇒ client-certificate-data from kube context\nkeyData ⇒ client-key-data from kube context\n```\nEncrypt the secret:\n```sh\nsops encrypt --in-place infra/argocd/clusters/k3d-apps-secret.yaml\n```\n\n### Deploy Applications \u0026 Monitoring Components\n\nDeploy all applications:\n```sh\nkubectl apply -f apps/init-argo-apps.yaml --context k3d-sandbox-apps\n```\n\nDeploy individual applications:\n```sh\nkubectl apply -f apps/argo-apps/\u003capp-name\u003e --context k3d-sandbox-apps\n```\n\n## 📂 Project Structure\n\n### **Apps** - Our Own Applications\n\nWe use this space for defining applications developed by us. Applications are deployed to `k3d-sandbox-apps` cluster. We have defined here, `production`, `staging`, `dev` as well as `ephemeral environments` created from PR environments.\n\nThe application deployed here is just an example of a basic backend/frontend service. The code and helm chart can be found on the [playground-sandbox repo](https://github.com/Utwo/playground-sandbox).\n\nAvailable URLs:\n* http://dev.127.0.0.1.nip.io:8000\n* http://staging.127.0.0.1.nip.io:8000\n* http://127.0.0.1.nip.io:8000\n\n### **Infra** - Infrastructure Services\n\n#### ArgoCD\n\n[ArgoCD](./infra/argo-apps/argocd.yaml) is a continuous deployment tool for gitOps workflows.\n\nSSO with Github Login is enabled for the ArgoCD web UI using the Github OAuth App.\nAnother Github APP is used to:\n* read the content of the playground-sandbox repo and bypass the rate limit rule.\n* send argocd notifications on open pull requests when we deploy an ephemeral environment\n\nIt uses one repo to deploy to multiple clusters. Registered clusters can be found in the [clusters](./infra/argocd/clusters/) folder. For each cluster, we define a couple of labels that are later used to deploy or fill information in the CD pipeline.\n\nVisit ArgoCD UI at https://argocd.127.0.0.1.nip.io\n\n#### SOPS operator\n\n[SOPS operator](./infra/argo-apps/sops-secret-operator.yaml) is used to decrypt the sops secrets stored in the git repository and transform them into Kubernetes secrets\n\nBelow is an example of how to create a secret with Sops to safely store it on git.\n\nCreate a yaml SopsSecret:\n\n```yaml\napiVersion: isindir.github.com/v1alpha3\nkind: SopsSecret\nmetadata:\n  name: sopssecret-sample\nspec:\n  secretTemplates:\n    - name: sopssecret-sample\n      labels:\n        label0: value0\n        labelK: valueK\n      annotations:\n        key0: value0\n        keyN: valueN\n      stringData:\n        data-name0: data-value0\n        data-nameL: data-valueL\n```\n\nEncrypt the secret:\n```sh\nsops encrypt test.yaml \u003e test.enc.yaml\n```\n\nDecrypt the secret:\n```sh\nSOPS_AGE_KEY_FILE=./ignore/key.txt sops test.enc.yaml\n```\n\n#### Argo Rollouts\n[Argo Rollouts](./infra/argo-apps/argo-rollouts.yaml) is a Kubernetes controller and set of CRDs that provide advanced deployment capabilities such as blue-green, canary, canary analysis, experimentation, and progressive delivery features to Kubernetes.\n\nWe use Argo Rollouts to enable canary deployments for the `playground-sandbox` app using `ingress-ngnix`.\nTo expose the Argo Rollouts web UI locally run:\n```sh\nkubectl port-forward services/argo-rollouts-k3d-apps-dashboard 3100:3100 -n argo-rollouts --context k3d-sandbox-apps\n```\n\n#### Kargo\n[Kargo](./infra/argo-apps/kargo.yaml) is a continuous promotion orchestration layer, that complements Argo CD for Kubernetes. With Kargo we can define and promote the steps necessary to deploy to `dev`, `staging`, and `production`. The project is defined [here](./infra/kargo/projects/playground-sandbox/). It listens for both this repo and the application repo and when there is a new change, it generates the plain manifests from the app helm chart. The output is then pushed to [stage/dev branch](https://github.com/Utwo/k8s-playground/tree/stage/dev), [stage/staging branch](https://github.com/Utwo/k8s-playground/tree/stage/staging) or [stage/production branch](https://github.com/Utwo/k8s-playground/tree/stage/production) and applied to ArgoCD.\n\nTo expose the Kargo web UI locally run:\n```sh\nkubectl port-forward services/kargo-api 4000:443 -n kargo --context k3d-sandbox-control\n```\n\n#### Keda\n[KEDA](./infra/argo-apps/keda.yaml) is a Kubernetes-based Event Driven Autoscaler. With KEDA, you can drive the scaling of any container in Kubernetes based on the number of events needing to be processed.\n\nWe use Keda to scale ephemeral environments to 0 when they receive no traffic.\n\n### Monitoring - observability related services\n\nServices related to monitoring. Most of them should be deployed to a dedicated monitoring cluster.\n\n#### Grafana\nGrafana is used for visualizing metrics, aggregated logs, and tracings. It has SSO with Github enabled.\nTo expose the Grafana web UI locally run:\n```sh\nkubectl port-forward services/grafana 3000:3000 -n grafana --context k3d-sandbox-control\n```\n\n#### Victoria Metrics\n[Victoria Metrics](./monitoring/argo-apps/victoria-metrics.yaml) is used as a drop-in replacement for Prometheus with long-time storage. This service is exposed in order for other clusters to be able to write prometheus metrics. All prometheus instances/alloy are sending data to vm-single, from there vm-alert is reading and sending data to alertmanager and back to vm-single. We enable victoria metrics operator just for the rules.\n\nhttps://docs.victoriametrics.com/guides/multi-regional-setup-dedicated-regions/\n\n#### Grafana Tempo\n[Tempo](./monitoring/argo-apps/tempo.yaml) is used for storing traces.\n\n#### Grafana Loki\n[Loki](./monitoring/argo-apps/loki.yaml) is used for storing logs. This service is exposed in order for other clusters to be able to send logs here.\n\n#### Victoria Metrics logs\n[vmlogs](./monitoring/argo-apps/victoria-metrics.yaml) is used for collecting logs. It was added just as a proof of concept. It does not have backup/recovery solutions and it cannot offload old logs to a bucket. For now, we will rely on Loki for storing logs.\n\n#### Kube Prometheus Stack\n[kube-prometheus-stack](./monitoring/argo-apps/kube-prometheus-stack.yaml) is used just for deploying Prometheus rules, alerts and Grafana dashboards. Everything else is disabled because Grafana Alloy is used for collecting metrics and ServiceMonitors.\n\n#### Grafana Alloy\n[Alloy](./monitoring/argo-apps/alloy.yaml) is an open-telemetry collector distribution, used to collect metrics, logs, traces and, profiles. It is installed on every cluster that has the label `alloy: true`. Alloy also installs Prometheus CRDs to collect metrics from `ServiceMonitor`. It is also used to collect Kubernetes events. Logs are sent to Loki, traces to Tempo and, metrics to Victoria Metrics. All the data can be visualized in Grafana.\n\n#### Opentelemetry Kube Stack\n[Opentelemetry Kube Stack](./monitoring/argo-apps/opentelemetry-kube-stack.yaml) is an open-telemetry collector distribution, used to collect metrics, logs and traces. It is similar to Grafana Alloy but I couldn't make it work with the kube-prometheus-stack dashboards. We miss the `job` label and because of that the dashboards are not populated. Open [issue](https://github.com/open-telemetry/opentelemetry-helm-charts/issues/1545#issuecomment-2694671722).\n\n#### Alertmanager\nAlerts defined in kube-prometheus-stack are sent to alert-manager. From there we can define multiple routes to send them to external services like Pagerduty.\nAlerts from Grafana can be sent directly to external services like Pagerduty or to our own Alertmanager. One option would be to have a contact point defined in Grafana for every application/service, and every contact point to be mapped to a different PagerDuty service.\n\n---\nAlert Manager and Grafana can be installed using kube-prometheus-stack, but I prefer to handle them as a separate Argo application. This approach makes it easier to swap or update components.\nIn Grafana, datasources, providers, dashboards, alerting configurations, and plugins can be loaded through the Helm values file or from ConfigMaps using a sidecar. We can place the ConfigMaps in any namespace; for example, we can store the Victoria Metrics datasource in the Victoria Metrics folder.\nNote that Promtail has been discontinued, and Alloy is now the recommended option for collecting logs.\nKubernetes Event Exporter is no longer necessary, as Alloy also collects Kubernetes events.\n\n### Services - other services and deployments\n\nServices that cannot be added to any other category.\n\n[Trivy](./services/argo-apps/trivy.yaml) is a security scanner that finds vulnerabilities, misconfigurations, secrets, and SBOM in containers.\n\n[Windmill](./services/argo-apps/windmill.yaml) is a developer platform and workflow engine. Turn scripts into auto-generated UIs, APIs and, cron jobs. Compose them as workflows or data pipelines.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futwo%2Fk8s-playground","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Futwo%2Fk8s-playground","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Futwo%2Fk8s-playground/lists"}