{"id":28244545,"url":"https://github.com/m5lk3n/agility","last_synced_at":"2026-05-06T03:31:26.553Z","repository":{"id":65961519,"uuid":"328012443","full_name":"m5lk3n/agility","owner":"m5lk3n","description":"A cloud-native implementation of how to measure agility using Kubernetes Deployment Frequency","archived":false,"fork":false,"pushed_at":"2025-12-05T16:35:26.000Z","size":696,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-09T03:41:08.723Z","etag":null,"topics":["devops","go","golang","grafana-dashboard","kpi","kubernetes","prometheus-exporter","prometheus-metrics"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/m5lk3n.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2021-01-08T21:06:46.000Z","updated_at":"2025-12-05T16:35:26.000Z","dependencies_parsed_at":"2025-04-14T22:28:25.230Z","dependency_job_id":"6fb7897a-3f84-43fd-9005-3311544b525d","html_url":"https://github.com/m5lk3n/agility","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/m5lk3n/agility","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m5lk3n%2Fagility","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m5lk3n%2Fagility/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m5lk3n%2Fagility/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m5lk3n%2Fagility/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/m5lk3n","download_url":"https://codeload.github.com/m5lk3n/agility/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/m5lk3n%2Fagility/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32677870,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-06T02:33:58.958Z","status":"ssl_error","status_checked_at":"2026-05-06T02:33:39.611Z","response_time":117,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["devops","go","golang","grafana-dashboard","kpi","kubernetes","prometheus-exporter","prometheus-metrics"],"created_at":"2025-05-19T08:13:50.465Z","updated_at":"2026-05-06T03:31:26.545Z","avatar_url":"https://github.com/m5lk3n.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# agility\n\nThis is a cloud-native implementation of how to measure agility (read: digital transformation/DevOps adoption) using Kubernetes *Deployment Frequency*.\n\n## Motivation\n\nIn their book \"[Accelerate - The Science Behind DevOps: Building and Scaling High Performing Technology Organizations](https://itrevolution.com/book/accelerate/)\" Forsgren, Humble, and Kim list the following indicators as keys to measure agility:\n\n1. Deployment Frequency (DF) (or Throughput)\n2. Change Failure Rate\n3. Mean Time to Recovery (MTTR)\n4. Lead Time (or Cycle Time)\n\nThis project addresses the first KPI.\n\nThe Deployment Frequency is the amount of deployments (to production) per time period.\n\nAs per the [State of DevOps 2019](https://services.google.com/fh/files/misc/state-of-devops-2019.pdf) report, high performing teams deploy 4 times a day. For 2020, in [Data-Driven Benchmarks for High Performing Engineering Teams](https://www.youtube.com/watch?v=iUFpRFvlT2U), CircleCI observed a mean value of 8 deployments per day. This value is used as the main indicator in dashboard gauges below.\n\nBy providing an actual implementation to determine and to visualize deployment rates, this project called \"agility\" contributes to measure and to control a DevOps transformation in an organization.\n\nFor more details, see [background](docs/BACKGROUND.md).\n\nAll project documentation is [here](docs/).\n\n## Short Version\n\nAlthough it is highly advised to follow this README in full, here's the short version to get \"agility\" up and running:\n\n```bash\n$ make build # step 1\n$ make install # step 2\n# this will deploy the built system on Kubernetes\n```\n\nStep 3: [Configure](#configure)\n\nStep 4: [Access](#access)\n\nStep 5: [Explore](#explore)\n\n## Specification\n\n### KPI\n\nThis project's approach is purely based upon Kubernetes deployments as containerized applications in general and the Kubernetes container orchestrator in particular represent the state of the art. Therefore, IMHO, Kubernetes should be at the heart of an implementation of a digital transformation nowadays.\n\nBy watching deployments directly on Kubernetes using its API, any deployment, be it directly via `kubectl deploy`, or indirectly via Helm or CD tooling like ArgoCD, is captured. This makes \"agility\" agnostic to the deployment tooling or mechanism.\n\nSince this system is deployed as a light-weight container on a cluster, it can also be used in different, separate environments, not only production to give more valuable insights.\n\nAdditionally, the *deploymentwatcher* supports deployments per application and per namespace. For both, configurable include/exclude regexp patterns are available.\n\n### Stack\n\nThis project is truly cloud-native, it deploys and runs on Kubernetes only. It stores its settings (aforementioned name patterns so far) in a Kubernetes configmap.\n\nObserved Kubernetes deployments are measured and exported using [Prometheus Node Exporter](https://github.com/prometheus/node_exporter). Those measurements are scraped (regularly pulled) by [Prometheus](https://prometheus.io/), the de facto industry standard for monitoring on Kubernetes at the time of writing. Prometheus is not only a monitoring system but also a time series database to collect scraped data.\n\nAnother industry standard used here is [Grafana](https://grafana.com/) which facilitates great results illustration, such as deployment rates.\n\n## Realization\n\n### Metric idea\n\n\"agility\"'s goal is to capture deployment per app and namespace. When it comes to Prometheus Node Exporter and measurements, the specific metric format must be determined. There are at least two options here:\n\nA. `deployed{app=\u003cname\u003e,namespace=\u003cname\u003e}=\u003cUnixTimeOfDeployment\u003e`:\n\nDiscussion:\n\n- It's basically the same as the built-in `kube_deployment_created`\n- It doesn't capture all counts, only the latest timestamp, i.e. there are missed hits in between scrapes\n- It's less intuitive than an increasing count\n\nB. `deployed_count{app=\u003cname\u003e,namespace=\u003cname\u003e}=\u003cnumOfDeployments\u003e`:\n\nDiscussion:\n\n- It's an atomic count which requires history\n- It's conceptually easier to grasp than the `UnixTimeOfDeployment` value approach\n\nTherefore, *option B, `deployed_count`, is the chosen metric format*, but in this project here with an in-memory limitation, i.e. no permanent storage in Prometheus' time series for now. In the long-run, this is neglectable as we're primarily interested in recent agility, keeping values above certain thresholds.\n\nExpressed in [PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/), the Prometheus Query Language, that means for example:\n\n- Get the increase in number of deployments (for an app), e.g. for the last 24h: `increase(deployed_count[24h])`\n- Compute the frequency, e.g. for the last week: `increase(deployed_count[7d])/7`\n\n(The interested reader can find more PromQL examples [here](https://prometheus.io/docs/prometheus/latest/querying/examples/).)\n\n### Design\n\nAt its core, \"agility\" has 3 layers:\n\n1. A Kubernetes Deployments Watcher that is being notified about app installations on the cluster.\n2. A \"Storage Map\" that counts observed app deployments per namespace.\n3. A Prometheus Node Exporter that exposes map data.\n\nThe following high-level diagram visualizes the architecture more in detail, with the 3 core layers in **bold**:\n![Design](docs/design.png \"Design\")\n\n\"agility\"'s *deploymentwatcher* and *nodeexporter* run in the same pod. The *deploymentswatcher* settings for app and namespace name patterns are coming from a configmap (being read upon pod start only, i.e. after any change to the settings, the pod must be restarted).\n\nExported deployment counts end up in Prometheus' data store. While deployment counts and rates can be queried in Prometheus directly, they are also available via dedicated Grafana dashboards. More information about \"agility\"'s  Prometheus integration and Grafana dashboard setup can be found below.\n\nLast, and most importantly, the end user's access to the KPI measurements illustrated with \"agility\"'s Grafana dashboards is simply via browser client which is depicted in above's diagram in orange. No fanciness here.\n\n### Prerequisites\n\nLocally, to build and deploy, the following is required:\n\n- Make\n- Go 1.14+\n- Docker 19.03.6+\n- kubectl v1.20.2+\n- Helm v3.4.2\n\nAs far as Kubernetes is concerned:\n\n- [Kubernetes 1.19.1 installed](docs/K8S.md) with [Prometheus and Grafana deployed](docs/MONITORING.md)\n- Kubernetes privileges to manage RBAC settings (to be precise: authorization to grant the permission to watch deployments is needed, see [ClusterRole](chart/templates/rbac.yaml))\n\n## Build\n\nThere are two services in this project:\n\n1. a \"df-backend\" where *deploymentswatcher* and *nodeexporter* reside. It's basically a go binary :-)\n2. a \"df-frontend\" which is a simple API server to provide a liveness and a readiness endpoint for the backend. It's basically a go binary :-)\n\nBoth binaries are wrapped up with a Dockerfile. The resulting image can easily be deployed on Kubernetes using the [Helm chart](chart/).\n\nTo build the individual services:\n\n### \"df-backend\"\n\n```bash\ncd df-backend # my pwd is ~/go/src/lttl.dev/agility/df-backend\n# to invoke all steps individually\n$ make init\ngo mod init lttl.dev/agility/df-backend\ngo: creating new go.mod: module lttl.dev/agility/df-backend\n$ make get\ngo get k8s.io/client-go@v0.19.1\n# see also https://github.com/jtestard/client-go/blob/master/INSTALL.md#add-client-go-as-a-dependency\n$ make build\nGOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o df-backend\n# for full Makefile usage info, run: make\n# ...\n# or short: make all\n```\n\nThe outcome of `make` above is the `df-backend` binary.\n\n### \"df-frontend\"\n\n```bash\ncd df-frontend # pwd is ~/go/src/lttl.dev/agility/df-frontend\n# to invoke all steps individually\n$ make init\n# ...\n$ make get\n# ...\n$ make build\n# ...\n# for full Makefile usage info, run: make\n# ...\n# or short: make all\n```\n\nThe outcome of `make` above is the `df-frontend` binary.\n\n### Shortcut\n\nInstead of building both binaries individually, but to compile both services at once and to bake them into a container image, use (and this should be the standard way):\n\n```bash\n# from this project directory\n$ make build\n...\nSuccessfully tagged lttl.dev/agility-df:0.1.1\n```\n\n## Install\n\nFinally, to install the previously built image on Kubernetes as follows:\n\n```bash\n# uncomment the following if you're using kind to load the image\n# $ make load\n# from this project directory\n$ make install\n```\n\nThis will deploy \"agility\" on Kubernetes.\n\n## Configure\n\nTo configure the *nodeexporter* in Prometheus, add the following snippet to `prometheus-server`'s Configmap under *scrape_configs*:\n\n```yaml\nscrape_configs:\n- job_name: 'df_node_exporter_metrics'\n  # scrape_interval: 5s # for testing only \n  metrics_path: /metrics\n  static_configs:\n    - targets:\n      - agility-df-backend.agility.svc:8080\n  # optional, remove system-generated labels:\n  metric_relabel_configs:\n    - source_labels: [ job ]\n      target_label: job\n      action: replace\n      replacement: ''\n    - source_labels: [ instance ]\n      target_label: instance\n      action: replace\n      replacement: ''\n```\n\nRemove the `prometheus-server` pod to force a restart.\n\n## Access\n\n### Import Prometheus Dashboards in Grafana\n\nWith aforementioned Kubernetes installation at hand, browse to Grafana (here assumed to be available under [http://localhost:3000](helper/expose-grafana.sh)). Follow Dashboards -\u003e Manage -\u003e [Import](http://localhost:3000/dashboard/import) to upload [these dashboards](grafana-dashboards/).\n\nFor thresholds used in gauges, see [background](docs/BACKGROUND.md).\n\n### Optional: Access \"df-backend\" directly\n\n```bash\n# forward local port 8088 to df-frontend port (80), to enable e.g.: curl http://localhost:8088/ready\n$ helper/expose-df-frontend.sh\n# forward local port 8089 to df-backend (node-exporter) port (8080), to enable: curl localhost:8089/metrics\n$ helper/expose-df-backend.sh\n```\n\n## Explore\n\n### Optional: Sanity Check\n\n```bash\n# run some dummy deployments ...\n$ helper/deploy-dummy-apps.sh\n```\n\n### Read Measurements\n\nObserve and explore the Grafana dashboards like the following:\n\n![Example Short-term Throughput Gauges](docs/grafana_df_short_term.png \"Example Short-term Throughput Gauges\")\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm5lk3n%2Fagility","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fm5lk3n%2Fagility","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fm5lk3n%2Fagility/lists"}