{"id":13819690,"url":"https://github.com/kubernetes-sigs/usage-metrics-collector","last_synced_at":"2025-08-07T23:56:46.843Z","repository":{"id":65526202,"uuid":"568139218","full_name":"kubernetes-sigs/usage-metrics-collector","owner":"kubernetes-sigs","description":"High fidelity and scalable capacity and usage metrics for Kubernetes clusters","archived":false,"fork":false,"pushed_at":"2025-03-04T18:57:46.000Z","size":25318,"stargazers_count":128,"open_issues_count":3,"forks_count":24,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-21T06:55:27.504Z","etag":null,"topics":["capacity","kubernetes","metrics","prometheus-exporter","utilization"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kubernetes-sigs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"code-of-conduct.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-11-19T15:19:22.000Z","updated_at":"2025-04-04T04:20:09.000Z","dependencies_parsed_at":"2023-10-05T05:26:05.649Z","dependency_job_id":"f90e1bd5-2b65-494a-a440-e7ba7d3adbd4","html_url":"https://github.com/kubernetes-sigs/usage-metrics-collector","commit_stats":null,"previous_names":[],"tags_count":9,"template":false,"template_full_name":"kubernetes/kubernetes-template-project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Fusage-metrics-collector","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Fusage-metrics-collector/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Fusage-metrics-collector/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kubernetes-sigs%2Fusage-metrics-collector/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kubernetes-sigs","download_url":"https://codeload.github.com/kubernetes-sigs/usage-metrics-collector/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254488270,"owners_count":22079393,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["capacity","kubernetes","metrics","prometheus-exporter","utilization"],"created_at":"2024-08-04T08:00:51.745Z","updated_at":"2025-05-16T07:31:46.952Z","avatar_url":"https://github.com/kubernetes-sigs.png","language":"Go","funding_links":[],"categories":["kubernetes","Go"],"sub_categories":[],"readme":"# Usage Metrics Collector\n\nThe usage-metrics-collector is a Prometheus metrics collector optimized for collecting kube usage and\ncapacity metrics.\n\n## Motivation\n\nWhy not just use promql and recording rules?\n\n- Scale\n  - Aggregate at collection time to reduce prometheus work\n  - Export aggregated metrics without the raw metrics to reduce prometheus storage\n- Insight\n  - Join labels at collection time (e.g. set the priority class on pod metrics)\n  - Set hard to resolve labels (e.g. set the workload kind on pod metrics)\n  - View node-level cgroup utilization (e.g. kubepods vs system.slice metrics)\n- Fidelity\n  - Scrape utilization at 1s intervals as raw metrics\n  - Perform aggregations on the 1s interval metrics (e.g. get the p95 1s utilization sample for all replicas of a workload)\n\n### Example\n\n[collector.yaml](config/metrics-prometheus-collector/configmaps/collector.yaml)\n\n## Architecture\n\n### considerations\n\n- Metrics must be highly configurable\n  - The metrics labels derived from objects\n  - The aggregations (sum, average, median, max, p95, histograms, etc) must be configurable\n- Metrics should be able to be pushed to additional sources such as cloud storage buckets, BigQuery, etc\n- Metric computation must scale to large clusters with lots of churn.\n  - Run aggregations in parallel\n  - Re-use previous results as much as possible\n- Scrapes should always immediately get a result, even when complex aggregations in large clusters take minutes to compute.\n- Utilization metrics should not be published until the data is present from a sufficient number of nodes.\n  This is to prevent showing \"low\" utilization numbers before all nodes have sent results.\n- There should be no graphs in data.  There must always be at least 1 ready and healthy replica which can be scraped by prometheus.\n- Sampler pods which become unhealthy due to issues on a node should be continuously recreated until they are functional again.\n- All cluster objects and utilization samples are cached in the collector so memory must be optimized.\n- It is difficult to horizontally scale the collector.  Offloading computations to the samplers is preferred.\n\n### metrics-node-sampler\n\n- Runs as a DaemonSet on each Node\n- Periodically reads utilization data for cpu and memory and stores in ring buffer\n  - Period and number of samples is configurable\n- Reads host metrics directly from cgroups psuedo filesystem (e.g. cpu usage for all of kubepods cgroup)\n- Reads container metrics from containerd (e.g. cpu usage for an individual container)\n- Periodically pushes metrics to collectors\n  - After time period\n  - After new pod starts running\n  - Before shutting down\n- Finds collectors to push to via DNS\n- Collectors can register manually with each sampler\n- Performs some precomputations such as averages.\n\n### metrics-prometheus-collector\n\n- Runs as a Deployment with multiple replicas for HA\n- Highly configurable metrics\n  - Metric labels may be derived from annotations / labels on other objects (e.g. pod metrics should have metric labels pulled from node conditions)\n  - Metrics may be pre-aggregated prior to being scraped by prometheus (e.g. reduce cardinality, produce quantiles and histograms)\n- Periodically get the metrics and cache them to be scraped (i.e. minimize scrape time by eagerly computing results)\n- Registers itself and all collector replicas with each sampler\n- Recieves metrics from node-samplers as a utilization source\n- Waits until has sufficient samples before providing results\n- Waits until results have been scraped before marking self as Ready\n- Can write additional metrics to local files\n\n### collector side-cars\n\n- May expose additional metrics read from external sources\n- May write local files to persistent storage for futher analysis\n\n## Exposed Metrics\n\nA sample of the exposed metrics is available in [METRICS.md](METRICS.md).\n\nIn addition to these metrics, a series of performance related metrics are published for the collection process.\nThese metrics are documented in [performance analysis document](docs/performance-analysis.md).\n\n## Getting started\n\n**Note**: No usage-metrics-collector container image is publicly hosted.  Folks will need to build and publish\nthis own until this is resolved.\n\n### Installing into a cluster\n\n#### Kind cluster\n\n**Important**: requires using cgroups v1.\n\n- Must set for Docker on Mac using [these docs](https://docs.docker.com/desktop/release-notes/#for-mac-28)\n- Must set for GKE for 1.26+ clusters\n\n1. Create a kind cluster\n  - `kind create cluster`\n2. Build the image\n  - `docker build . -t usage-metrics-collector:v0.0.0`\n3. Load the image into kind\n  - `kind load docker-image usage-metrics-collector:v0.0.0`\n4. Install the config\n  - `kustomize build config | kubectl apply -f -`\n5. Update your context to use the usage-metrics-collector namespace by default\n  - `kubectl config set-context --current --namespace=usage-metrics-collector`\n\n### Kicking the tires\n\n1. Make sure the pods are healthy\n  - `kubectl get pods`\n2. Make sure the services have endpoints\n  - `kubectl describe services`\n3. Get the metrics from the collector itself\n  - `kubectl exec -t -i $(kubectl get pods -o name -l app=metrics-prometheus-collector) -- curl localhost:8080/metrics`\n  - wait for service to be ready\n  - `kubectl port-forward service/metrics-prometheus-collector 8080:8080`\n  - visit `localhost:8080/metrics` in your browser\n4. Get the metrics from prometheus\n  - `kubectl port-forward $(kubectl get pods -o name -l app=prometheus) 9090:9090`\n  - visit `localhost:9090/` in your browser\n5. View the metrics in Grafana\n  - `kubectl port-forward service/grafana 3000:3000`\n  - visit `localhost:3000` in your browser\n  - enter `admin` for the username and password\n  - go to \"Explore\"\n  - change the source to \"prometheus\"\n  - enter `kube_usage_` into the metric field\n  - remove the label filters\n  - click \"Run Query\"\n\n### Specifying aggregation rules\n\n1. Edit [config/metrics-prometheus-collector/configmaps/collector.yaml](config/metrics-prometheus-collector/configmaps/collector.yaml)\n2. Run `make run-local`\n3. View the updated metrics in grafana\n\n## Code of conduct\n\nParticipation in the Kubernetes community is governed by the [Kubernetes Code of Conduct](code-of-conduct.md).\n\n[owners]: https://git.k8s.io/community/contributors/guide/owners.md\n[Creative Commons 4.0]: https://git.k8s.io/website/LICENSE\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubernetes-sigs%2Fusage-metrics-collector","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkubernetes-sigs%2Fusage-metrics-collector","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkubernetes-sigs%2Fusage-metrics-collector/lists"}