{"id":15178838,"url":"https://github.com/danielstankw/grafana-metrics-scraper","last_synced_at":"2026-01-27T11:06:16.160Z","repository":{"id":248925362,"uuid":"830102282","full_name":"danielstankw/Grafana-metrics-scraper","owner":"danielstankw","description":"Reduce the cardinality of your metrics! - scrape all the metrics from Grafana dashboard and make informed decision.","archived":false,"fork":false,"pushed_at":"2024-07-19T12:58:16.000Z","size":35,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-05T11:42:35.256Z","etag":null,"topics":["cardinality","grafana","grafana-dashboard","monitoring","prometheus","prometheus-metrics","python3","relabel","scrape"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/danielstankw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-17T15:37:20.000Z","updated_at":"2024-07-19T12:58:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"dee2b69c-5418-4588-a39e-2aec9412bd85","html_url":"https://github.com/danielstankw/Grafana-metrics-scraper","commit_stats":{"total_commits":18,"total_committers":1,"mean_commits":18.0,"dds":0.0,"last_synced_commit":"2d62dbb22c5c067b3bbe4cab8490985860bbfe70"},"previous_names":["danielstankw/grafana-metrics"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielstankw%2FGrafana-metrics-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielstankw%2FGrafana-metrics-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielstankw%2FGrafana-metrics-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/danielstankw%2FGrafana-metrics-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/danielstankw","download_url":"https://codeload.github.com/danielstankw/Grafana-metrics-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240365958,"owners_count":19790006,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cardinality","grafana","grafana-dashboard","monitoring","prometheus","prometheus-metrics","python3","relabel","scrape"],"created_at":"2024-09-27T15:40:14.870Z","updated_at":"2026-01-27T11:06:16.112Z","avatar_url":"https://github.com/danielstankw.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## About\n\nThe `grafana_metrics_used.py` script extends the functionality of my [Query Scraper](https://github.com/danielstankw/Grafana-metrics/tree/main/query_scraper).   \n\nIt connects to the Grafana API and scrapes all the dashboards for two key items: **variables** and **PromQL queries**.   \n\nThe goal is to gather all the metrics used in the dashboards.\nAfter collecting the queries and variables, the script preprocesses them to extract only the metrics. It then removes any duplicates and sorts the metrics in alphabetical order.\n\n#### Step 1: We go from this\n```json\n    {\n        \"dashboard\": \"MyDashboard / K8s / Cluster / Ephemeral Storage\",\n        \"metric_names\": [\n            \"node_filesystem_avail_bytes\",\n            \"node_filesystem_size_bytes\",\n            \"up\"\n        ]\n    },\n    {\n        \"dashboard\": \"MyDashboard / K8s / Cluster View\",\n        \"metric_names\": [\n            \"kube_namespace_created\",\n            \"container_cpu_usage_seconds_total\",\n            \"kube_ingress_info\",\n\n```\n#### Step 2: ... to this - clean :) \n```json\n    \"cluster:namespace:pod_cpu:active:kube_pod_container_resource_limits\",\n    \"cluster:namespace:pod_cpu:active:kube_pod_container_resource_requests\",\n    \"cluster:namespace:pod_memory:active:kube_pod_container_resource_limits\",\n    \"cluster:namespace:pod_memory:active:kube_pod_container_resource_requests\",\n    \"container_cpu_usage_seconds_total\",\n    \"container_memory_cache\",\n    \"container_memory_rss\",\n    \"container_memory_swap\",\n    \"container_memory_working_set_bytes\",\n    \"container_network_receive_bytes_total\",\n    \"container_network_receive_packets_total\",\n    \"container_network_transmit_bytes_total\",\n    \"container_network_transmit_packets_total\",\n    \"kube_configmap_info\",\n```\n\nBy obtaining this list of metrics, users can identify the exact metrics required for the existing dashboards to function correctly. Any unnecessary metrics can be restricted or deleted, reducing the number of metrics scraped by Prometheus. This results in faster query times, saved memory, and a more robust monitoring solution.\n\n## How to run the script? \n\n# 1. Docker\n\n\n# 2. Python\n\nTo run you will need to modify the following in the `grafana_query_scraper.py`, and provide your grafana URL as well as Grafana API key (*read further to see how to get it*). \n\n```py\nif __name__ == \"__main__\":\n\n    # Your grafana url \n    grafana_url = \"\u003cYOUR-GRAFANA-URL\u003e\"  \n    # API key - read the readme for instructions.\n    api_key = \"\u003cYOUR-GRAFANA-API-KEY\u003e\"\n    # File name where the result of scrape will be saved\n    file_name = 'grafana_queries.json'\n```\n\nInstall the relevant dependancies and execute the script:\n```bashrc\npip install requests\npython3 grafana_query_scrapper.py\n```\n\n## Getting Grafana API key\n\nTo connect with Grafana API an API key is required. Below I present guide on how to get it.\n\u003cdiv style=\"text-align: center;\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/db1b3108-3fdc-436b-8b8c-0afcc665b665\" alt=\"Image 1\" width=\"900\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/67794c8a-255f-46c4-abd5-0049f0a8a4b2\" alt=\"Image 2\" width=\"900\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/c0dab608-8c0d-42d3-b341-490a6a3926f8\" alt=\"Image 3\" width=\"900\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/6955109f-155e-4023-aa44-7556458f3faf\" alt=\"Image 4\" width=\"900\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/a5a7c9be-49b4-427d-a3d2-238fab6b07b2\" alt=\"Image 5\" width=\"900\"\u003e\n    \u003cimg src=\"https://github.com/user-attachments/assets/bf5522aa-d772-4f6d-98a7-2cc9c2fe6bde\" alt=\"Image 6\" width=\"900\"\u003e\n\u003c/div\u003e\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielstankw%2Fgrafana-metrics-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdanielstankw%2Fgrafana-metrics-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdanielstankw%2Fgrafana-metrics-scraper/lists"}