{"id":20673730,"url":"https://github.com/cpg314/ch-grafana-cache","last_synced_at":"2026-02-12T05:04:33.296Z","repository":{"id":242417701,"uuid":"809358862","full_name":"cpg314/ch-grafana-cache","owner":"cpg314","description":"Extract Clickhouse SQL queries from a dashboard and execute them.","archived":false,"fork":false,"pushed_at":"2025-02-26T08:01:11.000Z","size":79,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-18T13:22:25.775Z","etag":null,"topics":["chproxy","clickhouse","grafana"],"latest_commit_sha":null,"homepage":"https://c.pgdm.ch/code","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cpg314.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-02T13:21:59.000Z","updated_at":"2025-02-26T08:01:15.000Z","dependencies_parsed_at":"2024-06-02T22:22:52.964Z","dependency_job_id":"8c6feff4-479e-48c5-8a7c-2a5c2fcfdf17","html_url":"https://github.com/cpg314/ch-grafana-cache","commit_stats":null,"previous_names":["cpg314/ch-grafana-cache"],"tags_count":13,"template":false,"template_full_name":null,"purl":"pkg:github/cpg314/ch-grafana-cache","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fch-grafana-cache","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fch-grafana-cache/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fch-grafana-cache/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fch-grafana-cache/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cpg314","download_url":"https://codeload.github.com/cpg314/ch-grafana-cache/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cpg314%2Fch-grafana-cache/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29359360,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T01:03:07.613Z","status":"online","status_checked_at":"2026-02-12T02:00:06.911Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chproxy","clickhouse","grafana"],"created_at":"2024-11-16T20:42:22.969Z","updated_at":"2026-02-12T05:04:33.272Z","avatar_url":"https://github.com/cpg314.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ch-grafana-cache\n\nThis utility is meant to be used with the [Clickhouse Grafana data source](https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/). It:\n\n- Extracts the SQL queries from a Grafana dashboard, and\n- Sends them to a Clickhouse server for execution.\n\nThe main use case is to perform caching of the responses, e.g. via [chproxy's caching feature](https://www.chproxy.org/configuration/caching/) or [Clickhouse's query cache](https://clickhouse.com/docs/en/operations/query-cache), to make the dashboards execute faster and with less load on the database servers.\n\nVariables are supported, even those depending on others. The tool runs over all combinations of variables.\n\n## Usage\n\n```console\n$ ch-grafana-cache --help\nExecute Clickhouse SQL queries from a Grafana dashboard.\n\nCall with either --grafana-url and --dashboard, or with --json\n\nUsage: ch-grafana-cache [OPTIONS] \u003cCOMMAND\u003e\n\nCommands:\n  print    Print SQL statements, with syntax highlighting\n  execute  Execute the queries\n  help     Print this message or the help of the given subcommand(s)\n\nOptions:\n      --grafana-url \u003cGRAFANA_URL\u003e\n          Base Grafana URL\n\n          [env: GRAFANA_URL=https://grafana.corp.com/]\n\n      --dashboard \u003cDASHBOARD\u003e\n          Grafana dashboard id\n\n      --json \u003cJSON\u003e\n          Dashboard JSON file\n\n      --theme \u003cTHEME\u003e\n          Synctect for syntax highlighting. Pass any invalid value to see the list of available themes\n\n          [env: CH_GRAFANA_CACHE_THEME=Nord]\n\n  -h, --help\n          Print help (see a summary with '-h')\n\n  -V, --version\n          Print version\n\n$ ch-grafana-cache execute --help\nExecute the queries\n\nUsage: ch-grafana-cache execute [OPTIONS] --url \u003cURL\u003e --username \u003cUSERNAME\u003e\n\nOptions:\n      --url \u003cURL\u003e                        URL to the Clickhouse HTTP endpoint [env: CLICKHOUSE_URL=]\n      --username \u003cUSERNAME\u003e              Clickhouse username [env: CLICKHOUSE_USERNAME=]\n      --password \u003cPASSWORD\u003e              [env: CLICKHOUSE_PASSWORD=]\n      --variables-yaml \u003cVARIABLES_YAML\u003e  YAML file of the form variable_name: [ values ] to manually specify the values of some variables in the dashboard\n  -h, --help                             Print help\n```\n\nExamples\n\n```console\n$ # Printing the SQL queries in the dashboard\n$ ch-grafana-cache --grafana https://grafana.corp.com --dashboard mydashboard print\nVariables:\n\n...\n\nPanels:\n...\n\n$ # Executing the SQL queries in the dashboard across all combinations\n$ ch-grafana-cache --grafana https://grafana.corp.com --dashboard mydashboard execute --clickhouse http://chproxy.clickhouse.internal --username default\nINFO ch_grafana_cache: Retrieving dashboard\nINFO ch_grafana_cache: Retrieved dashboard 'mydashboard'\nINFO ch_grafana_cache: 166 variables combinations found. Executing queries...\nINFO ch_grafana_cache: Executing combination i=0 n_combinations=166\nINFO ch_grafana_cache: Executed combination duration=178.932498ms size_mb=0.107275\n```\n\n### Verifying that `chproxy` caching works\n\n- Clear the `chproxy` cache.\n- Close the Grafana dashboard\n- Run `ch_grafana_cache`\n  - The `chproxy` logs should give a `cache miss` for every query.\n- Open the Grafana dashboard.\n  - The `chproxy` logs should give a `cache hit` for every query.\n\nIf the dashboard gives cache misses, printing the cache key in chproxy ([here](https://github.com/ContentSquare/chproxy/blob/2d4c2bf185cb32bc127330b6f8d8614ba4ebbe61/cache/key.go#L86)) might allow understanding the difference between the cache queries and the Grafana ones. For example, a different HTTP compression setting will result in cache misses.\n\n## Installation\n\n- Get a precompiled binary or package from the [releases page](https://github.com/cpg314/ch-grafana-cache/releases); or\n- Build yourself:\n  ```\n  $ cargo make packages\n  ```\n\n## Development\n\n```\n$ docker run --name clickhouse --rm -p 8123:8123 clickhouse/clickhouse-server:latest\n$ # Install checkalot https://github.com/cpg314/checkalot\n$ cargo checkalot\n```\n\n## Alternative solutions\n\nIt does not seem possible to execute the queries without loading the Grafana front-end. For example, the [Grafana snapshot API](https://grafana.com/docs/grafana/latest/developers/http_api/snapshot/) states that it is meant to be called by the UI and requires the full dashboard payload.\n\nAn alternative implementation would be to load the front-end via a headless web-browser. This is much heavier, but simpler in several aspects (e.g. no need to reimplement templating or variable fetching). To support variables, the browser would need to interact with the page.\n\n## Current limitations\n\n- It is assumed that the queries do not use time range information at all.\n- The Clickhouse queries are sent directly (using the HTTP interface), rather than through the Grafana data source.\n- Only the `${varname}` [variable syntax](https://grafana.com/docs/grafana/latest/dashboards/variables/variable-syntax/) is supported.\n- It is assumed that the Clickhouse datasources are the ones containing `clickhouse` in their type.\n- The queries retrieving variables must be sent twice (once for parsing with the tabular format, once in native format for caching). The could be avoided by using the native format parsing from [klickhouse](https://docs.rs/klickhouse/latest/klickhouse/).\n- It is assumed that interdependent variables are topologically sorted.\n- Authentication to Grafana is not supported (but easy to add).\n- ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fch-grafana-cache","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcpg314%2Fch-grafana-cache","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcpg314%2Fch-grafana-cache/lists"}