{"id":13696126,"url":"https://github.com/go-graphite/carbon-clickhouse","last_synced_at":"2025-04-12T18:47:56.733Z","repository":{"id":42652760,"uuid":"72667908","full_name":"go-graphite/carbon-clickhouse","owner":"go-graphite","description":"Graphite metrics receiver with ClickHouse as storage","archived":false,"fork":false,"pushed_at":"2025-03-13T01:50:12.000Z","size":15806,"stargazers_count":190,"open_issues_count":16,"forks_count":49,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-04-03T21:13:00.781Z","etag":null,"topics":["carbon","clickhouse","graphite","timeseries"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/go-graphite.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-11-02T18:10:37.000Z","updated_at":"2025-04-01T04:44:31.000Z","dependencies_parsed_at":"2024-04-13T00:41:37.766Z","dependency_job_id":"75fd1086-826c-474c-800e-fa6ecf97b448","html_url":"https://github.com/go-graphite/carbon-clickhouse","commit_stats":null,"previous_names":["lomik/carbon-clickhouse"],"tags_count":32,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-graphite%2Fcarbon-clickhouse","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-graphite%2Fcarbon-clickhouse/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-graphite%2Fcarbon-clickhouse/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/go-graphite%2Fcarbon-clickhouse/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/go-graphite","download_url":"https://codeload.github.com/go-graphite/carbon-clickhouse/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248617848,"owners_count":21134197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["carbon","clickhouse","graphite","timeseries"],"created_at":"2024-08-02T18:00:36.505Z","updated_at":"2025-04-12T18:47:56.704Z","avatar_url":"https://github.com/go-graphite.png","language":"Go","funding_links":[],"categories":["Integrations"],"sub_categories":["Metrics and Monitoring"],"readme":"[![deb](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/go-graphite/stable)\n[![rpm](https://img.shields.io/badge/rpm-packagecloud.io-844fec.svg)](https://packagecloud.io/go-graphite/stable)\n\n# carbon-clickhouse\nGraphite metrics receiver with ClickHouse as storage\n\n## Production status\nLast releases are stable and ready for production use\n\n## TL;DR\n[Preconfigured docker-compose](https://github.com/lomik/graphite-clickhouse-tldr)\n\n### Docker\nDocker images are available on [packages](https://github.com/lomik/carbon-clickhouse/pkgs/container/carbon-clickhouse) page.\n\n## Build\n\nRequired golang 1.18+\n\n```sh\n# build binary\ngit clone https://github.com/lomik/carbon-clickhouse.git\ncd carbon-clickhouse\nmake\n```\n\n## ClickHouse configuration\n\n1. Add `graphite_rollup` section to config.xml. Sample [here](https://github.com/lomik/graphite-clickhouse-tldr/blob/master/rollup.xml). You can use [carbon-schema-to-clickhouse](https://github.com/bzed/carbon-schema-to-clickhouse) for generate rollup xml from graphite [storage-schemas.conf](http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-schemas-conf).\n\n2. Create tables.\n\n```sql\nCREATE TABLE graphite (\n  Path String,\n  Value Float64,\n  Time UInt32,\n  Date Date,\n  Timestamp UInt32\n) ENGINE = GraphiteMergeTree('graphite_rollup')\nPARTITION BY toYYYYMM(Date)\nORDER BY (Path, Time);\n\n-- optional table for faster metric search\nCREATE TABLE graphite_index (\n  Date Date,\n  Level UInt32,\n  Path String,\n  Version UInt32\n) ENGINE = ReplacingMergeTree(Version)\nPARTITION BY toYYYYMM(Date)\nORDER BY (Level, Path, Date);\n\n-- optional table for storing Graphite tags\nCREATE TABLE graphite_tagged (\n  Date Date,\n  Tag1 String,\n  Path String,\n  Tags Array(String),\n  Version UInt32\n) ENGINE = ReplacingMergeTree(Version)\nPARTITION BY toYYYYMM(Date)\nORDER BY (Tag1, Path, Date);\n```\n\n[GraphiteMergeTree documentation](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/graphitemergetree/)\n\nYou can create Replicated tables. See [ClickHouse documentation](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replication/)\n\n## Configuration\n```\n$ carbon-clickhouse -help\nUsage of carbon-clickhouse:\n  -check-config=false: Check config and exit\n  -config=\"\": Filename of config\n  -config-print-default=false: Print default config\n  -version=false: Print version\n```\n\nDate are broken by default (not always in UTC), but this used from start of project, and can produce some bugs.  \nChange to UTC requires points/index/tags tables rebuild (Date recalc to true UTC) or queries with wide Date range.  \nSet `data.utc-date = true` for this.  \nWithout UTC date is required to run carbon-clickhouse and graphite-clickhouse in one timezone.\n\n```toml\n[common]\n# Prefix for store all internal carbon-clickhouse graphs. Supported macroses: {host}\nmetric-prefix = \"carbon.agents.{host}\"\n# Endpoint for store internal carbon metrics. Valid values: \"\" or \"local\", \"tcp://host:port\", \"udp://host:port\"\nmetric-endpoint = \"local\"\n# Interval of storing internal metrics. Like CARBON_METRIC_INTERVAL\nmetric-interval = \"1m0s\"\n# GOMAXPROCS\nmax-cpu = 1\n\n[logging]\n# \"stderr\", \"stdout\" can be used as file name\nfile = \"/var/log/carbon-clickhouse/carbon-clickhouse.log\"\n# Logging error level. Valid values: \"debug\", \"info\", \"warn\" \"error\"\nlevel = \"info\"\n\n[data]\n# Folder for buffering received data\npath = \"/data/carbon-clickhouse/\"\n# Rotate (and upload) file iniciated on size and interval\n# Rotate (and upload) file size (in bytes, also k, m and g units can be used)\n# chunk-max-size = '512m'\nchunk-max-size = 0\n# Rotate (and upload) file interval\n# Minimize chunk-interval for minimize lag between point receive and store\nchunk-interval = \"1s\"\n# Auto-increase chunk interval if the number of unprocessed files is grown\n# Sample, set chunk interval to 10 if unhandled files count \u003e= 5 and set to 60s if unhandled files count \u003e= 20:\n# chunk-auto-interval = \"5:10s,20:60s\"\nchunk-auto-interval = \"\"\n\n# Compression algorithm to use when storing temporary files.\n# Might be useful to reduce space usage when Clickhouse is unavailable for an extended period of time.\n# Currently supported: none, lz4\ncompression = \"none\"\n\n# Compression level to use.\n# For \"lz4\" 0 means use normal LZ4, \u003e=1 use LZ4HC with this depth (the higher - the better compression, but slower)\ncompression-level = 0\n\n# Date are broken by default (not always in UTC)\n#utc-date = false\n\n[upload.graphite]\ntype = \"points\"\ntable = \"graphite\"\nthreads = 1\nurl = \"http://localhost:8123/\"\n# compress-data enables gzip compression while sending to clickhouse\ncompress-data = true\ntimeout = \"1m0s\"\n# save zero value to Timestamp column (for point and posts-reverse tables)\nzero-timestamp = false\n\n[upload.graphite_index]\ntype = \"index\"\ntable = \"graphite_index\"\nthreads = 1\nurl = \"http://localhost:8123/\"\ntimeout = \"1m0s\"\ncache-ttl = \"12h0m0s\"\n# Store hash of metric in memory instead of full metric name\n# Allowed values: \"\", \"city64\" (empty value - disabled)\nhash = \"\"\n# If daily index should be disabled, default is `false`\ndisable-daily-index = false\n\n# # You can define additional upload destinations of any supported type:\n# # - points\n# # - index\n# # - tagged (is described below)\n# # - points-reverse (same scheme as points, but path 'a1.b2.c3' stored as 'c3.b2.a1')\n\n# # For uploaders with types \"points\" and \"points-reverse\" there is a possibility to ignore data using patterns. E.g.\n# [upload.graphite]\n# type = \"graphite\"\n# table = \"graphite.points\"\n# threads = 1\n# url = \"http://localhost:8123/\"\n# timeout = \"30s\"\n# ignored-patterns = [\n#     \"a1.b2.*.c3\",\n# ]\n\n# # Extra table which can be used as index for tagged series\n# # Also, there is an opportunity to avoid writing tags for some metrics.\n# # Example below, ignored-tagged-metrics.\n# [upload.graphite_tagged]\n# type = \"tagged\"\n# table = \"graphite_tagged\"\n# threads = 1\n# url = \"http://localhost:8123/\"\n# timeout = \"1m0s\"\n# cache-ttl = \"12h0m0s\"\n# ignored-tagged-metrics = [\n#     \"a.b.c.d\",  # all tags (but __name__) will be ignored for metrics like a.b.c.d?tagName1=tagValue1\u0026tagName2=tagValue2...\n#     \"*\",  # all tags (but __name__) will be ignored for all metrics; this is the only special case with wildcards\n# ]\n#\n# It is possible to connect to clickhouse with OpenSSL certificates (mTLS) like below:\n# [upload.graphite]\n# type = \"points\"\n# table = \"graphite\"\n# threads = 1\n# compress-data = true\n# zero-timestamp = false\n# timeout = \"1m0s\"\n# url = \"https://localhost:8443/\" # url is https\n# [upload.graphite.tls]\n# ca-cert = [ \"\u003cpath/to/rootCA.crt\u003e\", \"\u003cpath/to/other/rootCA.crt\u003e\" ]\n# server-name = \"\u003cserver-name\u003e\"\n# insecure-skip-verify = false # if true, server certificates will not be validated\n# [[upload.graphite.tls.certificates]]\n# key = \"\u003cpath/to/client.key\u003e\"\n# cert = \"\u003cpath/to/client.crt\u003e\"\n\n[udp]\nlisten = \":2003\"\nenabled = true\n# drop received point if timestamp \u003e now + value. 0 - don't drop anything\ndrop-future = \"0s\"\n# drop received point if timestamp \u003c now - value. 0 - don't drop anything\ndrop-past = \"0s\"\n# drop metrics with names longer than this value. 0 - don't drop anything\ndrop-longer-than = 0\n\n[tcp]\nlisten = \":2003\"\nenabled = true\ndrop-future = \"0s\"\ndrop-past = \"0s\"\ndrop-longer-than = 0\n\n[pickle]\nlisten = \":2004\"\nenabled = true\ndrop-future = \"0s\"\ndrop-past = \"0s\"\ndrop-longer-than = 0\n\n# https://github.com/lomik/carbon-clickhouse/blob/master/grpc/carbon.proto\n[grpc]\nlisten = \":2005\"\nenabled = false\ndrop-future = \"0s\"\ndrop-past = \"0s\"\ndrop-longer-than = 0\n\n[prometheus]\nlisten = \":2006\"\nenabled = false\ndrop-future = \"0s\"\ndrop-past = \"0s\"\ndrop-longer-than = 0\n\n[telegraf_http_json]\nlisten = \":2007\"\nenabled = false\ndrop-future = \"0s\"\ndrop-past = \"0s\"\ndrop-longer-than = 0\n# the character to join telegraf metric and field (default is \"_\" for historical reason and Prometheus compatibility)\nconcat = \".\"\n\n# Golang pprof + some extra locations\n#\n# Last 1000 points dropped by \"drop-future\", \"drop-past\" and \"drop-longer-than\" rules:\n# /debug/receive/tcp/dropped/\n# /debug/receive/udp/dropped/\n# /debug/receive/pickle/dropped/\n# /debug/receive/grpc/dropped/\n# /debug/receive/prometheus/dropped/\n# /debug/receive/telegraf_http_json/dropped/\n[pprof]\nlisten = \"localhost:7007\"\nenabled = false\n\n# You can use tag matching like in InfluxDB. Format is exactly the same.\n# It will parse all metrics that don't have tags yet.\n# For more information see https://docs.influxdata.com/influxdb/v1.7/supported_protocols/graphite/\n# Example:\n# [convert_to_tagged]\n# enabled = true \n# separator = \"_\"\n# tags = [\"region=us-east\", \"zone=1c\"]\n# templates = [\n#     \"generated.* .measurement.cpu  metric=idle\",\n#     \"* host.measurement* template_match=none\",\n# ] \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgo-graphite%2Fcarbon-clickhouse","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgo-graphite%2Fcarbon-clickhouse","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgo-graphite%2Fcarbon-clickhouse/lists"}