{"id":13582108,"url":"https://github.com/last9/timescaledb-metrics","last_synced_at":"2025-04-28T12:22:04.943Z","repository":{"id":40540142,"uuid":"296034307","full_name":"last9/timescaledb-metrics","owner":"last9","description":"Send TimescaleDB policy stats (and other things) as metrics.","archived":false,"fork":false,"pushed_at":"2023-11-17T10:39:59.000Z","size":124,"stargazers_count":21,"open_issues_count":1,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-04-18T15:17:34.895Z","etag":null,"topics":["cloudwatch","timescaledb"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/last9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-16T13:03:01.000Z","updated_at":"2022-12-30T05:00:37.000Z","dependencies_parsed_at":"2023-11-17T11:48:59.237Z","dependency_job_id":null,"html_url":"https://github.com/last9/timescaledb-metrics","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/last9%2Ftimescaledb-metrics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/last9%2Ftimescaledb-metrics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/last9%2Ftimescaledb-metrics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/last9%2Ftimescaledb-metrics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/last9","download_url":"https://codeload.github.com/last9/timescaledb-metrics/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251311458,"owners_count":21569041,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cloudwatch","timescaledb"],"created_at":"2024-08-01T15:02:25.936Z","updated_at":"2025-04-28T12:22:04.924Z","avatar_url":"https://github.com/last9.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"\u003ca href=\"https://last9.io\"\u003e\u003cimg src=\"https://last9.github.io/assets/last9-github-badge.svg\" align=\"right\" /\u003e\u003c/a\u003e\n\n# timescaledb-metrics\n\nSend [TimescaleDB](https://github.com/timescale/timescaledb)\n[policy stats](https://docs.timescale.com/latest/api#timescaledb_information-policy_stats)\n(and other things) as metrics.\n\n# Usage\n\nWe advise not to pass credentials in configurations, but via Environment or\nInstance Profiles\n\n`env AWS_DEFAULT_REGION=\"ap-south-1\" CLOUDWATCH_NAMESPACE=\"timescaledb-dev\" DATABASE_URL=\"postgres://dbuser:4c77c8b8-a9e1-c2cd32feaeeb@10.29.152.230:5432/db\" ./timescaledb-metrics`\n\n## Output\n\n![Cloudwatch](cloudwatch.png)\n\n# Motivation\n\nWhile running timescaleDB, there are a lot of background jobs and policies which\nneed constant observation.\n\nExample of such Jobs are:\n\n- drop_chunks policy\n- compress_chunks policy\n- reorder policy\n- continuous_aggregate policy\n\netc.\n\nExample of the stats that can be pulled from TimescaleDB tables.\n\n```\n| entity_name            | job_type             | last_start_on | last_success_on | total_failures |\n|------------------------|----------------------|---------------|-----------------|----------------|\n| sli_300                | drop_chunks          | 1600262129    | 1600262129      | 0              |\n| custom_sli_archive     | reorder              | 1600195293    | 1600195293      | 6              |\n| custom_sli_archive     | compress_chunks      | 1600248496    | 1600248496      | 0              |\n| custom_sli_archive     | drop_chunks          | 1600259297    | 1600259297      | 0              |\n| sli_archive            | reorder              | 1600195323    | 1600195323      | 10             |\n| sli_300                | continuous_aggregate | 1600262169    | 1600262170      | 0              |\n| sli_archive            | drop_chunks          | 1600259297    | 1600259297      | 0              |\n| sli_archive            | compress_chunks      | 1600257782    | 1600257782      | 8              |\n| sli_violations_archive | drop_chunks          | 1600248501    | 1600248501      | 0              |\n```\n\nKeeping a manual eye on these is painful and tedious, and by emitting these as\nmetrics we can be alerted when things go wrong.\n\nWhile some may argue that the interval of these jobs is not as periodic where\nthey qualify to be a metric system, but as the usage increases these policies\nend up running closer to every 5 minutes. Alerts on such slopes can catch\nstaleness pretty effectively.\n\nAnother problem with the default\n[policy_stats](https://docs.timescale.com/latest/api#timescaledb_information-policy_stats)\nis that the job_id is hardly human readable. It's tough to keep a track of an\nIntegerID since it changes, on any recreation. But that's a minor win.\n\n# Components\n\nOnly timescaleDB.\n\n## Considerations\n\n- As of now we output to Cloudwatch but it can surely be extended to other\n  metric sources. If you want more metric sources, send a Pull Request or just\n  open a new Issue.\n\n- The reason to not use go-metrics, is the overall lack of precision of\n  timestamps which Cloudwatch is excellent at.\n\n## Prometheus Exporter\n\nYou can export timescale db metrics to Prometheus by setting the path of this\n[file](./postgres_exporter/queries.yaml) as an environment variable which the\n[postgres-exporter](https://github.com/wrouesnel/postgres_exporter) recognizes\nby default.\n\n`export PG_EXPORTER_EXTEND_QUERY_PATH=\u003cpath_to_queries.yaml\u003e`\n\nOr you can just add this to your existing custom `queries.yaml`\n\nReferences:\nhttps://github.com/wrouesnel/postgres_exporter#adding-new-metrics-via-a-config-file\n\n---\n\n# About Last9 \n\nThis project is sponsored and maintained by [Last9](https://last9.io). Last9 builds reliability tools for SRE and DevOps.\n\n\u003ca href=\"https://last9.io\"\u003e\u003cimg src=\"https://last9.github.io/assets/email-logo-green.png\" alt=\"\" loading=\"lazy\" height=\"40px\" /\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flast9%2Ftimescaledb-metrics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flast9%2Ftimescaledb-metrics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flast9%2Ftimescaledb-metrics/lists"}