{"id":48577639,"url":"https://github.com/clickhouse/pg_stat_ch","last_synced_at":"2026-06-13T02:03:41.233Z","repository":{"id":338524068,"uuid":"1141914436","full_name":"ClickHouse/pg_stat_ch","owner":"ClickHouse","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-03T04:13:25.000Z","size":1116,"stargazers_count":100,"open_issues_count":7,"forks_count":3,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-04-03T07:41:50.564Z","etag":null,"topics":["clickhouse","metrics","monitoring","postgresql","postgresql-extension"],"latest_commit_sha":null,"homepage":"https://clickhouse.com/blog/pg_stat_ch-postgres-extension-stats-to-clickhouse","language":"Perl","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ClickHouse.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-25T16:45:13.000Z","updated_at":"2026-03-23T15:12:23.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ClickHouse/pg_stat_ch","commit_stats":null,"previous_names":["clickhouse/pg_stat_ch"],"tags_count":9,"template":false,"template_full_name":null,"purl":"pkg:github/ClickHouse/pg_stat_ch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fpg_stat_ch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fpg_stat_ch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fpg_stat_ch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fpg_stat_ch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ClickHouse","download_url":"https://codeload.github.com/ClickHouse/pg_stat_ch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ClickHouse%2Fpg_stat_ch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31562697,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-08T14:31:17.711Z","status":"ssl_error","status_checked_at":"2026-04-08T14:31:17.202Z","response_time":54,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clickhouse","metrics","monitoring","postgresql","postgresql-extension"],"created_at":"2026-04-08T16:03:07.189Z","updated_at":"2026-06-13T02:03:41.226Z","avatar_url":"https://github.com/ClickHouse.png","language":"Perl","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/logo_dark.png#gh-dark-mode-only\" alt=\"pg_stat_ch logo\" width=\"512\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/logo_light.png#gh-light-mode-only\" alt=\"pg_stat_ch logo\" width=\"512\"\u003e\n\n# pg_stat_ch: PostgreSQL Query Telemetry Exporter to ClickHouse\n\n\u003c/div\u003e\n\nA PostgreSQL extension that captures per-query execution telemetry and exports it to ClickHouse in real-time. Unlike pg_stat_statements which aggregates statistics in PostgreSQL, pg_stat_ch exports **raw events** to ClickHouse where aggregation happens via ClickHouse's powerful analytical engine.\n\n## Quickstart\n\nRun local PostgreSQL + ClickHouse with schema preloaded:\n\n```bash\n./scripts/quickstart.sh up\n./scripts/quickstart.sh check\n```\n\nStop:\n\n```bash\n./scripts/quickstart.sh down\n```\n\nSee [docker/quickstart/README.md](docker/quickstart/README.md) for endpoints and stack details.\n\n## Table of Contents\n\n- [Overview](#overview)\n- [Architecture](#architecture)\n- [Features](#features)\n- [Supported Versions](#supported-versions)\n- [Building from Source](#building-from-source)\n- [Installation](#installation)\n- [Recommended PostgreSQL Settings](#recommended-postgresql-settings)\n- [Configuration](#configuration)\n- [SQL API](#sql-api)\n- [Example Queries](#example-queries)\n- [Testing](#testing)\n- [Troubleshooting](#troubleshooting)\n- [License](#license)\n\n## Overview\n\n\u003cdiv align=\"center\"\u003e\n\n\u003ca href=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/query-insights-overview.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/query-insights-overview.png\" alt=\"Query insights overview: queries per second, latency percentiles (p50/p95/p99), operations breakdown, rows returned, buffer hit ratio, and errors\" width=\"90%\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/query-insights-slow-query-patterns.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/query-insights-slow-query-patterns.png\" alt=\"Slow query patterns: per-pattern calls, errors, average/p95/max latency, total runtime, rows returned, and cache hit ratio\" width=\"90%\" /\u003e\u003c/a\u003e\n\n\u003ca href=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/query-insights-recent-queries.png\"\u003e\u003cimg src=\"https://raw.githubusercontent.com/ClickHouse/pg_stat_ch/main/images/query-insights-recent-queries.png\" alt=\"Recent queries: per-execution operation, query text, duration, rows, database, user, and blocks read\" width=\"90%\" /\u003e\u003c/a\u003e\n\n\u003c/div\u003e\n\npg_stat_ch captures detailed telemetry for every query executed in PostgreSQL and exports it to ClickHouse via a single data pipeline:\n\n```\nPostgreSQL Hooks (foreground) → Shared Memory Queue → Background Worker → ClickHouse\n```\n\n**Key design principles:**\n- **Zero network I/O on query path** - Events are queued in shared memory, not sent synchronously\n- **Raw events, not aggregates** - All aggregation (p50/p95/p99, top queries, errors) happens in ClickHouse\n- **Bounded memory** - Fixed-size ring buffer with overflow counters; dropped events don't block queries\n- **Minimal overhead** - ~5μs p99 overhead per captured statement\n\n## Architecture\n\n### Data Flow\n\n1. **PostgreSQL Hooks** capture query start/end with full instrumentation\n2. **Shared Memory Ring Buffer** stores events (MPSC: multi-producer, single-consumer)\n3. **Background Worker** dequeues batches and exports to ClickHouse\n4. **ClickHouse** stores raw events in `events_raw` table; views/MVs provide aggregates\n\n## Features\n\n- **Full Query Telemetry**: Timing, row counts, buffer usage, WAL usage, CPU time\n- **All Statement Types**: DML (SELECT/INSERT/UPDATE/DELETE/MERGE), DDL, utility statements\n- **Error Capture**: SQLSTATE codes and error levels via emit_log_hook\n- **JIT Instrumentation** (PG15+): Function count, generation/inlining/optimization/emission time\n- **Parallel Worker Stats** (PG18+): Planned vs launched workers\n- **Client Context**: Application name, client IP address\n- **Query Text**: Captured with truncation (2KB max)\n- **Graceful Degradation**: Queue overflow drops events with counters; ClickHouse unavailability doesn't block PostgreSQL\n\n## Supported Versions\n\nPostgreSQL 16, 17, and 18 are fully supported. See [docs/version-compatibility.md](docs/version-compatibility.md) for the feature matrix and version-specific fields.\n\n## Building from Source\n\nPrerequisites:\n\n- CMake 3.16+, C++17 compiler (GCC 9+, Clang 10+)\n- PostgreSQL 16+ development headers\n- [mise](https://mise.jdx.dev/) (recommended) or manual PostgreSQL installation\n\n[clickhouse-c](https://github.com/ClickHouse/clickhouse-c) is vendored as a submodule and statically linked.\n\n```bash\ngit submodule update --init --recursive\n\n# Using mise (recommended)\nmise run build              # Debug build (use build:16/17/18 for specific versions)\nmise run build:release      # Release build\nmise run install            # Install the extension\n\n# Or manually\ncmake -B build -G Ninja -DPG_CONFIG=/path/to/pg_config\ncmake --build build \u0026\u0026 cmake --install build\n```\n\n## Installation\n\n### 1. Configure PostgreSQL\n\nAdd to `postgresql.conf`:\n\n```ini\nshared_preload_libraries = 'pg_stat_ch'\ntrack_io_timing = on   # Enables I/O timing columns for captured events\n\n# ClickHouse connection (change for your setup)\npg_stat_ch.clickhouse_host = 'localhost'\npg_stat_ch.clickhouse_port = 9000\npg_stat_ch.clickhouse_database = 'pg_stat_ch'\n\n# TLS (recommended for production)\npg_stat_ch.clickhouse_use_tls = on\npg_stat_ch.clickhouse_skip_tls_verify = off\n```\n\n### 2. Restart PostgreSQL\n\n```bash\n# Restart PostgreSQL using your service manager (systemd, brew services, Docker, etc.)\n```\n\n### 3. Set Up Schema on ClickHouse\n\nQuickstart path: run `./scripts/quickstart.sh up` and schema setup is handled automatically.\n\nManual / existing ClickHouse paths:\n\n```bash\nclickhouse-client \u003c docker/init/00-schema.sql\n```\n\nCreates `events_raw` plus materialized views. See [docs/clickhouse.md](docs/clickhouse.md) for full schema details.\n\n### 4. Create the Extension\n\n```sql\nCREATE EXTENSION pg_stat_ch;\n```\n\n### 5. Verify\n\n```sql\nSELECT pg_stat_ch_version();\nSELECT * FROM pg_stat_ch_stats();\n```\n\n## Configuration\n\n### GUC Variables\n\n| Parameter | Type | Default | Reload | Description |\n|-----------|------|---------|--------|-------------|\n| `pg_stat_ch.enabled` | bool | `on` | SIGHUP | Enable/disable telemetry collection |\n| `pg_stat_ch.clickhouse_host` | string | `localhost` | Restart | ClickHouse server hostname |\n| `pg_stat_ch.clickhouse_port` | int | `9000` | Restart | ClickHouse native protocol port |\n| `pg_stat_ch.clickhouse_user` | string | `default` | Restart | ClickHouse username |\n| `pg_stat_ch.clickhouse_password` | string | `\"\"` | Restart | ClickHouse password |\n| `pg_stat_ch.clickhouse_database` | string | `pg_stat_ch` | Restart | ClickHouse database name |\n| `pg_stat_ch.queue_capacity` | int | `65536` | Restart | Ring buffer size (must be power of 2) |\n| `pg_stat_ch.string_area_size` | int | `64MB` | Restart | DSA size for variable-length query and error strings |\n| `pg_stat_ch.flush_interval_ms` | int | `1000` | SIGHUP | Export batch interval in milliseconds |\n| `pg_stat_ch.batch_max` | int | `10000` | SIGHUP | Maximum events per ClickHouse insert |\n| `pg_stat_ch.clickhouse_use_tls` | bool | `off` | Restart | Enable TLS for ClickHouse connections |\n| `pg_stat_ch.clickhouse_skip_tls_verify` | bool | `off` | Restart | Skip TLS certificate verification (insecure) |\n| `pg_stat_ch.log_min_elevel` | enum | `warning` | Superuser | Minimum error level to capture (debug5..panic) |\n\nSee [Error Level Values](docs/version-compatibility.md#error-fields) for the complete list of error levels and their numeric values in ClickHouse.\n\n## SQL API\n\n| Function | Description |\n|----------|-------------|\n| `pg_stat_ch_version()` | Returns extension version string |\n| `pg_stat_ch_stats()` | Queue and exporter statistics ([column details](docs/troubleshooting.md#sql-api-reference)) |\n| `pg_stat_ch_reset()` | Reset all queue counters to zero |\n| `pg_stat_ch_flush()` | Trigger immediate flush of queued events to ClickHouse |\n\n## Example Queries\n\nRaw per-execution events enable percentiles, time-series, and per-app drill-downs:\n\n```sql\n-- Slowest queries for a specific application, with percentiles\nSELECT query_id,\n       count() AS calls,\n       quantile(0.95)(duration_us) / 1000 AS p95_ms,\n       quantile(0.99)(duration_us) / 1000 AS p99_ms\nFROM pg_stat_ch.events_raw\nWHERE app = 'myapp'\n  AND ts_start \u003e now() - INTERVAL 1 HOUR\nGROUP BY query_id\nORDER BY p99_ms DESC\nLIMIT 10;\n```\n\nSee [docs/clickhouse.md](docs/clickhouse.md) for materialized view definitions and more query examples.\n\n## Testing\n\n```bash\nmise run test:all                          # Run all tests\nmise run test:regress                      # SQL regression tests only\n./scripts/run-tests.sh 18 all             # Specific PG version\n./scripts/run-tests.sh ../postgres/install_tap tap  # TAP tests with local PG build\n```\n\nSee [docs/testing.md](docs/testing.md) for test types, TAP test setup, and a full listing of test files.\n\n## Troubleshooting\n\nCommon issues: extension not loading (check `shared_preload_libraries`), events not appearing (check `pg_stat_ch_stats()` for errors), high queue usage or dropped events (tune `pg_stat_ch.queue_capacity`, `pg_stat_ch.flush_interval_ms`, `pg_stat_ch.batch_max`).\n\nSee [docs/troubleshooting.md](docs/troubleshooting.md) for detailed solutions.\n\n## License\n\nThis project is licensed under the Apache License, Version 2.0. See [LICENSE.md](LICENSE.md) for the full license text.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Fpg_stat_ch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fclickhouse%2Fpg_stat_ch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fclickhouse%2Fpg_stat_ch/lists"}