{"id":16117302,"url":"https://github.com/nickbabcock/pg-collectd","last_synced_at":"2025-03-18T10:30:58.272Z","repository":{"id":33530263,"uuid":"155292616","full_name":"nickbabcock/pg-collectd","owner":"nickbabcock","description":"An alternative and opinionated postgres collectd writer","archived":false,"fork":false,"pushed_at":"2024-12-10T08:31:28.000Z","size":351,"stargazers_count":13,"open_issues_count":7,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-16T19:21:42.760Z","etag":null,"topics":["collectd","collectd-plugin","postgres","timescaledb"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nickbabcock.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-10-29T23:01:14.000Z","updated_at":"2024-09-29T13:01:59.000Z","dependencies_parsed_at":"2024-10-27T18:02:39.099Z","dependency_job_id":"bdb41a9f-6cac-47c9-bb89-e4bfde85e74e","html_url":"https://github.com/nickbabcock/pg-collectd","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Fpg-collectd","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Fpg-collectd/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Fpg-collectd/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nickbabcock%2Fpg-collectd/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nickbabcock","download_url":"https://codeload.github.com/nickbabcock/pg-collectd/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244201734,"owners_count":20415137,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["collectd","collectd-plugin","postgres","timescaledb"],"created_at":"2024-10-09T20:43:53.048Z","updated_at":"2025-03-18T10:30:57.972Z","avatar_url":"https://github.com/nickbabcock.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![ci](https://github.com/nickbabcock/pg-collectd/actions/workflows/ci.yml/badge.svg)](https://github.com/nickbabcock/pg-collectd/actions/workflows/ci.yml)\n\n# pg-collectd\n\npg-collectd provides an alternative and opinionated postgres collectd writer\nplugin, where flexibility is traded in for performance and ease of use. A quick\nrundown.\n\n- No dependencies (other than collectd). pg-collectd utilizes the (unofficial) [collectd rust plugin](https://github.com/nickbabcock/collectd-rust-plugin) for low cost binding to collectd's C API. The [pure rust postgres\n  driver](https://github.com/sfackler/rust-postgres) is statically compiled\n  into the plugin. No need to rely on libpq.\n- Simplified insertion as the data is expanded and denormalized, so instead of\n  writing a function that receives an array of values / identifiers, these are\n  expanded so everything fits into a single table and columns contain single\n  values (not arrays).\n- a 4x reduction in db cpu usage compared to using [collectd's default postgres writer + setup](https://github.com/collectd/collectd/blob/92c3b2ed5f8e49737e29b11244585960a3478494/contrib/postgresql/collectd_insert.sql) (a conservative estimate)\n\nHere are the downsides:\n\n- Not an officially supported collectd plugin\n- Not as feature rich (eg: currently no support for TLS connections / does not support custom table names)\n- Only tested on a limited subset of collectds (though it may work on other\n  versions depending on if collectd changed its C API)\n- Only distributed as debs or source (eg: no rpms / apt repository)\n\n## Compatibility\n\n- Collectd 5.7+\n- Postgres 7.4+\n\n## Installation\n\nFirst we must set up the database with the following schema:\n\n```sql\nCREATE TABLE IF NOT EXISTS collectd_metrics (\n   time TIMESTAMPTZ NOT NULL,\n   plugin TEXT,\n   plugin_instance TEXT,\n   type_instance TEXT,\n   type TEXT,\n   host TEXT,\n   metric TEXT,\n   value DOUBLE PRECISION\n);\n```\n\n- (Optional) If using the TimescaleDB extension for postgres, the statements below contains provide good defaults\n\n```sql\nSELECT create_hypertable('collectd_metrics', 'time', chunk_time_interval =\u003e interval '1 day');\nALTER TABLE collectd_metrics SET (\n   timescaledb.compress,\n   timescaledb.compress_segmentby = 'plugin',\n   timescaledb.compress_orderby = 'time'\n);\nSELECT add_compression_policy('collectd_metrics', INTERVAL '7 days');\nSELECT add_retention_policy('collectd_metrics', INTERVAL '90 days');\n```\n\n- Create a user that only has `INSERT` permissions on `collectd_metrics`:\n\n```sql\nCREATE USER collectd WITH PASSWORD 'xxx';\nGRANT INSERT ON collectd_metrics TO collectd;\n```\n\n- Download the appropriate package from the [latest\n  release](https://github.com/nickbabcock/pg-collectd/releases/latest) (see\n  the compatibility list shown earlier)\n- Install with `dpkg -i pg-collectd-*.deb`\n- Edit collectd configuration (eg: `/etc/collectd/collectd.conf`)\n\n```\nLoadPlugin pg_collectd\n\u003cPlugin pg_collectd\u003e\n    BatchSize 1000\n    Connection \"postgresql://\u003cuser\u003e:\u003cpassword\u003e@\u003chost\u003e:\u003cport\u003e/\u003cdb\u003e\"\n    StoreRates true\n    LogTimings INFO\n\u003c/Plugin\u003e\n```\n\n- Restart collectd\n\nNot using Ubuntu / Debian? No problem, [build from source](#building).\n\n## Configuration Option\n\n- BatchSize: number of values to batch (eg: rows in the csv) before copying them to the database. Default is 100, which is extremely conservative. Test what is appropriate for you, but 500 to 1000 works well for me. Note that it is possible for the number of rows inserted to not be exactly equal to batch size, as `NaN` rates are not stored and some metrics given to write contain more than one value.\n- Connection (see [postgres connection uri documentation](https://www.postgresql.org/docs/10/static/libpq-connect.html#id-1.7.3.8.3.6))\n- StoreRates: Controls whether DERIVE and COUNTER metrics are converted to a rate before sending. Default is true.\n- LogTimings: The level at which to log performance timings. The default is `DEBUG` to cut down on potential log spam, though there is no problem setting it to `INFO` (or `WARN` / `ERROR` for that matter), as only a single line is logged per batched insert.\n\n## Performance Secret Sauce\n\nThe original postgres writer for collectd works by having a long lasting\ntransaction writing many individual statements committed at `CommitInterval`\nseconds. [A quote from postgres's official\ndocumentation](https://www.postgresql.org/docs/9.2/static/populate.html) leads\nus to ponder low hanging fruit:\n\n\u003e Note that loading a large number of rows using COPY is almost always faster\n\u003e than using INSERT, even if PREPARE is used and multiple insertions are\n\u003e batched into a single transaction.\n\nTo take advantage of this, pg-collectd batches up a certain number of values\n(`BatchSize`), and then formats those values as part of a in-memory CSV file\nthat can `COPY` over to postgres. What's nice is that memory allocations are\namortized such that over time, no memory is allocated for the in-memory CSV\nfile, only the CPU time for formatting the CSV is needed.\n\n## Building\n\nTo build the repo for collectd, ensure you have [Rust\ninstalled](https://rustup.rs/) and then execute the build process:\n\n```\ncargo build --release\n```\n\nThe resulting `./target/release/libpg_collectd.so` should be copied (locally\nor remotely) to `/usr/lib/collectd/pg_collectd.so`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickbabcock%2Fpg-collectd","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnickbabcock%2Fpg-collectd","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnickbabcock%2Fpg-collectd/lists"}