{"id":13584064,"url":"https://github.com/mahlonsmith/netdata-timescale-relay","last_synced_at":"2025-04-06T22:31:54.234Z","repository":{"id":47080704,"uuid":"205740437","full_name":"mahlonsmith/netdata-timescale-relay","owner":"mahlonsmith","description":"A Netdata json formatter for storing metrics in Timescale.","archived":false,"fork":false,"pushed_at":"2023-11-20T11:48:14.000Z","size":91,"stargazers_count":17,"open_issues_count":1,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-06T00:39:51.526Z","etag":null,"topics":["json","monitoring","netdata","nim","postgresql","timescale"],"latest_commit_sha":null,"homepage":null,"language":"Nim","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mahlonsmith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-09-01T22:40:23.000Z","updated_at":"2023-09-20T03:29:17.000Z","dependencies_parsed_at":"2024-11-06T00:43:13.067Z","dependency_job_id":null,"html_url":"https://github.com/mahlonsmith/netdata-timescale-relay","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahlonsmith%2Fnetdata-timescale-relay","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahlonsmith%2Fnetdata-timescale-relay/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahlonsmith%2Fnetdata-timescale-relay/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mahlonsmith%2Fnetdata-timescale-relay/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mahlonsmith","download_url":"https://codeload.github.com/mahlonsmith/netdata-timescale-relay/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247563898,"owners_count":20958971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["json","monitoring","netdata","nim","postgresql","timescale"],"created_at":"2024-08-01T15:03:59.241Z","updated_at":"2025-04-06T22:31:53.971Z","avatar_url":"https://github.com/mahlonsmith.png","language":"Nim","funding_links":[],"categories":["Nim"],"sub_categories":[],"readme":"\nNetdata-TSRelay\n===============\n\nWhat's this?\n------------\n\nThis program is designed to accept JSON streams from\n[Netdata](https://my-netdata.io/) clients, and write metrics to a\nPostgreSQL table - specifically, [Timescale](http://timescale.com)\nbacked tables (though that's not technically a requirement.)\n\n\nInstallation\n------------\n\nYou'll need a working [Nim](http://nim-lang.org) build environment and\nPostgreSQL development headers to compile the binary.\n\nSimply run `make` to build it.  Put it wherever you please.\n\n\nConfiguration\n-------------\n\nThere are a few assumptions that should be satisfied before running\nthis successfully.\n\n### Database setup\n\nYou'll need to create the destination table.\n\n```sql\nCREATE TABLE netdata (\n\ttime timestamptz default now() not null,\n\thost text not null,\n\tmetrics jsonb default '{}'::jsonb not null\n);\n```\n\nIndex it based on how you intend to query the data, including JSON\nfunctional indexing, etc.  See PostgreSQL documentation for details.\n\nStrongly encouraged:  Promote this table to a Timescale \"hypertable\".\nSee [Timescale](http://timescale.com) docs for that, but a quick example\nto partition automatically at weekly boundaries would look something\nlike this, if you're running v0.9.0 or better:\n\n```sql\nSELECT create_hypertable( 'netdata', 'time', migrate_data =\u003e true, chunk_time_interval =\u003e '1 week'::interval );\n```\n\nTimescale also has some great examples and advice for efficient [JSON\nindexing](https://docs.timescale.com/timescaledb/latest/how-to-guides/schema-management/json/#json)\nand queries.\n\n\n### Netdata\n\nYou'll likely want to pare down what netdata is sending.  Here's an\nexample configuration for `exporting.conf` -- season this to taste (what\ncharts to send and frequency.)\n\nNote: This example uses the \"exporting\" module introduced in\nNetdata v1.23.  If your netdata is older than that, you'll be using\nthe deprecated \"backend\" instead in the main `netdata.conf` file.\n\n```\n[exporting:global]\n\tenabled  = yes\n\thostname = your-hostname\n\n[json:timescale]\n\tenabled              = yes\n\tdata source          = average\n\tdestination          = localhost:14866\n\tprefix               = netdata\n\tupdate every         = 10\n\tbuffer on failures   = 10\n\tsend charts matching = !cpu.cpu* !ipv6* !users.* nfs.rpc net.* net_drops.* net_packets.* !system.interrupts* system.* disk.* disk_space.* disk_ops.* mem.*\n```\n\n\nRunning the Relay\n-----------------\n\n### Options\n\n  * [-q|--quiet]:    Quiet mode.  No output at all. Ignored if -d is supplied.\n  * [-d|--debug]:    Debug mode.  Show incoming data.\n  * [-D|--dropconn]: Drop the TCP connection to netdata between samples.\n                     This may be more efficient depending on your environment and\n                     number of clients.  Defaults to false.\n  * [-o|--dbopts]:   PostgreSQL connection information.  (See below for more details.)\n  * [-h|--help]:     Display quick help text.\n  * [-a|--listen-addr]: A specific IP address to listen on.  Defaults to **INADDR_ANY**.\n  * [-p|--listen-port]: The port to listen for netdata JSON streams.\n                     Default is **14866**.\n  * [-P|--persistent]: Don't disconnect from the database between samples. This may be\n                     more efficient with a small number of clients, when not using a\n                     pooler, or with a very high sample size/rate.  Defaults to false.\n  * [-T|--dbtable]:  Change the table name to insert to.  Defaults to **netdata**.\n  * [-t|--timeout]:  Maximum time in milliseconds to wait for data.  Slow\n                     connections may need to increase this from the default **500** ms.\n  * [-v|--version]:  Show version.\n\n\n\n**Notes**\n\nNim option parsing might be slightly different than what you're used to.\nFlags that require arguments must include an '=' or ':' character.\n\n  * --timeout=1000  *valid*\n  * --timeout:1000  *valid*\n  * -t:1000  *valid*\n  * --timeout 1000  *invalid*\n  * -t 1000  *invalid*\n\nAll database connection options are passed as a key/val string to the\n*dbopts* flag.  The default is:\n\n\t\"host=localhost dbname=netdata application_name=netdata-tsrelay\"\n\n... which uses the default PostgreSQL port, and connects as the running\nuser.\n\nReference the [PostgreSQL Documentation](https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS)\nfor all available options (including how to store passwords in a\nseparate file, enable SSL mode, etc.)\n\n\n### Daemonizing\n\nUse a tool of your choice to run this at system\nstartup in the background.  My personal preference is\n[daemontools](https://cr.yp.to/daemontools.html), but I won't judge you\nif you use something else.\n\nHere's an example using the simple\n[daemon](https://www.freebsd.org/cgi/man.cgi?query=daemon\u0026apropos=0\u0026sektion=8\u0026manpath=FreeBSD+11.0-RELEASE+and+Ports\u0026arch=default\u0026format=html) wrapper tool:\n\n\t# daemon \\\n\t\t-o /var/log/netdata_tsrelay.log \\\n\t\t-p /var/run/netdata_tsrelay.pid \\\n\t\t-u nobody -cr \\\n\t\t/usr/local/bin/netdata_tsrelay \\\n\t\t\t--dbopts=\"dbname=metrics user=metrics host=db-master port=6432 application_name=netdata-tsrelay\"\n\n### Scaling\n\nThough performant by default, if you're going to be storing a LOT of\ndata (or have a lot of netdata clients), here are some suggestions for\ngetting the most bang for your buck:\n\n  * Use the [pgbouncer](https://pgbouncer.github.io/) connection\n    pooler.\n  * DNS round robin the hostname where **netdata_tsrelay** lives across\n    *N* hosts -- you can horizontally scale without any gotchas.\n  * Edit your **netdata.conf** file to only send the metrics you are\n    interested in.\n  * Decrease the frequency at which netdata sends its data. (When in\n    \"average\" mode, it averages over that time automatically.)\n  * Use [Timescale](http://timescale.com) hypertables.\n  * Add database indexes specific to how you intend to consume the data.\n  * Use the PostgreSQL\n    [JSON Operators](https://www.postgresql.org/docs/current/static/functions-json.html#FUNCTIONS-JSONB-OP-TABLE),\n\twhich take advantage of GIN indexing.\n  * Put convenience SQL VIEWs around the data you're fetching later, for\n    easier graph building with [Grafana](https://grafana.com/) (or whatever.)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahlonsmith%2Fnetdata-timescale-relay","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmahlonsmith%2Fnetdata-timescale-relay","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmahlonsmith%2Fnetdata-timescale-relay/lists"}