Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/mahlonsmith/netdata-timescale-relay
A Netdata json formatter for storing metrics in Timescale.
https://github.com/mahlonsmith/netdata-timescale-relay
json monitoring netdata nim postgresql timescale
Last synced: 3 days ago
JSON representation
A Netdata json formatter for storing metrics in Timescale.
- Host: GitHub
- URL: https://github.com/mahlonsmith/netdata-timescale-relay
- Owner: mahlonsmith
- Created: 2019-09-01T22:40:23.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-11-20T11:48:14.000Z (12 months ago)
- Last Synced: 2024-08-02T15:54:37.001Z (3 months ago)
- Topics: json, monitoring, netdata, nim, postgresql, timescale
- Language: Nim
- Size: 88.9 KB
- Stars: 17
- Watchers: 2
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
Netdata-TSRelay
===============
What's this?
------------
This program is designed to accept JSON streams from
[Netdata](https://my-netdata.io/) clients, and write metrics to a
PostgreSQL table - specifically, [Timescale](http://timescale.com)
backed tables (though that's not technically a requirement.)
Installation
------------
You'll need a working [Nim](http://nim-lang.org) build environment and
PostgreSQL development headers to compile the binary.
Simply run `make` to build it. Put it wherever you please.
Configuration
-------------
There are a few assumptions that should be satisfied before running
this successfully.
### Database setup
You'll need to create the destination table.
```sql
CREATE TABLE netdata (
time timestamptz default now() not null,
host text not null,
metrics jsonb default '{}'::jsonb not null
);
```
Index it based on how you intend to query the data, including JSON
functional indexing, etc. See PostgreSQL documentation for details.
Strongly encouraged: Promote this table to a Timescale "hypertable".
See [Timescale](http://timescale.com) docs for that, but a quick example
to partition automatically at weekly boundaries would look something
like this, if you're running v0.9.0 or better:
```sql
SELECT create_hypertable( 'netdata', 'time', migrate_data => true, chunk_time_interval => '1 week'::interval );
```
Timescale also has some great examples and advice for efficient [JSON
indexing](https://docs.timescale.com/timescaledb/latest/how-to-guides/schema-management/json/#json)
and queries.
### Netdata
You'll likely want to pare down what netdata is sending. Here's an
example configuration for `exporting.conf` -- season this to taste (what
charts to send and frequency.)
Note: This example uses the "exporting" module introduced in
Netdata v1.23. If your netdata is older than that, you'll be using
the deprecated "backend" instead in the main `netdata.conf` file.
```
[exporting:global]
enabled = yes
hostname = your-hostname
[json:timescale]
enabled = yes
data source = average
destination = localhost:14866
prefix = netdata
update every = 10
buffer on failures = 10
send charts matching = !cpu.cpu* !ipv6* !users.* nfs.rpc net.* net_drops.* net_packets.* !system.interrupts* system.* disk.* disk_space.* disk_ops.* mem.*
```
Running the Relay
-----------------
### Options
* [-q|--quiet]: Quiet mode. No output at all. Ignored if -d is supplied.
* [-d|--debug]: Debug mode. Show incoming data.
* [-D|--dropconn]: Drop the TCP connection to netdata between samples.
This may be more efficient depending on your environment and
number of clients. Defaults to false.
* [-o|--dbopts]: PostgreSQL connection information. (See below for more details.)
* [-h|--help]: Display quick help text.
* [-a|--listen-addr]: A specific IP address to listen on. Defaults to **INADDR_ANY**.
* [-p|--listen-port]: The port to listen for netdata JSON streams.
Default is **14866**.
* [-P|--persistent]: Don't disconnect from the database between samples. This may be
more efficient with a small number of clients, when not using a
pooler, or with a very high sample size/rate. Defaults to false.
* [-T|--dbtable]: Change the table name to insert to. Defaults to **netdata**.
* [-t|--timeout]: Maximum time in milliseconds to wait for data. Slow
connections may need to increase this from the default **500** ms.
* [-v|--version]: Show version.
**Notes**
Nim option parsing might be slightly different than what you're used to.
Flags that require arguments must include an '=' or ':' character.
* --timeout=1000 *valid*
* --timeout:1000 *valid*
* -t:1000 *valid*
* --timeout 1000 *invalid*
* -t 1000 *invalid*
All database connection options are passed as a key/val string to the
*dbopts* flag. The default is:
"host=localhost dbname=netdata application_name=netdata-tsrelay"
... which uses the default PostgreSQL port, and connects as the running
user.
Reference the [PostgreSQL Documentation](https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-PARAMKEYWORDS)
for all available options (including how to store passwords in a
separate file, enable SSL mode, etc.)
### Daemonizing
Use a tool of your choice to run this at system
startup in the background. My personal preference is
[daemontools](https://cr.yp.to/daemontools.html), but I won't judge you
if you use something else.
Here's an example using the simple
[daemon](https://www.freebsd.org/cgi/man.cgi?query=daemon&apropos=0&sektion=8&manpath=FreeBSD+11.0-RELEASE+and+Ports&arch=default&format=html) wrapper tool:
# daemon \
-o /var/log/netdata_tsrelay.log \
-p /var/run/netdata_tsrelay.pid \
-u nobody -cr \
/usr/local/bin/netdata_tsrelay \
--dbopts="dbname=metrics user=metrics host=db-master port=6432 application_name=netdata-tsrelay"
### Scaling
Though performant by default, if you're going to be storing a LOT of
data (or have a lot of netdata clients), here are some suggestions for
getting the most bang for your buck:
* Use the [pgbouncer](https://pgbouncer.github.io/) connection
pooler.
* DNS round robin the hostname where **netdata_tsrelay** lives across
*N* hosts -- you can horizontally scale without any gotchas.
* Edit your **netdata.conf** file to only send the metrics you are
interested in.
* Decrease the frequency at which netdata sends its data. (When in
"average" mode, it averages over that time automatically.)
* Use [Timescale](http://timescale.com) hypertables.
* Add database indexes specific to how you intend to consume the data.
* Use the PostgreSQL
[JSON Operators](https://www.postgresql.org/docs/current/static/functions-json.html#FUNCTIONS-JSONB-OP-TABLE),
which take advantage of GIN indexing.
* Put convenience SQL VIEWs around the data you're fetching later, for
easier graph building with [Grafana](https://grafana.com/) (or whatever.)