{"id":17526297,"url":"https://github.com/stohrendorf/csv-prometheus-exporter","last_synced_at":"2025-03-06T05:32:15.817Z","repository":{"id":38980874,"uuid":"153519677","full_name":"stohrendorf/csv-prometheus-exporter","owner":"stohrendorf","description":"Prometheus Exporter for CSV-based files over SSH","archived":false,"fork":false,"pushed_at":"2023-09-28T10:46:20.000Z","size":479,"stargazers_count":20,"open_issues_count":1,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-03-14T20:06:05.926Z","etag":null,"topics":["c-sharp","csv","log-analysis","prometheus-exporter","ssh"],"latest_commit_sha":null,"homepage":"","language":"C#","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stohrendorf.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-10-17T20:29:46.000Z","updated_at":"2024-02-23T08:33:33.000Z","dependencies_parsed_at":"2022-08-25T21:53:30.571Z","dependency_job_id":"bf91cddc-1137-411f-a9ef-d811bc42b856","html_url":"https://github.com/stohrendorf/csv-prometheus-exporter","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stohrendorf%2Fcsv-prometheus-exporter","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stohrendorf%2Fcsv-prometheus-exporter/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stohrendorf%2Fcsv-prometheus-exporter/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stohrendorf%2Fcsv-prometheus-exporter/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stohrendorf","download_url":"https://codeload.github.com/stohrendorf/csv-prometheus-exporter/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":220753131,"owners_count":16697311,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["c-sharp","csv","log-analysis","prometheus-exporter","ssh"],"created_at":"2024-10-20T15:01:35.691Z","updated_at":"2024-10-20T15:04:41.588Z","avatar_url":"https://github.com/stohrendorf.png","language":"C#","funding_links":[],"categories":["C# #"],"sub_categories":[],"readme":"# CSV Prometheus Exporter\n\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstohrendorf%2Fcsv-prometheus-exporter.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstohrendorf%2Fcsv-prometheus-exporter?ref=badge_shield)\n\n**Usecase:** If you have a service on many servers and want to expose throughput-like metrics with thousands of events\nper second to prometheus, and you can access the servers through SSH, this is for you. Just add a CSV-like log writer to\nyour software, attach this exporter to the servers, and attach Prometheus to this service. Instead of transferring\nmegabytes of Prometheus protocol data over the network, this service extracts and accummulates metrics incrementally\nfrom the attached services, reducing traffic to a fraction. The time-resolution precision of the metrics is only based\non the sampling frequency.\n\nIt is capable of processing at least 100 servers with thousands of requests per second on a small VM (peak usage 16\ncores, average usage 4 cores, average RAM usage below 300MB, average incoming SSH traffic below 600kB/s, not including\nresource requirements for Prometheus itself).\n\n\u003e \"CSV-like\" in this case means \"space-separated, double-quote delimited format\"; this piece of software was primarily\n\u003e developed for parsing access logs, but if needed, it can be extended to parse any CSV-based format that\n\u003e [FastCsvParser](https://github.com/bopohaa/CsvParser) can handle.\n\nMetrics are exposed at `host:5000/metrics`.\n\n## Configuration\n\nThe configuration format is defined as follows. Identifiers prefixed with a `$` are names that can be\nchosen freely. Stuff enclosed within `[...]` is optional, the text enclosed within `\u003c...\u003e` describes\nthe expected types. Please note that the tilde (`~`) is equivalent to `null`.\n\n```\nglobal:\n  # The metrics' time-to-live.\n  ttl: \u003cseconds = 60\u003e\n  [background-resilience: \u003cinteger = 1\u003e] # how many \"ttl\" time spans to keep the metrics in background after they\n                                         # exceed their ttl\n  [long-term-resilience: \u003cinteger = 10\u003e] # how many \"ttl\" time spans to keep long-term metrics in background after they\n                                         # exceed their ttl\n  # If prefix is set, all metrics (including process metrics)\n  # will be exposed as \"prefix:metric-name\".\n  [prefix: \u003cstring\u003e]\n  [histograms: \u003cbuckets-list\u003e]\n  format: \u003cmetrics-list\u003e\n\n[script: \u003cscript-name\u003e]\n# If a script is given, but no reload-interval, it is executed only once at startup.\n[reload-interval: \u003cseconds\u003e]\n\nssh:\n  [\u003cconnection-settings\u003e]\n  environments: \u003clist-of-environments\u003e\n\n\u003cbuckets-list\u003e :=\n  # Numbers do not need to be ordered, an implicit \"+Inf\" will be added.\n  # If the values are not set, the default will be used, which is\n  # [.005, .01, .025, .05, .075, .1, .25, .5, .75, 1, 2.5, 5, 7.5, 10]\n  $bucket_name_1: \u003c~ | list-of-numbers\u003e\n  $bucket_name_2: \u003c~ | list-of-numbers\u003e\n  ...\n\n\u003cmetrics-list\u003e :=\n  $metric_name_1: \u003ctype\u003e\n  $metric_name_2: \u003ctype\u003e\n  \n\u003ctype\u003e :=\n  # Example: clf_number + request_bytes_sent \n  (number | clf_number | label | request_header) [+ $bucket_name]\n\n\u003clist-of-environments\u003e :=\n  $environment_name_1:\n    hosts: \u003chostnames-or-ip-list\u003e\n    [connection: \u003cconnection-settings\u003e]\n  $environment_name_2:\n    hosts: \u003chostnames-or-ip-list\u003e\n    [connection: \u003cconnection-settings\u003e]\n  ...\n\n# Note that a few restrictions exist for the connection settings.\n#   1. \"file\" and \"user\" are defined as required, but this means only\n#      that they must be either set at the global \"ssh\" level, or at\n#      the environment level.\n#   2. At least one of \"password\" or \"pkey\" must be set.\n#   3. If one of the settings is not set explicitly on environment level,\n#      the value is inherited from the global \"ssh\" level.\n\u003cconnection-settings\u003e :=\n  file: \u003cstring\u003e\n  user: \u003cstring\u003e\n  [password: \u003cstring\u003e]\n  [pkey: \u003cstring\u003e]\n  [connect-timeout: \u003cseconds = 30\u003e]\n  [read-timeout-ms: \u003cms = 60000]\n```\n\nThe supported `type` values are:\n\n* `number` for floating point values, exposed as a `counter`, or as a `histogram`.\n* `clf_number`, which behaves the same as `number`, except that a single `-` will be mapped to zero.\n* `label` will use the column's value as a label.\n* `request_header` expects a HTTP request header; a value of `POST /foo HTTP/1.1` will emit the labels\n  `request_method=\"POST\"`, `request_uri=\"/foo\"` and `request_http_version=\"HTTP/1.1\"`.\n\nEach metric (except process metrics) will contain the label `environment`.\n\nScalar metrics will be accumulated; successfully processed lines will be counted in `lines_parsed`, with its labels\nset to the CSV line's labels. If something bad happens, the erroneous lines will be counted in `parser_errors`,\nbut as the entry could not reliably parsed at this point, it will only contain the `environment` label.\n\nFor example, to parse access logs, you could use this.\n\n```yaml\nformat: # based on \"%h %l %u %t \\\"%r\\\" %\u003es %b\"\n- remote_host: label\n- ~ # ignore remote logname\n- remote_user: label\n- ~ # ignore timestamp\n- request_header: request_header # special parser that emits the labels \"request_http_version\", \"request_uri\" and \"request_method\"\n- status: label\n- body_bytes_sent: clf_number  # maps a single dash to zero, otherwise behaves like \"number\"\n```\n\nPlace your `scrapeconfig.yml` either in `/etc`, or\nprovide the environment variable `SCRAPECONFIG` with a config file path;\n[see here for a config file example](./scrapeconfig.example.yml), showing all of its features.\n\n# Installation\n\nA docker image, containing `python3` and `curl`, is available\n[here](https://hub.docker.com/r/stohrendorf/csv-prometheus-exporter/).\n\n# Technical \u0026 Practical Notes\n\n## The TTL Thing\n\nMetrics track when they were last updated. If a metric doesn't change within the TTL specified in the\nconfig file (which defaults to 60 seconds), it will not be exposed via `/metrics` anymore; this is the\nfirst phase of garbage collection to avoid excessive traffic. If a metric is in the first phase of garbage\ncollection, and doesn't receive an update for another `background-resilience` periods of the specified TTL,\nit will be fully evicted from the processing.\n\nA few metrics (namely, the `parser_errors` and `lines_parsed` metrics) are in \"long-term mode\"; they will be evicted\nafter `long-term-resilience` periods of the TTL; the `connected` metric will never be evicted.\n\nPractice has shown that this multi-phase metric garbage collection is strictly necessary to avoid excessive\nresponse sizes and to avoid the process to be clogged up by processing dead metrics. It doesn't have any known\nserious impacts on the metrics' values, though.\n\n## A note about Prometheus performance\n\nPerformance matters, and the exported metrics are not usable immediately in most cases. The following\nPrometheus rules have been tested in high-traffic situations, and sped up Prometheus queries immensely.\n\n**Adjust as necessary, these are only examples.**\n\n```yaml\ngroups:\n- name: access_logs_precompute\n  interval: 5s\n  rules:\n  - record: \"prefix:lines_parsed_per_second\"\n    expr: \"irate(prefix:lines_parsed_total[1m])\"\n  - record: \"prefix:body_bytes_sent_per_second\"\n    expr: \"irate(prefix:body_bytes_sent_total[1m])\"\n  - record: \"prefix:request_length_per_second\"\n    expr: \"irate(prefix:request_length_total[1m])\"\n  - record: \"prefix:request_time_per_second\"\n    expr: \"irate(prefix:request_time_total[1m])\"\n```\n\n## License\n\n[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fstohrendorf%2Fcsv-prometheus-exporter.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fstohrendorf%2Fcsv-prometheus-exporter?ref=badge_large)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstohrendorf%2Fcsv-prometheus-exporter","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstohrendorf%2Fcsv-prometheus-exporter","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstohrendorf%2Fcsv-prometheus-exporter/lists"}