{"id":19212383,"url":"https://github.com/leafo/lapis-stats","last_synced_at":"2025-05-12T20:37:44.354Z","repository":{"id":66222845,"uuid":"70853489","full_name":"leafo/lapis-stats","owner":"leafo","description":"Statsd and Influxdb support for Lua, OpenResty \u0026 Lapis","archived":false,"fork":false,"pushed_at":"2025-04-12T19:48:55.000Z","size":31,"stargazers_count":8,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-20T17:39:52.782Z","etag":null,"topics":["influxdb","lapis","lua","luasocket","moonscript","statsd"],"latest_commit_sha":null,"homepage":null,"language":"Lua","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leafo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2016-10-13T22:33:13.000Z","updated_at":"2025-04-12T19:48:59.000Z","dependencies_parsed_at":null,"dependency_job_id":"df0b917f-0c22-4fae-bc48-19ae64a04f40","html_url":"https://github.com/leafo/lapis-stats","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leafo%2Flapis-stats","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leafo%2Flapis-stats/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leafo%2Flapis-stats/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leafo%2Flapis-stats/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leafo","download_url":"https://codeload.github.com/leafo/lapis-stats/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253817586,"owners_count":21969007,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["influxdb","lapis","lua","luasocket","moonscript","statsd"],"created_at":"2024-11-09T13:46:44.589Z","updated_at":"2025-05-12T20:37:44.320Z","avatar_url":"https://github.com/leafo.png","language":"Lua","readme":"# lapis-stats\n\nVarious helper modules for metrics and stats collection for Lua, OpenResty \u0026amp; Lapis\n\n* [statsd](#statsd)\n* [victoriametrics](#victoriametrics)\n* [influxdb](#influxdb)\n\n\n## Lapis Actions\n\nThis library includes command-line actions that can be executed using the Lapis\ncommand-line tool. These actions are designed to collect metrics from various\nsystem components and format them for ingestion into monitoring systems like\nVictoriaMetrics.\n\nThey are typically run like this:\n\n```bash\nlapis _ \u003caction_name\u003e [options]\n```\n\n### `stat_system`\n\nCollects system metrics like CPU and Disk usage and outputs them in Prometheus\nexposition format. It relies on standard Linux command-line tools (`mpstat`,\n`df`, `iostat`).\n\n**Usage:**\n\n```bash\n# Print system metrics to standard output\nlapis _ stat_system\n\n# Send metrics directly to VictoriaMetrics (configured in Lapis config)\nlapis _ stat_system --send\n\n# Only collect CPU metrics, average over 5 seconds\nlapis _ stat_system --skip-disk --interval 5\n\n# Specify hostname label explicitly\nlapis _ stat_system --hostname my-server-01\n```\n\nExample output:\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand output\u003c/summary\u003e\n\n```\ncpu_usage_percent{host=\"my-server\",cpu=\"all\",mode=\"usr\"} 0.41\ncpu_usage_percent{host=\"my-server\",cpu=\"all\",mode=\"sys\"} 0.27\ncpu_usage_percent{host=\"my-server\",cpu=\"all\",mode=\"iowait\"} 0.09\ncpu_usage_percent{host=\"my-server\",cpu=\"all\",mode=\"irq\"} 0.08\ncpu_usage_percent{host=\"my-server\",cpu=\"all\",mode=\"soft\"} 0.04\ncpu_usage_percent{host=\"my-server\",cpu=\"all\",mode=\"idle\"} 98.81\ndisk_used_bytes{host=\"my-server\",mount=\"/dev\"} 0\ndisk_available_bytes{host=\"my-server\",mount=\"/dev\"} 16351863\ndisk_usage_percent{host=\"my-server\",mount=\"/dev\"} 0\ndisk_used_bytes{host=\"my-server\",mount=\"/\"} 672345435\ndisk_available_bytes{host=\"my-server\",mount=\"/\"} 462930847\ndisk_usage_percent{host=\"my-server\",mount=\"/\"} 59.184189251962\ndisk_total_read_kb{host=\"my-server\",device=\"nvme0n1\"} 7837643\ndisk_total_written_kb{host=\"my-server\",device=\"nvme0n1\"} 190737331\n```\n\n\u003c/details\u003e\n\n**Options:**\n\n*   `--send`: Send metrics directly to the VictoriaMetrics server configured in the Lapis application configuration under the `victoriametrics` key instead of printing out the metrics.\n*   `--skip-cpu`: Do not collect CPU metrics.\n*   `--skip-disk`: Do not collect Disk metrics (usage, available, read/write stats).\n*   `--hostname`: Manually specify the hostname to be used in the `host` label for all metrics. Defaults to the system's hostname.\n*   `--interval \u003cseconds\u003e`: The interval (in seconds) over which to average CPU usage when running `mpstat`. Defaults to `2`.\n\n**Dependencies:** Requires `mpstat` (often part of the `sysstat` package), `df`, and `iostat` to be installed and available in the system's `PATH`.\n\n### `stat_postgres`\n\nCollects metrics from a PostgreSQL database instance, including database statistics and optionally PgBouncer statistics. Outputs metrics in Prometheus exposition format.\n\n**Usage:**\n\n```bash\n# Print PostgreSQL metrics for the configured database to standard output\nlapis _ stat_postgres\n\n# Include PgBouncer metrics\nlapis _ stat_postgres --pgbouncer\n\n# Send metrics directly to VictoriaMetrics\nlapis _ stat_postgres --send --pgbouncer\n```\n\nExample output:\n\n\u003cdetails\u003e\n\u003csummary\u003eClick to expand output\u003c/summary\u003e\n\n```\npg_stat_database_tup_returned{db=\"myapp_prod\"} 7494922644470\npg_stat_database_tup_fetched{db=\"myapp_prod\"} 3764706534927\npg_stat_database_tup_updated{db=\"myapp_prod\"} 6195027996\npg_stat_database_tup_deleted{db=\"myapp_prod\"} 370563436\npg_stat_database_tup_inserted{db=\"myapp_prod\"} 850411495\npg_stat_database_xact_commit{db=\"myapp_prod\"} 59651845850\npg_stat_database_xact_rollback{db=\"myapp_prod\"} 638\npg_stat_database_temp_files{db=\"myapp_prod\"} 36705\npg_stat_database_temp_bytes{db=\"myapp_prod\"} 858426472150\npg_stat_database_blks_read{db=\"myapp_prod\"} 72100357307\npg_stat_database_blks_hit{db=\"myapp_prod\"} 7022588231307\npg_stat_database_conflicts{db=\"myapp_prod\"} 0\npg_stat_database_blk_read_time{db=\"myapp_prod\"} 0\npg_stat_database_blk_write_time{db=\"myapp_prod\"} 0\npg_total_size_bytes{db=\"myapp_prod\"} 1152421658079\npg_table_size_bytes{db=\"myapp_prod\"} 677746524615\npg_indexes_size_bytes{db=\"myapp_prod\"} 474675133449\npgbouncer_bytes_sent{db=\"myapp_prod\"} 84607100369358\npgbouncer_xact_time{db=\"myapp_prod\"} 63110802854863\npgbouncer_query_time{db=\"myapp_prod\"} 63110769149900\npgbouncer_wait_time{db=\"myapp_prod\"} 3311031621235\npgbouncer_xact_count{db=\"myapp_prod\"} 59618950825\npgbouncer_query_count{db=\"myapp_prod\"} 59618959099\npgbouncer_bytes_received{db=\"myapp_prod\"} 8191684799142\n```\n\n\u003c/details\u003e\n\n\n**Options:**\n\n*   `--send`: Send metrics directly to the VictoriaMetrics server configured in the Lapis application configuration under the `victoriametrics` key.\n*   `--pgbouncer`: Connect to the `pgbouncer` database (using the same credentials as the main database connection configured in Lapis) and collect statistics using `SHOW stats_totals`. \u003chttps://www.pgbouncer.org/usage.html#show-commands\u003e\n\n**Dependencies:** Requires PostgreSQL connection details to be configured in the Lapis application configuration (e.g., under the `postgres` key). If `--pgbouncer` is used, the server must be pgbouncer so that metrics can be collected\n\n\n## statsd\n\nThe `lapis.statsd` module lets you send metrics to statsd compatible\naggregation server over UDP socket. This is suitable for high-throughput\nmetrics collection. Inside of OpenResty the non-blocking co-socket API is\nused. Otherwise, LuaSocket is used.\n\n\u003e Note: I recommend using [statsite](https://github.com/statsite/statsite)\n\nYou must configure your statsd location in your Lapis config:\n\n```lua\n-- config.lua\n\nlocal config = require(\"lapis.config\")\n\nconfig(\"development\", function()\n  statsd {\n    host = \"127.0.0.1\",\n    port = 8125,\n    -- debug: true,\n  }\nend)\n```\n\nInclude the module from `lapis.statsd`\n\n```lua\nlocal statsd = require(\"lapis.statsd\")\n\napp:get(\"/hello\", function(self)\n  statsd.counter(\"my_counter\", 5)\n  statsd.timer(\"my_counter\", 100)\n  statsd.value(\"hello\", 9)\n  statsd.guage(\"some_guage\", -1)\nend)\n```\n\nIf you're sending many metrics at once then you can take advantage of the\n`Pipeline` interface:\n\n```lua\napp:get(\"/hello\", function(self)\n  p = statsd.Pipeline()\n  p:counter(\"my_counter\", 5)\n  p:timer(\"my_counter\", 100)\n  p:value(\"hello\", 9)\n  p:guage(\"some_guage\", -1)\n  p:flush()\nend)\n```\n\n### Reference\n\n* `timer(key, value)`\n* `counter(key, value)`\n* `guage(key, value)`\n* `value(key, value)`\n\nThe `Pipeline` instance exposes all of the same functions, but as methods. (So\nyou should call them using `:`)\n\n\n\n## victoriametrics\n\nThe `lapis.victoriametrics` module provides a client to interface with a\n[VictoriaMetrics](https://victoriametrics.com/) server over its HTTP API.\n\nYou must configure your VictoriaMetrics server details in your Lapis config:\n\n```lua\n-- config.lua\n\nlocal config = require(\"lapis.config\")\n\nconfig(\"development\", {\n  victoriametrics = {\n    -- host = \"127.0.0.1\", -- default\n    -- port = 8428, -- default\n    -- username = \"my_user\", -- optional\n    -- password = \"my_password\", -- required if username is set\n  }\n})\n```\n\nYou can then get a client instance and interact with the API:\n\n```lua\nlocal vm = require(\"lapis.victoriametrics\").get_client()\n\n-- Execute an instant query\nlocal res, err = vm:query('sum(rate(my_counter_total[5m]))')\nif res then\n  print(res.status) -- \"success\"\n  -- process res.data\nend\n\n-- Execute a range query\nlocal start_time = os.time() - 3600 -- 1 hour ago\nlocal end_time = os.time()\nlocal res, err = vm:query_range('http_requests_total{host=\"example.com\"}', start_time, end_time, \"1m\")\n\n-- Write data using Prometheus exposition format\nlocal ok = vm:write([[\nmy_metric{label=\"value1\"} 123\nanother_metric{host=\"serverA\",region=\"us-east\"} 45.67\n]])\n\nif ok then\n  print(\"Write successful\")\nend\n```\n\n### Module Reference\n\n#### `encode_metric(name, labels, value)`\n\nFormats a single metric into the Prometheus exposition format string. This is a helper function used internally by the `write` method.\n\n*   `name`: (String) The name of the metric.\n*   `labels`: (Optional Table) A table of key-value pairs representing the metric's labels. Keys and values should be strings.\n*   `value`: (Optional Number/String) The value of the metric. Can be omitted if the metric doesn't have a value (e.g., info metrics, though less common).\n\nReturns a string formatted according to the [Prometheus Exposition Format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md).\n\n```lua\nlocal victoriametrics = require(\"lapis.victoriametrics\")\n\nlocal metric_string = victoriametrics.encode_metric(\"http_requests_total\", {\n  method = \"POST\",\n  path = \"/api/users\"\n}, 1027)\n\nprint(metric_string)\n-- Output: http_requests_total{method=\"POST\",path=\"/api/users\"} 1027\n\nlocal metric_string_no_value = victoriametrics.encode_metric(\"build_info\", {\n  version = \"1.2.3\",\n  revision = \"abcdef\"\n})\n\nprint(metric_string_no_value)\n-- Output: build_info{version=\"1.2.3\",revision=\"abcdef\"}\n```\n\n#### `get_client()`\n\nGet the singleton instance of the VictoriaMetrics client configured via the Lapis\nconfiguration.\n\n```lua\nlocal vm = require(\"lapis.victoriametrics\").get_client()\n```\n\n#### `write(metrics)`\n\nHelper function to write metrics to the singleton client returned by `get_client()`.\n\n#### `VictoriaMetrics`\n\nCreates a new VictoriaMetrics client instance with a specific configuration\ntable, bypassing the global Lapis configuration. The `config` table expects keys\nlike `host`, `port`, `username`, `password`.\n\n```lua\nlocal VictoriaMetrics = require(\"lapis.victoriametrics\").VictoriaMetrics\n\n-- Create a client connected to a specific server, ignoring global config\nlocal client = VictoriaMetrics({\n  host = \"victoriametrics.internal.example.com\",\n  port = 8428,\n  username = \"importer\",\n  password = \"supersecretpassword\"\n})\n\n-- Use the custom client\nlocal ok = client:write(\"my_custom_metric 123\")\n```\n\n### Client Reference\n\n#### `client:query(query, time, step)`\n\nExecutes an instant query at a single point in time.\nSee [VictoriaMetrics Instant Query API](https://docs.victoriametrics.com/keyConcepts.html#instant-query).\n\n*   `query`: (String) The MetricSQL query to execute.\n*   `time`: (Optional Number) Unix timestamp in seconds for the evaluation time. Defaults to now if omitted.\n*   `step`: (Optional String/Number) Evaluation step resolution.\n\nReturns a table containing the decoded JSON response on success (HTTP 200), or\n`nil` and an error message string on failure.\n\n#### `client:query_range(query, start, end, step)`\n\nExecutes a query over a range of time.\nSee [VictoriaMetrics Range Query API](https://docs.victoriametrics.com/keyConcepts.html#range-query).\n\n*   `query`: (String) The MetricSQL query to execute.\n*   `start`: (Number) Start Unix timestamp in seconds.\n*   `end`: (Number) End Unix timestamp in seconds.\n*   `step`: (Optional String/Number) Query resolution step width (e.g., \"1m\", 60).\n\nReturns a table containing the decoded JSON response on success (HTTP 200), or\n`nil` and an error message string on failure.\n\n#### `client:export(match, start, end)`\n\nExports raw data samples in JSON line format.\nSee [VictoriaMetrics Export API](https://docs.victoriametrics.com/#how-to-export-data-in-json-line-format).\n\n*   `match`: (String) A time series selector for filtering (e.g., `{__name__=\"my_metric\",job=\"my_job\"}`). Pass `true` to export *all* data (use with caution).\n*   `start`: (Optional Number) Start Unix timestamp in seconds.\n*   `end`: (Optional Number) End Unix timestamp in seconds.\n\nReturns the raw response body (JSON lines) as a string on success (HTTP 200), or\n`nil` and an error message string on failure.\n\n#### `client:write(metrics)`\n\nWrites time series data using the Prometheus exposition format.\nSee [Prometheus Exposition Format](https://github.com/prometheus/docs/blob/main/content/docs/instrumenting/exposition_formats.md) and [VictoriaMetrics Import API](https://docs.victoriametrics.com/#how-to-import-data-in-prometheus-exposition-format).\n\n*   `metrics`: (String) A string containing one or more metrics in Prometheus text format, separated by newlines.\n\nExample format:\n```\nmetric_name{label1=\"value1\",label2=\"value2\"} 123.45\n```\n(Timestamp in ms can optionally be provided at the end).\n\nReturns `true` on success (HTTP 204), or `false` otherwise.\n\n#### `client:import(data)`\n\nImports data using the VictoriaMetrics native import format (JSON line format).\nSee [VictoriaMetrics Import API](https://docs.victoriametrics.com/#how-to-import-data-in-json-line-format).\n\n*   `data`: (String) A string containing one or more data points in JSON line format, separated by newlines.\n\nExample format:\n```json\n{\"metric\":{\"__name__\": \"metric_name\", \"label1\": \"value1\"}, \"values\": [10], \"timestamps\": [1640995200000]}\n```\n\nReturns the decoded JSON response table on success, or `nil` and an error\nmessage string on failure. Status code checking might be required depending on\nAPI behavior for partial success/failure.\n\n#### `client:delete_series(name, name_confirm)`\n\n**WARNING: This is a destructive operation and cannot be undone.**\n\nDeletes *all* data points for the time series matching the provided selector(s).\nDue to the potential impact, the selector must be provided twice for confirmation.\nSee [VictoriaMetrics Delete API](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#how-to-delete-time-series).\n\n*   `name`: (String) The time series selector to delete (e.g., `http_requests_total`, `{job=\"my_app\"}`). This corresponds to the `match[]` parameter in the VM API.\n*   `name_confirm`: (String) Must be identical to `name` to confirm the deletion.\n\nReturns the raw response `body` and `status` code from the VictoriaMetrics API.\n\n#### `client:_request(path, opts)`\n\nMakes a manual HTTP request to the VictoriaMetrics server. Only use this function if none of the other client methods are suitable\n*   `path`: (String) The API endpoint path (e.g., `api/v1/query`).\n*   `opts`: (Optional Table) Request options:\n    *   `method`: (String) HTTP method (e.g., \"GET\", \"POST\"). Defaults to \"GET\" or \"POST\" if `body` is present.\n    *   `body`: (String or Table) The request body. If a table, it's URL-encoded (`application/x-www-form-urlencoded`).\n    *   `params`: (Table) A table of key-value pairs to be URL-encoded as query parameters.\n    *   `headers`: (Table) A table of additional HTTP headers.\n\n\n## influxdb\n\n\u003e **Warning:** This module was written for InfluxDB 1.x and has not been updated. Unknown if it works with newer versions of InfluxDB.\n\nThe `lapis.influxdb` module provides a way to configure and send data to\nInfluxDB over the HTTP API.\n\n\u003e TODO: only LuaSocket is used right now\n\nYou must configure your InfluxDB server in your Lapis config:\n\n```lua\n-- config.lua\n\nlocal config = require(\"lapis.config\")\n\nconfig(\"development\", function()\n  influxdb {\n    -- host = \"127.0.0.1\", -- default\n    -- port = 8086, -- default\n    username = \"influx\",\n    password = \"my-password\",\n    database = \"my-db\",\n  }\nend)\n```\n\nYou can then use the module to query data:\n\n```lua\nlocal influxdb = require(\"lapis.influxdb\")\n\nlocal res = influxdb.query([[\n  select * from \"counts.users\" where time \u003e now() - 1d group by time(1h)\n]])\n```\n\nOr write data points:\n\n```lua\nlocal res = influxdb.write {\n  \"count.users value=4\",\n  \"summary.ip count=32 tag=US\"\n}\n```\n\n### Reference\n\n#### `get_client()`\n\nGet the current instance of the InfluxDB client from the Lapis configuration.\n\n#### `query(query, values...)`\n\nSend a query to the current connection. The values are interpolated into the\nquery escaped where the character `?` appears.\n\n\n```lua\nlocal res = influxdb.query([[\n  select * from \"counts.users\" where tag = ?\n]], \"hello world\")\n```\n\nThe response is returned as an array table of results.\n\n#### `write(points)`\n\nWrite measurements to the database. `points` is an array table with all the\nmeasurements to write as a string. It uses the [same text syntax documented in\nthe InfluxDB\nmanual](https://docs.influxdata.com/influxdb/v1.0/guides/writing_data/).\n\n```lua\nlocal res = influxdb.write {\n  \"count.users value=4\",\n}\n```\n\n## Writing an InfluxDB sink for Statsd\n\nUsing this library you can write a command line script to use as a sink to\nhandle your `statsd` flush to InfluxDB.\n\n\u003e Note: I recommend using [statsite](https://github.com/statsite/statsite)\n\nYou might write something like this:\n\n```lua\n-- influxdb_sink.lua\nlocal influxdb = require(\"lapis.influxdb\").get_client()\n\nlocal points = {}\n\nfor line in io.stdin:lines() do\n  local name, val, time = line:match(\"^([^|]+)|([^|]+)|([^|]+)$\")\n  if name then\n     table.insert(points, name .. \" value=\" .. val)\n  end\nend\n\nif not next(points) then\n  return\nend\n\ninfluxdb:write(points)\n```\n\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleafo%2Flapis-stats","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleafo%2Flapis-stats","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleafo%2Flapis-stats/lists"}