{"id":18044668,"url":"https://github.com/groonga/fluent-plugin-groonga-query-log","last_synced_at":"2025-04-10T01:08:19.264Z","repository":{"id":36277073,"uuid":"40581553","full_name":"groonga/fluent-plugin-groonga-query-log","owner":"groonga","description":"A Fluentd plugin for parsing Groonga's query log","archived":false,"fork":false,"pushed_at":"2021-02-24T02:46:47.000Z","size":39,"stargazers_count":0,"open_issues_count":1,"forks_count":3,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-04-10T01:08:11.558Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/groonga.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-08-12T05:18:25.000Z","updated_at":"2021-02-24T02:46:49.000Z","dependencies_parsed_at":"2022-09-04T12:21:34.052Z","dependency_job_id":null,"html_url":"https://github.com/groonga/fluent-plugin-groonga-query-log","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groonga%2Ffluent-plugin-groonga-query-log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groonga%2Ffluent-plugin-groonga-query-log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groonga%2Ffluent-plugin-groonga-query-log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/groonga%2Ffluent-plugin-groonga-query-log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/groonga","download_url":"https://codeload.github.com/groonga/fluent-plugin-groonga-query-log/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248137887,"owners_count":21053775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-30T18:10:12.922Z","updated_at":"2025-04-10T01:08:19.246Z","avatar_url":"https://github.com/groonga.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# README\n\n## Name\n\nfluent-plugin-groonga-query-log\n\n## Description\n\nFluent-plugin-groonga-query-log is a Fluentd filter plugin to parse\n[Groonga](http://groonga.org/)'s\n[query log](http://groonga.org/docs/reference/log.html#query-log) with\nFluentd.\n\nYou can detect slow query in real time by using this plugin.\n\nHere is a sample configuration that stores slow queries to Groonga:\n\n    \u003csource\u003e\n      @type tail\n      path /var/log/groonga/query.log\n      pos_file /var/log/fluentd/groonga-query-log.pos\n      tag groonga.query\n      format none\n    \u003c/source\u003e\n\n    \u003cfilter groonga.query\u003e\n      @type groonga_query_log\n    \u003c/filter\u003e\n\n    \u003cfilter groonga.query\u003e\n      @type grep\n      regexp1 slow \\Atrue\\z\n    \u003c/filter\u003e\n\n    \u003cfilter groonga.query\u003e\n      @type record_transformer\n      enable_ruby true\n      renew_record true\n      keep_keys elapsed\n      \u003crecord\u003e\n        log ${JSON.generate(to_h)}\n      \u003c/record\u003e\n    \u003c/filter\u003e\n\n    \u003cmatch groonga.query\u003e\n      @type groonga\n      store_table SlowQueries\n\n      protocol http\n      host 127.0.0.1\n\n      buffer_type file\n      buffer_path /var/lib/fluentd/groonga.buffer\n      flush_interval 1\n    \u003c/match\u003e\n\nYou need to prepare your environment to use the configuration.\n\nCreate the following directories:\n\n    % sudo mkdir -p /var/log/fluentd\n    % sudo mkdir -p /var/lib/fluentd\n\nUser who runs Fluentd must have write permission of the\ndirectories. Set suitable permission to the directories:\n\n    % sudo chown -R fluentd-user:fluentd-user /var/log/fluentd\n    % sudo chown -R fluentd-user:fluentd-user /var/lib/fluentd\n\nRun Groonga that stores slow queries on `127.0.0.1`:\n\n    % groonga --protocol http -s DB_PATH\n\nRun `fluentd` with the configuration:\n\n    % fluentd --config groonga-slow-queries.conf\n\nNow, slow queries are stored `SlowQueries` table in Groonga:\n\n    % curl 'localhost:10041/d/select?table=SlowQueries\u0026output_pretty=yes'\n    [\n      [\n        0,\n        1453454123.58033,\n        8.70227813720703e-05\n      ],\n      [\n        [\n          [\n            66\n          ],\n          [\n            [\n              \"_id\",\n              \"UInt32\"\n            ],\n            [\n              \"elapsed\",\n              \"Float\"\n            ],\n            [\n              \"log\",\n              \"Text\"\n            ]\n          ],\n          [\n            1,\n            0.265,\n            \"{\\\"start_time\\\":...}\"\n          ],\n          [\n            2,\n            0.303,\n            \"{\\\"start_time\\\":...}\"\n          ],\n          ...\n        ]\n      ]\n    ]\n\nEach query log is stored as one record. Record has the following two\ncolumns:\n\n  * `elapsed`: The elapsed time to execute the query.\n\n  * `log`: The query details as JSON. It includes executed command,\n    elapsed time for each condition and so on.\n\n## Install\n\n    % gem install fluent-plugin-groonga-query-log\n\n## Usage\n\nYou can use `groonga-query-log` filter for parsing raw Groonga's query\nlog text.\n\nHere is a sample raw Groonga's query log text:\n\n    2015-08-12 15:50:40.130990|0x7fb07d113da0|\u003e/d/select?table=Entries\u0026match_columns=name\u0026query=xml\n    2015-08-12 15:50:40.296165|0x7fb07d113da0|:000000165177838 filter(10)\n    2015-08-12 15:50:40.296172|0x7fb07d113da0|:000000165184723 select(10)\n    2015-08-12 15:50:41.228129|0x7fb07d113da0|:000001097153433 output(10)\n    2015-08-12 15:50:41.228317|0x7fb07d113da0|\u003c000001097334986 rc=0\n\n`groonga-query-log` filter emits the following record by parsing the\nabove raw Groonga's query log text:\n\n    {\n      \"start_time\": \"2015-08-12T06:50:40.130990Z\",\n      \"last_time\": \"2015-08-12T06:50:41.228324Z\",\n      \"elapsed\": 1.0973349860000001,\n      \"return_code\": 0,\n      \"slow\": true,\n      \"command\": {\n        \"raw\": \"/d/select?table=Entries\u0026match_columns=name\u0026query=xml\",\n        \"name\": \"select\",\n        \"parameters\": [\n          {\n            \"key\": \"table\",\n            \"value\": \"Entries\"\n          },\n          {\n            \"key\": \"match_columns\",\n            \"value\": \"name\"\n          },\n          {\n            \"key\": \"query\",\n            \"value\": \"xml\"\n          }\n        ]\n      },\n      \"operations\": [\n        {\n          \"context\": \"query: xml\",\n          \"name\": \"filter\",\n          \"relative_elapsed\": 0.165177838,\n          \"slow\": true\n        },\n        {\n          \"context\": null,\n          \"name\": \"select\",\n          \"relative_elapsed\": 6.884999999999999e-06,\n          \"slow\": false\n        },\n        {\n          \"context\": null,\n          \"name\": \"output\",\n          \"relative_elapsed\": 0.93196871,\n          \"slow\": true\n        }\n      ]\n    }\n\nHere are parameters of this filter:\n\n  * `raw_data_column_name`: It specifies column name that stores raw\n    Groonga's query log text.\n    * Default: `message`\n\n  * `slow_operation_threshold`: It specifies threshold to treat an\n    operation is slow. If one or more operations in a query spend more\n    than the threshold, the query is slow query.\n    * Default: `0.1`\n\n  * `slow_response_threshold`: It specifies threshold to treat a\n    request is slow. If a request spends more than the threshold, the\n    query in the request is slow query.\n    * Default: `0.2`\n\n  * `flatten`: It specifies whether parsed query log is mapped to a\n    flat object or a nested object. A float object will be useful to\n    store the parsed log to non document oriented database such as\n    RDBMS.\n\n    Here is a sample record of parsed query log:\n\n        {\n          ...,\n          \"command\": {\n            \"raw\": \"/d/select?table=Entries\u0026match_columns=name\u0026query=xml\",\n            \"name\": \"select\",\n            \"parameters\": [\n              {\n                \"key\": \"table\",\n                \"value\": \"Entries\"\n              },\n              {\n                \"key\": \"match_columns\",\n                \"value\": \"name\"\n              },\n              {\n                \"key\": \"query\",\n                \"value\": \"xml\"\n              }\n            ]\n          },\n          ...\n        }\n\n    Here is the flatten record of the above record:\n\n        {\n          ...,\n          \"command.raw\": \"/d/select?table=Entries\u0026match_columns=name\u0026query=xml\",\n          \"command.name\": \"select\",\n          \"command.parameters[0].key\": \"table\",\n          \"command.parameters[0].value\": \"Entries\",\n          \"command.parameters[1].key\": \"match_columns\",\n          \"command.parameters[1].value\": \"name\",\n          \"command.parameters[0].key\": \"query\",\n          \"command.parameters[0].value\": \"xml\",\n          ...\n        }\n\n    * Default: `false` (nested object)\n\n  * `flatten_separator`: It specifies separator that is used when\n    `flatten` is `true`. If `flatten` is `true`, nested keys are\n    mapped to one flatten key. This separator is used to concatenate\n    nested keys.\n\n    `.` is used for nested object by default. For example,\n\n         {\n           \"a\": {\n             \"b\": 1\n           }\n         }\n\n    is flatten to the following:\n\n         {\n           \"a.b\": 1\n         }\n\n    `[...]` is used for element in an array by default. For example,\n\n         {\n           \"a\": [\n             1,\n             2\n           ]\n         }\n\n    is flatten to the following:\n\n         {\n           \"a[0]\": 1,\n           \"a[1]\": 2\n         }\n\n    If `\"_\"` is used as the separator,\n\n         {\n           \"a\": [\n             1,\n             2\n           ],\n           \"b\": {\n             \"c\": 3\n           }\n         }\n\n    is flatten to the following:\n\n         {\n           \"a_0\": 1,\n           \"a_1\": 2,\n           \"b_c\": 3\n         }\n\n    * Default: `.` for object and `[...]` for array\n\n## Authors\n\n* Kouhei Sutou `\u003ckou@clear-code.com\u003e`\n\n## License\n\nLGPL 3 or later. See doc/text/lgpl-3.txt for details.\n\n(Kouhei Sutou has a right to change the license including\ncontributed patches.)\n\n## Mailing list\n\n* English: [groonga-talk](https://lists.sourceforge.net/lists/listinfo/groonga-talk)\n* Japanese: [groonga-dev](http://lists.sourceforge.jp/mailman/listinfo/groonga-dev)\n\n## Source\n\nThe repository for fluent-plugin-groonga-query-log is on\n[GitHub](https://github.com/groonga/fluent-plugin-groonga-query-log/).\n\n## Thanks\n\n* ...\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroonga%2Ffluent-plugin-groonga-query-log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgroonga%2Ffluent-plugin-groonga-query-log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgroonga%2Ffluent-plugin-groonga-query-log/lists"}