{"id":28501926,"url":"https://github.com/fluent/fluent-plugin-sql","last_synced_at":"2025-07-05T01:30:36.923Z","repository":{"id":4314333,"uuid":"5447720","full_name":"fluent/fluent-plugin-sql","owner":"fluent","description":"SQL input/output plugin for Fluentd","archived":false,"fork":false,"pushed_at":"2024-03-22T08:25:44.000Z","size":140,"stargazers_count":111,"open_issues_count":53,"forks_count":60,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-06-08T16:08:23.778Z","etag":null,"topics":["activerecord","fluentd","fluentd-plugin","rdbms","sql"],"latest_commit_sha":null,"homepage":null,"language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fluent.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-08-17T03:43:44.000Z","updated_at":"2025-03-04T09:32:21.000Z","dependencies_parsed_at":"2024-02-28T14:45:10.847Z","dependency_job_id":"d7667ea0-e78e-4617-9a1e-09ef5ab211f4","html_url":"https://github.com/fluent/fluent-plugin-sql","commit_stats":{"total_commits":155,"total_committers":27,"mean_commits":"5.7407407407407405","dds":0.7032258064516129,"last_synced_commit":"48e3927d27fa6ff53a846f1a95d33b02094888cb"},"previous_names":["frsyuki/fluent-plugin-sql"],"tags_count":20,"template":false,"template_full_name":null,"purl":"pkg:github/fluent/fluent-plugin-sql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fluent","download_url":"https://codeload.github.com/fluent/fluent-plugin-sql/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fluent%2Ffluent-plugin-sql/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262894217,"owners_count":23380778,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["activerecord","fluentd","fluentd-plugin","rdbms","sql"],"created_at":"2025-06-08T16:08:29.278Z","updated_at":"2025-07-05T01:30:36.917Z","avatar_url":"https://github.com/fluent.png","language":"Ruby","readme":"# SQL input plugin for [Fluentd](http://fluentd.org) event collector\n\n## Overview\n\nThis SQL plugin has two parts:\n\n1. SQL **input** plugin reads records from RDBMSes periodically. An example use case would be getting \"diffs\" of a table (based on the \"updated_at\" field).\n2. SQL **output** plugin that writes records into RDBMSes. An example use case would be aggregating server/app/sensor logs into RDBMS systems.\n\n## Requirements\n\n| fluent-plugin-sql | fluentd    | ruby   |\n|-------------------|------------|--------|\n| \u003e= 1.0.0          | \u003e= v0.14.4 | \u003e= 2.1 |\n| \u003c  1.0.0          | \u003c  v0.14.0 | \u003e= 1.9 |\n\nNOTE: fluent-plugin-sql v2's buffer format is different from v1. If you update the plugin to v2, don't reuse v1's buffer.\n\n## Installation\n\n    $ fluent-gem install fluent-plugin-sql --no-document\n    $ fluent-gem install pg --no-document # for postgresql\n\nYou should install actual RDBMS driver gem together. `pg` gem for postgresql adapter or `mysql2` gem for `mysql2` adapter. Other adapters supported by [ActiveRecord](https://github.com/rails/rails/tree/master/activerecord) should work.\n\nWe recommend that mysql2 gem is higher than `0.3.12` and pg gem is higher than `0.16.0`.\n\nIf you use ruby 2.1, use pg gem 0.21.0 (\u003c 1.0.0) because ActiveRecord 5.1.4 or earlier doesn't support Ruby 2.1.\n\n## Input: How It Works\n\nThis plugin runs following SQL periodically:\n\nSELECT * FROM *table* WHERE *update\\_column* \u003e *last\\_update\\_column\\_value* ORDER BY *update_column* ASC LIMIT 500\n\nWhat you need to configure is *update\\_column*. The column should be an incremental column (such as AUTO\\_ INCREMENT primary key) so that this plugin reads newly INSERTed rows. Alternatively, you can use a column incremented every time when you update the row (such as `last_updated_at` column) so that this plugin reads the UPDATEd rows as well.\nIf you omit to set *update\\_column* parameter, it uses primary key.\n\nIt stores last selected rows to a file (named *state\\_file*) to not forget the last row when Fluentd restarts.\n\n## Input: Configuration\n\n    \u003csource\u003e\n      @type sql\n\n      host rdb_host\n\t  port rdb_port\n      database rdb_database\n      adapter mysql2_or_postgresql_or_etc\n      username myusername\n      password mypassword\n\n      tag_prefix my.rdb  # optional, but recommended\n\n      select_interval 60s  # optional\n      select_limit 500     # optional\n\n      state_file /var/run/fluentd/sql_state\n\n      \u003ctable\u003e\n        table table1\n        tag table1  # optional\n        update_column update_col1\n        time_column time_col2  # optional\n      \u003c/table\u003e\n\n      \u003ctable\u003e\n        table table2\n        tag table2  # optional\n        update_column updated_at\n        time_column updated_at  # optional\n        time_format %Y-%m-%d %H:%M:%S.%6N # optional\n      \u003c/table\u003e\n\n      # detects all tables instead of \u003ctable\u003e sections\n      #all_tables\n    \u003c/source\u003e\n\n* **host** RDBMS host\n* **port** RDBMS port\n* **database** RDBMS database name\n* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)\n* **username** RDBMS login user name\n* **password** RDBMS login password\n* **tag_prefix** prefix of tags of events. actual tag will be this\\_tag\\_prefix.tables\\_tag (optional)\n* **select_interval** interval to run SQLs (optional)\n* **select_limit** LIMIT of number of rows for each SQL (optional)\n* **state_file** path to a file to store last rows\n* **all_tables** reads all tables instead of configuring each tables in \\\u003ctable\\\u003e sections\n\n\\\u003ctable\\\u003e sections:\n\n* **tag** tag name of events (optional; default value is table name)\n* **table** RDBM table name\n* **update_column**: see above description\n* **time_column** (optional): if this option is set, this plugin uses this column's value as the the event's time. Otherwise it uses current time.\n* **primary_key** (optional): if you want to get data from the table which doesn't have primary key like PostgreSQL's View, set this parameter.\n* **time_format** (optional): if you want to specify the format of the date used in the query, useful when using alternative adapters which have restrictions on format\n\n## Input: Limitation\n\nYou should make sure target tables have index (and/or partitions) on the *update\\_column*. Otherwise SELECT causes full table scan and serious performance problem.\n\nYou can't replicate DELETEd rows.\n\n## Output: How It Works\n\nThis plugin takes advantage of ActiveRecord underneath. For `host`, `port`, `database`, `adapter`, `username`, `password`, `socket` parameters, you can think of ActiveRecord's equivalent parameters.\n\n## Output: Configuration\n\n    \u003cmatch my.rdb.*\u003e\n      @type sql\n      host rdb_host\n      port 3306\n      database rdb_database\n      adapter mysql2_or_postgresql_or_etc\n      username myusername\n      password mypassword\n      socket path_to_socket\n      remove_tag_prefix my.rdb # optional, dual of tag_prefix in in_sql\n\n      \u003ctable\u003e\n        table table1\n        column_mapping 'timestamp:created_at,fluentdata1:dbcol1,fluentdata2:dbcol2,fluentdata3:dbcol3'\n        # This is the default table because it has no \"pattern\" argument in \u003ctable\u003e\n        # The logic is such that if all non-default \u003ctable\u003e blocks\n        # do not match, the default one is chosen.\n        # The default table is required.\n      \u003c/table\u003e\n\n      \u003ctable hello.*\u003e # You can pass the same pattern you use in match statements.\n        table table2\n        # This is the non-default table. It is chosen if the tag matches the pattern\n        # AFTER remove_tag_prefix is applied to the incoming event. For example, if\n        # the message comes in with the tag my.rdb.hello.world, \"remove_tag_prefix my.rdb\"\n        # makes it \"hello.world\", which gets matched here because of \"pattern hello.*\".\n      \u003c/table\u003e\n      \n      \u003ctable hello.world\u003e\n        table table3\n        # This is the second non-default table. You can have as many non-default tables\n        # as you wish. One caveat: non-default tables are matched top-to-bottom and\n        # the events go into the first table it matches to. Hence, this particular table\n        # never gets any data, since the above \"hello.*\" subsumes \"hello.world\".\n      \u003c/table\u003e\n    \u003c/match\u003e\n\n* **host** RDBMS host\n* **port** RDBMS port\n* **database** RDBMS database name\n* **adapter** RDBMS driver name. You should install corresponding gem before start (mysql2 gem for mysql2 adapter, pg gem for postgresql adapter, etc.)\n* **username** RDBMS login user name\n* **password** RDBMS login password\n* **socket** RDBMS socket path\n* **pool** A connection pool synchronizes thread access to a limited number of database connections\n* **timeout** RDBMS connection timeout\n* **remove_tag_prefix** remove the given prefix from the events. See \"tag_prefix\" in \"Input: Configuration\". (optional)\n\n\\\u003ctable\\\u003e sections:\n\n* **table** RDBM table name\n* **column_mapping**: [Required] Record to table schema mapping. The format is consists of `from:to` or `key` values are separated by `,`. For example, if set 'item_id:id,item_text:data,updated_at' to **column_mapping**, `item_id` field of record is stored into `id` column and `updated_at` field of record is stored into `updated_at` column.\n* **\\\u003ctable pattern\\\u003e**: the pattern to which the incoming event's tag (after it goes through `remove_tag_prefix`, if given). The patterns should follow the same syntax as [that of \\\u003cmatch\\\u003e](https://docs.fluentd.org/configuration/config-file#how-match-patterns-work). **Exactly one \\\u003ctable\\\u003e element must NOT have this parameter so that it becomes the default table to store data**.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent%2Ffluent-plugin-sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffluent%2Ffluent-plugin-sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffluent%2Ffluent-plugin-sql/lists"}