https://github.com/toyama0919/fluent-plugin-mysql-bulk
fluent plugin mysql bulk insert is high performance and on duplicate key update respond
https://github.com/toyama0919/fluent-plugin-mysql-bulk
Last synced: 2 months ago
JSON representation
fluent plugin mysql bulk insert is high performance and on duplicate key update respond
- Host: GitHub
- URL: https://github.com/toyama0919/fluent-plugin-mysql-bulk
- Owner: toyama0919
- License: mit
- Created: 2014-01-03T13:10:39.000Z (over 11 years ago)
- Default Branch: master
- Last Pushed: 2016-02-22T07:46:15.000Z (about 9 years ago)
- Last Synced: 2025-03-01T07:51:12.426Z (3 months ago)
- Language: Ruby
- Homepage:
- Size: 26.4 KB
- Stars: 15
- Watchers: 4
- Forks: 12
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
**Deprecated : This plugin had been merged to [fluent-plugin-mysql](https://github.com/tagomoris/fluent-plugin-mysql). This plugin is not maintained anymore!!!!!**
---
# fluent-plugin-mysql-bulk, a plugin for [Fluentd](http://fluentd.org) [](http://travis-ci.org/toyama0919/fluent-plugin-mysql-bulk)
fluent plugin mysql bulk insert is high performance and on duplicate key update respond.
## Installation
### td-agent(Linux)
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mysql-bulk
### td-agent(Mac)
sudo /usr/local/Cellar/td-agent/1.1.XX/bin/fluent-gem install fluent-plugin-mysql-bulk
### fluentd only
gem install fluent-plugin-mysql-bulk
## Parameters
param|value
--------|------
host|database host(default: 127.0.0.1)
database|database name(require)
username|user(require)
password|password(default: blank)
column_names|bulk insert column (require)
key_names|value key names, ${time} is placeholder Time.at(time).strftime("%Y-%m-%d %H:%M:%S") (default : column_names)
table|bulk insert table (require)
on_duplicate_key_update|on duplicate key update enable (true:false)
on_duplicate_update_keys|on duplicate key update column, comma separator
## Configuration Example(bulk insert)
```
type mysql_bulk
host localhost
database test_app_development
username root
password hogehoge
column_names id,user_name,created_at,updated_at
table users
flush_interval 10s
```
Assume following input is coming:
```js
mysql.input: {"user_name":"toyama","created_at":"2014/01/03 21:35:15","updated_at":"2014/01/03 21:35:15","dummy":"hogehoge"}
mysql.input: {"user_name":"toyama2","created_at":"2014/01/03 21:35:21","updated_at":"2014/01/03 21:35:21","dummy":"hogehoge"}
mysql.input: {"user_name":"toyama3","created_at":"2014/01/03 21:35:27","updated_at":"2014/01/03 21:35:27","dummy":"hogehoge"}
```
then result becomes as below (indented):
```sql
+-----+-----------+---------------------+---------------------+
| id | user_name | created_at | updated_at |
+-----+-----------+---------------------+---------------------+
| 1 | toyama | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
| 2 | toyama2 | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
| 3 | toyama3 | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
+-----+-----------+---------------------+---------------------+
```
running query
```sql
INSERT INTO users (id,user_name,created_at,updated_at) VALUES (NULL,'toyama','2014/01/03 21:35:15','2014/01/03 21:35:15'),(NULL,'toyama2','2014/01/03 21:35:21','2014/01/03 21:35:21')
```
## Configuration Example(bulk insert , if duplicate error record update)
```
type mysql_bulk
host localhost
database test_app_development
username root
password hogehoge
column_names id,user_name,created_at,updated_at
table users
on_duplicate_key_update true
on_duplicate_update_keys user_name,updated_at
flush_interval 60s
```
Assume following input is coming:
```js
mysql.input: {"id":"1" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:03","updated_at":"2014/01/03 21:58:03"}
mysql.input: {"id":"2" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:06","updated_at":"2014/01/03 21:58:06"}
mysql.input: {"id":"3" ,"user_name":"toyama7","created_at":"2014/01/03 21:58:08","updated_at":"2014/01/03 21:58:08"}
mysql.input: {"id":"10","user_name":"toyama7","created_at":"2014/01/03 21:58:18","updated_at":"2014/01/03 21:58:18"}
```
then result becomes as below (indented):
```sql
+-----+-----------+---------------------+---------------------+
| id | user_name | created_at | updated_at |
+-----+-----------+---------------------+---------------------+
| 1 | toyama7 | 2014-01-03 21:35:15 | 2014-01-03 21:58:03 |
| 2 | toyama7 | 2014-01-03 21:35:21 | 2014-01-03 21:58:06 |
| 3 | toyama7 | 2014-01-03 21:35:27 | 2014-01-03 21:58:08 |
| 10 | toyama7 | 2014-01-03 21:58:18 | 2014-01-03 21:58:18 |
+-----+-----------+---------------------+---------------------+
```
if duplicate id , update username and updated_at
## Configuration Example(bulk insert,fluentd key different column name)
```
type mysql_bulk
host localhost
database test_app_development
username root
password hogehoge
column_names id,user_name,created_at,updated_at
key_names id,user,created_date,updated_date
table users
flush_interval 10s
```
Assume following input is coming:
```js
mysql.input: {"user":"toyama","created_date":"2014/01/03 21:35:15","updated_date":"2014/01/03 21:35:15","dummy":"hogehoge"}
mysql.input: {"user":"toyama2","created_date":"2014/01/03 21:35:21","updated_date":"2014/01/03 21:35:21","dummy":"hogehoge"}
mysql.input: {"user":"toyama3","created_date":"2014/01/03 21:35:27","updated_date":"2014/01/03 21:35:27","dummy":"hogehoge"}
```
then result becomes as below (indented):
```sql
+-----+-----------+---------------------+---------------------+
| id | user_name | created_at | updated_at |
+-----+-----------+---------------------+---------------------+
| 1 | toyama | 2014-01-03 21:35:15 | 2014-01-03 21:35:15 |
| 2 | toyama2 | 2014-01-03 21:35:21 | 2014-01-03 21:35:21 |
| 3 | toyama3 | 2014-01-03 21:35:27 | 2014-01-03 21:35:27 |
+-----+-----------+---------------------+---------------------+
```
## spec
```
bundle install
rake test
```
## todo
divide bulk insert(exsample 1000 per)
## Contributing
1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new [Pull Request](../../pull/new/master)
## Copyright
Copyright (c) 2013 Hiroshi Toyama. See [LICENSE](LICENSE) for details.