https://github.com/toyama0919/embulk-filter-mysql

https://github.com/toyama0919/embulk-filter-mysql

embulk embulk-plugin mysql prepared-statements

Last synced: 27 days ago
JSON representation

• Host: GitHub
• URL: https://github.com/toyama0919/embulk-filter-mysql
• Owner: toyama0919
• License: mit
• Created: 2016-04-25T10:06:32.000Z (over 8 years ago)
• Default Branch: master
• Last Pushed: 2017-03-31T07:57:38.000Z (over 7 years ago)
• Last Synced: 2024-10-07T00:19:58.888Z (about 1 month ago)
• Topics: embulk, embulk-plugin, mysql, prepared-statements
• Language: Ruby
• Size: 7.81 KB
• Stars: 4
• Watchers: 3
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.txt

Awesome Lists containing this project

README

        
# Mysql filter plugin for Embulk

[![Gem Version](https://badge.fury.io/rb/embulk-filter-mysql.svg)](http://badge.fury.io/rb/embulk-filter-mysql)

Mysql filter plugin for Embulk. Execute prepared statements query.

## Overview

* **Plugin type**: filter

## Configuration

- **host**: host(string, default: 'localhost')

- **user**: user(string, required)

- **password**: password(string, required)

- **database**: database(string, required)

- **port**: port(integer, default: 3306)

- **query**: query(string, required)

- **params**: params(array, required)

- **keep_input**: keep_input(bool, default: true)

## Example

```yaml

in:

  type: s3

  bucket: machine-learning-production

  path_prefix: customer-approval/batch-prediction/result/

  endpoint: s3.amazonaws.com

  auth_method: {{ env.EMBULK_S3_AUTH_METHOD }}

  decoders:

  - {type: gzip}

  parser:

    type: csv

    delimiter: ","

    skip_header_lines: 1

    allow_extra_columns: true

    allow_optional_columns: true

    columns:

      - {name: user_id, type: long}

      - {name: trueLabel, type: string}

      - {name: bestAnswer, type: string}

      - {name: score, type: double}

filters:

  - type: mysql

    host: {{ env.DATABASE_HOST | default: "localhost" }}

    user: {{ env.APPLICATION_USERNAME }}

    password: {{ env.APPLICATION_DATABASE_PASS }}

    database: {{ env.APPLICATION_DATABASE }}

    keep_input: true

    query: |

      select 

        id,

        last_name,

        first_name,

        company_name

      from

        user

      where id = ?

    params:

      - user_id

out:

  type: stdout

```

#### input CSV

```

user_id,trueLabel,bestAnswer,score

1,0,0,1.5

2,0,0,1.5

3,0,0,1.5

```

#### Running query

```

  select 

    id,

    last_name,

    first_name,

    company_name

  from

    user

  where id = 1;

  select 

    id,

    last_name,

    first_name,

    company_name

  from

    user

  where id = 2;

  select 

    id,

    last_name,

    first_name,

    company_name

  from

    user

  where id = 3;

```

## Build

```

$ rake

```