Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sonots/embulk-input-vertica
Vertica input plugins for Embulk loads records from vertica
https://github.com/sonots/embulk-input-vertica
Last synced: 2 months ago
JSON representation
Vertica input plugins for Embulk loads records from vertica
- Host: GitHub
- URL: https://github.com/sonots/embulk-input-vertica
- Owner: sonots
- License: other
- Created: 2016-02-16T09:57:41.000Z (almost 9 years ago)
- Default Branch: master
- Last Pushed: 2016-06-08T02:45:57.000Z (over 8 years ago)
- Last Synced: 2024-11-14T02:43:34.107Z (2 months ago)
- Language: Java
- Size: 1000 KB
- Stars: 1
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: COPYING
Awesome Lists containing this project
README
# Vertica input plugins for Embulk
Vertica input plugins for Embulk loads records from vertica.
## Overview
* **Plugin type**: input
* **Resume supported**: yes
## Configuration
- **host**: database host name (string, required)
- **port**: database port number (integer, 5433)
- **user**: database login user name (string, required)
- **password**: database login password (string, default: "")
- **database**: destination database name (string, required)
- **schema**: destination schema name (string, default: "public")
- **fetch_rows**: number of rows to fetch one time (used for java.sql.Statement#setFetchSize) (integer, default: 10000)
- **connect_timeout**: timeout for establishment of a database connection. (integer (seconds), default: 300)
- **socket_timeout**: timeout for socket read operations. 0 means no timeout. (integer (seconds), default: 1800)
- **resource_pool**: set session resource pool (string, default: null)
- **ssl**: ~~enables SSL. data will be encrypted but CA or certification will not be verified (boolean, default: false)~~ Not supported yet
- **options**: extra JDBC properties (hash, default: {})
- If you write SQL directly,
- **query**: SQL to run (string)
- If **query** is not set,
- **table**: destination table name (string, required)
- **select**: comma-separated list of columns to select (string, default: "*")
- **where**: WHERE condition to filter the rows (string, default: no-condition)
- **default_timezone**: If the sql type of a column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted int this default_timezone. You can overwrite timezone for each columns using column_options option. (string, default: `UTC`)
- **column_options**: advanced: a key-value pairs where key is a column name and value is options for the column.
- **value_type**: embulk get values from database as this value_type. Typically, the value_type determines `getXXX` method of `java.sql.PreparedStatement`.
(string, default: depends on the sql type of the column. Available values options are: `long`, `double`, `float`, `decimal`, `boolean`, `string`, `date`, `time`, `timestamp`)
- **type**: Column values are converted to this embulk type.
Available values options are: `boolean`, `long`, `double`, `string`, `timestamp`).
By default, the embulk type is determined according to the sql type of the column (or value_type if specified).
- **timestamp_format**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted by this timestamp_format. And if the embulk type is `timestamp`, this timestamp_format may be used in the output plugin. For example, stdout plugin use the timestamp_format, but *csv formatter plugin doesn't use*. (string, default : `%Y-%m-%d` for `date`, `%H:%M:%S` for `time`, `%Y-%m-%d %H:%M:%S` for `timestamp`)
- **timezone**: If the sql type of the column is `date`/`time`/`datetime` and the embulk type is `string`, column values are formatted in this timezone.
(string, value of default_timezone option is used by default)
## Example
```yaml
in:
type: vertica
host: localhost
user: myuser
password: ""
database: my_database
schema: sandbox
table: my_table
select: "col1, col2, col3"
where: "col4 != 'a'"
```
If you need a complex SQL,
```yaml
in:
type: vertica
host: localhost
user: myuser
password: ""
database: my_database
query: |
SELECT t1.id, t1.name, t2.id AS t2_id, t2.name AS t2_name
FROM schema1.table1 AS t1
LEFT JOIN table2 AS t2
ON t1.id = t2.t1_id
```
Advanced configuration:
```yaml
in:
type: vertica
host: localhost
user: myuser
password: ""
database: my_database
schema: sandbox
table: "my_table"
select: "col1, col2, col3"
where: "col4 != 'a'"
column_options:
col1: {type: long}
col3: {type: string, timestamp_format: "%Y/%m/%d", timezone: "+0900"}
```
## LICENSE
See [./COPYING](COPYING)
Note that vertica jdbc drivers are not licensed under the Apace License, 2.0. It follows [Vertica Community Edition HP Software License Agreement](https://my.vertica.com/software-license-agreement/)
## Development
Run example:
```
$ ./gradlew classpath
$ embulk run -I lib example/example.yml
```
Run test:
```
$ ./gradlew test
```
Run checkstyle:
```
$ ./gradlew check
```
Release gem:
```
$ ./gradlew gemPush
```