https://github.com/embulk/embulk-output-sftp
Store files on remote server using SFTP
https://github.com/embulk/embulk-output-sftp
embulk embulk-output-plugin embulk-plugin sftp
Last synced: 23 days ago
JSON representation
Store files on remote server using SFTP
- Host: GitHub
- URL: https://github.com/embulk/embulk-output-sftp
- Owner: embulk
- License: apache-2.0
- Created: 2015-10-01T17:06:40.000Z (over 9 years ago)
- Default Branch: master
- Last Pushed: 2024-02-09T12:36:34.000Z (over 1 year ago)
- Last Synced: 2025-04-05T09:51:05.016Z (about 2 months ago)
- Topics: embulk, embulk-output-plugin, embulk-plugin, sftp
- Language: Java
- Homepage:
- Size: 468 KB
- Stars: 5
- Watchers: 14
- Forks: 5
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE
Awesome Lists containing this project
README
# SFTP file output plugin for Embulk
[](https://travis-ci.org/embulk/embulk-output-sftp)
Stores files on a SFTP Server
## Overview
* **Plugin type**: file output
* **Load all or nothing**: no
* **Resume supported**: no
* **Cleanup supported**: no
## Configuration
- **host**: (string, required)
- **port**: (int, default: `22`)
- **user**: (string, required)
- **password**: (string, default: `null`)
- **secret_key_file**: (string, default: `null`) [see below](#secret-keyfile-configuration)
- **secret_key_passphrase**: (string, default: `""`)
- **user_directory_is_root**: (boolean, default: `true`)
- **timeout**: sftp connection timeout seconds (integer, default: `600`)
- **path_prefix**: Prefix of output paths (string, required)
- **file_ext**: Extension of output files (string, required)
- **sequence_format**: Format for sequence part of output files (string, default: `".%03d.%02d"`)
- **rename_file_after_upload**: Upload `file_ext` + ".tmp" first, then rename it after upload finish (boolean, default: `false`)
- **local_buffering**: Use local temp file to buffer records. If `false`, plugin will buffer records to remote file directly, with ".tmp" as filename suffix (boolean, default: `true`)
- **temp_file_threshold**: Maximum file size of local temp file, plugin will flush (append) to remote file when local temp file reaches threshold (long, default: `5368709120`, ie. 5GiB, min: 50MiB, max: 10GiB)
### Proxy configuration
- **proxy**:
- **type**: (string(http | socks | stream), required, default: `null`)
- **http**: use HTTP Proxy
- **socks**: use SOCKS Proxy
- **stream**: Connects to the SFTP server through a remote host reached by SSH
- **host**: (string, required)
- **port**: (int, default: `22`)
- **user**: (string, optional)
- **password**: (string, optional, default: `null`)
- **command**: (string, optional)
## Example
```yaml
out:
type: sftp
host: 127.0.0.1
port: 22
user: civitaspo
secret_key_file: /Users/civitaspo/.ssh/id_rsa
secret_key_passphrase: secret_pass
user_directory_is_root: false
timeout: 600
path_prefix: /data/sftp
file_ext: _20151020.tsv
sequence_format: ".%01d%01d"
temp_file_threshold: 10737418240 # 10GiB
```
With proxy
```yaml
out:
type: sftp
host: 127.0.0.1
port: 22
user: embulk
secret_key_file: /Users/embulk/.ssh/id_rsa
secret_key_passphrase: secret_pass
user_directory_is_root: false
timeout: 600
path_prefix: /data/sftp
proxy:
type: http
host: proxy_host
port: 8080
user: proxy_user
password: proxy_secret_pass
command:
```
### Secret Keyfile configuration
Please set path of secret_key_file as follows.
```yaml
out:
type: sftp
...
secret_key_file: /path/to/id_rsa
...
```
You can also embed contents of secret_key_file at config.yml.
```yaml
out:
type: sftp
...
secret_key_file:
content |
-----BEGIN RSA PRIVATE KEY-----
ABCDEFG...
HIJKLMN...
OPQRSTU...
-----END RSA PRIVATE KEY-----
...
```
## Run Example
```
$ ./gradlew classpath
```
Use `vagrant` to start a remote sshd server:
```
$ vagrant up
```
Run:
```
$ embulk run -Ilib example/sample.yml
```
## Build
```
$ ./gradlew gem # -t to watch change of files and rebuild continuously
$ ./gradlew bintrayUpload # release embulk-output-sftp to Bintray maven repo
```
## Note
This plugin uses "org.apache.commons:commons-vfs" and the library uses the logger "org.apache.commons.logging.Log". So, this plugin suppress the logger's message except when embulk log level is debug.
## Contributors
- Satoshi Akama (@sakama)
- Rudolph Miller (@Rudolph-Miller)
- Naotoshi Seo (@sonots)