Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/embulk/embulk-output-ftp
FTP file output plugin for Embulk
https://github.com/embulk/embulk-output-ftp
embulk embulk-output-plugin ftp
Last synced: about 2 months ago
JSON representation
FTP file output plugin for Embulk
- Host: GitHub
- URL: https://github.com/embulk/embulk-output-ftp
- Owner: embulk
- Created: 2016-07-20T06:21:29.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2021-07-07T09:31:15.000Z (over 3 years ago)
- Last Synced: 2024-08-10T11:07:28.199Z (5 months ago)
- Topics: embulk, embulk-output-plugin, ftp
- Language: Java
- Homepage:
- Size: 213 KB
- Stars: 0
- Watchers: 8
- Forks: 3
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# FTP file output plugin for Embulk
[](https://travis-ci.org/embulk/embulk-output-ftp)
This plugin support **FTP**, **FTPES(FTPS explicit)**, **FTPS(FTPS implicit)** and doesn't support **SFTP**.
If you want to use SFTP, please use [embulk-output-sftp](https://github.com/civitaspo/embulk-output-sftp).
## Overview
* **Plugin type**: file input
* **Resume supported**: no
* **Cleanup supported**: yes
## Configuration
- **host**: FTP server address (string, required)
- **port**: FTP server port number (integer, default: `21`. `990` if `ssl` is true and `ssl_explicit` is false)
- **user**: user name to login (string, optional)
- **password**: password to login (string, default: `""`)
- **path_prefix** prefix of target files (string, required)
- **sequence_format** Format for sequence part of output files (string, default: `".%03d.%02d"`)
- **file_ext** e.g. "csv.gz, json.gz" (string, required)
- **passive_mode**: use passive mode (boolean, default: true)
- **ascii_mode**: use ASCII mode instead of binary mode (boolean, default: false)
- **ssl**: use FTPS (SSL encryption). (boolean, default: false)
- **ssl_explicit** use FTPS(explicit) instead of FTPS(implicit). (boolean, default:true)
- **ssl_verify**: verify the certification provided by the server. By default, connection fails if the server certification is not signed by one the CAs in JVM's default trusted CA list. (boolean, default: true)
- **ssl_verify_hostname**: verify server's hostname matches with provided certificate. (boolean, default: true)
- **ssl_trusted_ca_cert_file**: if the server certification is not signed by a certificate authority, set path to the X.508 certification file (pem file) of a private CA (string, optional)
- **ssl_trusted_ca_cert_data**: similar to `ssl_trusted_ca_cert_file` but embed the contents of the PEM file as a string value instead of path to a local file (string, optional)
### FTP / FTPS default port number
FTP and FTPS server listens following port number(TCP) as default.
Please be sure to configure firewall rules.
| | FTP | FTPS(explicit) = FTPES | FTPS(implicit) = FTPS |
|:------------------------|----:|-----------------------:|----------------------:|
| Control channel port | 21 | 21 | 990 (\*1) |
| Data channel port (\*2) | 20 | 20 | 989 |
1. If you're using both of FTPS(implicit) and FTP, server also use 21/TCP for FTP.
2. If you're using passive mode, data channel port can be taken between 1024 and 65535.
## Example
Simple FTP:
```yaml
out:
type: ftp
host: ftp.example.net
port: 21
user: anonymous
path_prefix: /ftp/file/path/prefix
file_ext: csv
formatter:
type: csv.gz
header_line: false
encoders:
- {type: gzip}
```
FTPS encryption without server certificate verification:
```yaml
out:
type: ftp
host: ftp.example.net
port: 21
user: anonymous
password: "mypassword"
ssl: true
ssl_verify: false
path_prefix: /ftp/file/path/prefix
file_ext: csv
```
FTPS encryption with server certificate verification:
```yaml
out:
type: ftp
host: ftp.example.net
port: 21
user: anonymous
password: "mypassword"
ssl: true
ssl_verify: true
ssl_verify_hostname: false # to disable server hostname verification (optional)
# if the server use self-signed certificate, or set path to the pem file (optional)
ssl_trusted_ca_cert_file: /path/to/ca_cert.pem
# or embed contents of the pem file here (optional)
ssl_trusted_ca_cert_data: |
-----BEGIN CERTIFICATE-----
MIIFV...
...
...
-----END CERTIFICATE-----
path_prefix: /ftp/file/path/prefix
file_ext: csv
```
## Build
```
$ ./gradlew gem # -t to watch change of files and rebuild continuously
```
## Test
Firstly install Docker and Docker compose then `docker-compose up -d`,
so that an FTP server will be locally launched then you can run tests with `./gradlew test`.
```sh
$ docker-compose up -d
Creating network "embulk-output-ftp_default" with the default driver
Creating embulk-output-ftp_server ... done
Creating embulk-output-ftps_server ... done
$ docker-compose ps
Name Command State Ports
---------------------------------------------------------------------------------------------------
embulk-output-ftp_server /usr/sbin/run-vsftpd.sh Up 20/tcp, 0.0.0.0:11021->21/tcp, 0.0.0.0:65000->65000/tcp, 0.0.0.0:65001->65001/tcp, 0.0.0.0:65002->65002/tcp, 0.0.0.0:65003->65003/tcp,
0.0.0.0:65004->65004/tcp
embulk-output-ftps_server /usr/sbin/run-vsftpd-with- ... Up 20/tcp, 0.0.0.0:990->21/tcp, 0.0.0.0:65005->65005/tcp, 0.0.0.0:65006->65006/tcp, 0.0.0.0:65007->65007/tcp, 0.0.0.0:65008->65008/tcp
$ ./gradlew test # -t to watch change of files and rebuild continuously
```
If you want to use other FTP server to test, configure the following environment variables.
```
FTP_TEST_HOST (default: localhost)
FTP_TEST_PORT (default: 11021)
FTP_TEST_SSL_PORT (default:990)
FTP_TEST_USER (default: scott)
FTP_TEST_PASSWORD (default: tigger)
FTP_TEST_SSL_TRUSTED_CA_CERT_FILE
FTP_TEST_SSL_TRUSTED_CA_CERT_DATA
```
If you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.
```xml
$ vi ~/Library/LaunchAgents/environment.plist
Label
my.startup
ProgramArguments
sh
-c
launchctl setenv FTP_TEST_HOST ftp.example.com
launchctl setenv FTP_TEST_USER username
launchctl setenv FTP_TEST_PASSWORD password
launchctl setenv FTP_TEST_SSL_TRUSTED_CA_CERT_FILE /path/to/cert.pem
launchctl setenv FTP_TEST_SSL_TRUSTED_CA_CERT_DATA "-----BEGIN CERTIFICATE-----
ABCDEFG...
EFGHIJKL...
-----END CERTIFICATE-----"
RunAtLoad
$ launchctl load ~/Library/LaunchAgents/environment.plist
$ launchctl getenv FTP_TEST_HOST //try to get value.
Then start your applications.
```
## Acknowledgement
This program is forked from [embulk-input-ftp](https://github.com/embulk/embulk-input-ftp) and originally written by @frsyuki, modified by @sakama.