{"id":19651308,"url":"https://github.com/embulk/embulk-input-ftp","last_synced_at":"2025-04-28T16:31:23.722Z","repository":{"id":31215380,"uuid":"34776445","full_name":"embulk/embulk-input-ftp","owner":"embulk","description":"Embulk FTP input plugin","archived":false,"fork":false,"pushed_at":"2022-07-07T01:40:35.000Z","size":346,"stargazers_count":5,"open_issues_count":4,"forks_count":3,"subscribers_count":10,"default_branch":"master","last_synced_at":"2025-04-16T05:33:00.480Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/embulk.png","metadata":{"files":{"readme":"README.md","changelog":"ChangeLog","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-29T06:32:55.000Z","updated_at":"2022-07-05T07:58:46.000Z","dependencies_parsed_at":"2022-09-11T20:52:06.847Z","dependency_job_id":null,"html_url":"https://github.com/embulk/embulk-input-ftp","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-input-ftp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-input-ftp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-input-ftp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-input-ftp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/embulk","download_url":"https://codeload.github.com/embulk/embulk-input-ftp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251345919,"owners_count":21574806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T15:06:00.672Z","updated_at":"2025-04-28T16:31:23.169Z","avatar_url":"https://github.com/embulk.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FTP file input plugin for Embulk\n\n## Overview\n\nThis plugin support **FTP**, **FTPES(FTPS explicit)**, **FTPS(FTPS implicit)** and doesn't support **SFTP**.\n\nIf you want to use SFTP, please use [embulk-input-sftp](https://github.com/embulk/embulk-input-sftp).\n\nembulk-input-ftp v0.2.0+ requires Embulk v0.9.12+\n\n* **Plugin type**: file input\n* **Resume supported**: yes\n* **Cleanup supported**: yes\n\n## Configuration\n\n- **host**: FTP server address (string, required)\n- **port**: FTP server port number (integer, default: `21`. `990` if `ssl` is true and `ssl_explicit` is false)\n- **user**: user name to login (string, optional)\n- **password**: password to login (string, default: `\"\"`)\n- **path_prefix** prefix of target files (string, required)\n- **path_match_pattern**: regexp to match file paths. If a file path doesn't match with this pattern, the file will be skipped (regexp string, optional)\n- **incremental** enables incremental loading(boolean, optional. default: true. If incremental loading is enabled, config diff for the next execution will include last_path parameter so that next execution skips files before the path. Otherwise, last_path will not be included.\n- **passive_mode**: use passive mode (boolean, default: true)\n- **ascii_mode**: use ASCII mode instead of binary mode (boolean, default: false)\n- **ssl**: use FTPS (SSL encryption). (boolean, default: false)\n- **ssl_explicit** use FTPS(explicit) instead of FTPS(implicit). (boolean, default:true)\n- **ssl_verify**: verify the certification provided by the server. By default, connection fails if the server certification is not signed by one the CAs in JVM's default trusted CA list. (boolean, default: true)\n- **ssl_verify_hostname**: verify server's hostname matches with provided certificate. (boolean, default: true)\n- **ssl_trusted_ca_cert_file**: if the server certification is not signed by a certificate authority, set path to the X.508 certification file (pem file) of a private CA (string, optional)\n- **ssl_trusted_ca_cert_data**: similar to `ssl_trusted_ca_cert_file` but embed the contents of the PEM file as a string value instead of path to a local file (string, optional)\n\n### FTP / FTPS default port number\n\nFTP and FTPS server usually listens following port number(TCP) as default.\n\nPlease be sure to configure firewall rules.\n\n|                         | FTP | FTPS(explicit) = FTPES | FTPS(implicit) = FTPS |\n|:------------------------|----:|-----------------------:|----------------------:|\n| Control channel port    |  21 |                     21 |             990 (\\*1) |\n| Data channel port (\\*2) |  20 |                     20 |                   989 |\n\n1. If you're using both of FTPS(implicit) and FTP, server may also listen 21/TCP for unecnrypted FTP.\n2. If you're using `passive mode`, data channel port can be taken between 1024 and 65535.\n\n## Example\n\nSimple FTP:\n\n```yaml\nin:\n  type: ftp\n  host: ftp.example.net\n  port: 21\n  user: anonymous\n  path_prefix: /ftp/file/path/prefix\n```\n\nFTPS encryption without server certificate verification:\n\n```yaml\nin:\n  type: ftp\n  host: ftp.example.net\n  port: 21\n  user: anonymous\n  password: \"mypassword\"\n  path_prefix: /ftp/file/path/prefix\n\n  ssl: true\n  ssl_verify: false\n```\n\nFTPS encryption with server certificate verification:\n\n```yaml\nin:\n  type: ftp\n  host: ftp.example.net\n  port: 21\n  user: anonymous\n  password: \"mypassword\"\n  path_prefix: /ftp/file/path/prefix\n\n  ssl: true\n  ssl_verify: true\n\n  ssl_verify_hostname: false   # to disable server hostname verification (optional)\n\n  # if the server use self-signed certificate, or set path to the pem file (optional)\n  ssl_trusted_ca_cert_file: /path/to/ca_cert.pem\n\n  # or embed contents of the pem file here (optional)\n  ssl_trusted_ca_cert_data: |\n      -----BEGIN CERTIFICATE-----\n      MIIFV...\n      ...\n      ...\n      -----END CERTIFICATE-----\n```\n\n## Build\n\n```\n$ ./gradlew embulk-input-ftp:gem\n```\n\n## Release\n\n```\n$ ./gradlew clean embulk-util-ftp:publishMavenPublicationToMavenCentralRepository embulk-input-ftp:gemPush\n```\n\n## Test\n\nFirstly install Docker and Docker compose then `docker-compose up -d`,\nso that an FTP server will be locally launched then you can run tests with `./gradlew test`.\n\n```sh\n$ docker-compose up -d\nCreating network \"embulk-input-ftp_default\" with the default driver\nCreating embulk-input-ftp_server ... done\n\n$ docker-compose ps\n         Name              Command     State                       Ports                           \n---------------------------------------------------------------------------------------------------\nembulk-input-ftp_server   /start-ftp   Up      0.0.0.0:11021-\u003e21/tcp, 0.0.0.0:65000-\u003e65000/tcp, ...\n\n$ ./gradlew test  # -t to watch change of files and rebuild continuously\n```\n\nIf you want to use other FTP server to test, configure the following environment variables.\n\n```\nFTP_TEST_HOST (default: localhost)\nFTP_TEST_PORT (default: 11021)\nFTP_TEST_SSL_PORT (default:990)\nFTP_TEST_USER (default: scott)\nFTP_TEST_PASSWORD (default: tiger)\nFTP_TEST_SSL_TRUSTED_CA_CERT_FILE\nFTP_TEST_SSL_TRUSTED_CA_CERT_DATA\n```\n\nIf you're using Mac OS X El Capitan and GUI Applications(IDE), like as follows.\n```xml\n\n\u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003c!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\"\u003e\n\u003cplist version=\"1.0\"\u003e\n\u003cdict\u003e\n  \u003ckey\u003eLabel\u003c/key\u003e\n  \u003cstring\u003emy.startup\u003c/string\u003e\n  \u003ckey\u003eProgramArguments\u003c/key\u003e\n  \u003carray\u003e\n    \u003cstring\u003esh\u003c/string\u003e\n    \u003cstring\u003e-c\u003c/string\u003e\n    \u003cstring\u003e\n      launchctl setenv FTP_TEST_HOST ftp.example.com\n      launchctl setenv FTP_TEST_USER username\n      launchctl setenv FTP_TEST_PASSWORD password\n      launchctl setenv FTP_TEST_SSL_TRUSTED_CA_CERT_FILE /path/to/cert.pem\n      launchctl setenv FTP_TEST_SSL_TRUSTED_CA_CERT_DATA \"-----BEGIN CERTIFICATE-----\n      ABCDEFG...\n      EFGHIJKL...\n      -----END CERTIFICATE-----\"\n    \u003c/string\u003e\n  \u003c/array\u003e\n  \u003ckey\u003eRunAtLoad\u003c/key\u003e\n  \u003ctrue/\u003e\n\u003c/dict\u003e\n\u003c/plist\u003e\n\n$ launchctl load ~/Library/LaunchAgents/environment.plist\n$ launchctl getenv FTP_TEST_HOST //try to get value.\n\nThen start your applications.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-input-ftp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fembulk%2Fembulk-input-ftp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-input-ftp/lists"}