{"id":19651286,"url":"https://github.com/embulk/embulk-output-sftp","last_synced_at":"2025-04-28T16:31:22.315Z","repository":{"id":1772235,"uuid":"43509296","full_name":"embulk/embulk-output-sftp","owner":"embulk","description":"Store files on remote server using SFTP","archived":false,"fork":false,"pushed_at":"2024-02-09T12:36:34.000Z","size":479,"stargazers_count":5,"open_issues_count":6,"forks_count":5,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-05T09:51:05.016Z","etag":null,"topics":["embulk","embulk-output-plugin","embulk-plugin","sftp"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/embulk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2015-10-01T17:06:40.000Z","updated_at":"2022-05-30T07:58:25.000Z","dependencies_parsed_at":"2024-02-07T07:27:45.675Z","dependency_job_id":"4d4383d4-3e13-4485-bc13-3bca98a03995","html_url":"https://github.com/embulk/embulk-output-sftp","commit_stats":{"total_commits":185,"total_committers":7,"mean_commits":"26.428571428571427","dds":0.6054054054054054,"last_synced_commit":"6cf442dd45d9fa50c1edbff194dce35209487c0f"},"previous_names":[],"tags_count":27,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-sftp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-sftp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-sftp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-sftp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/embulk","download_url":"https://codeload.github.com/embulk/embulk-output-sftp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251345912,"owners_count":21574806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embulk","embulk-output-plugin","embulk-plugin","sftp"],"created_at":"2024-11-11T15:05:57.762Z","updated_at":"2025-04-28T16:31:21.714Z","avatar_url":"https://github.com/embulk.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SFTP file output plugin for Embulk\n[![Build Status](https://travis-ci.org/embulk/embulk-output-sftp.svg)](https://travis-ci.org/embulk/embulk-output-sftp)\n\nStores files on a SFTP Server\n\n## Overview\n\n* **Plugin type**: file output\n* **Load all or nothing**: no\n* **Resume supported**: no\n* **Cleanup supported**: no\n\n## Configuration\n\n- **host**: (string, required)\n- **port**: (int, default: `22`)\n- **user**: (string, required)\n- **password**: (string, default: `null`)\n- **secret_key_file**: (string, default: `null`) [see below](#secret-keyfile-configuration)\n- **secret_key_passphrase**: (string, default: `\"\"`)\n- **user_directory_is_root**: (boolean, default: `true`)\n- **timeout**: sftp connection timeout seconds (integer, default: `600`)\n- **path_prefix**: Prefix of output paths (string, required)\n- **file_ext**: Extension of output files (string, required)\n- **sequence_format**: Format for sequence part of output files (string, default: `\".%03d.%02d\"`)\n- **rename_file_after_upload**: Upload `file_ext` + \".tmp\" first, then rename it after upload finish (boolean, default: `false`)\n- **local_buffering**: Use local temp file to buffer records. If `false`, plugin will buffer records to remote file directly, with \".tmp\" as filename suffix (boolean, default: `true`)\n- **temp_file_threshold**: Maximum file size of local temp file, plugin will flush (append) to remote file when local temp file reaches threshold (long, default: `5368709120`, ie. 5GiB, min: 50MiB, max: 10GiB)\n\n### Proxy configuration\n\n- **proxy**:\n    - **type**: (string(http | socks | stream), required, default: `null`)\n        - **http**: use HTTP Proxy\n        - **socks**: use SOCKS Proxy\n        - **stream**: Connects to the SFTP server through a remote host reached by SSH\n    - **host**: (string, required)\n    - **port**: (int, default: `22`)\n    - **user**: (string, optional)\n    - **password**: (string, optional, default: `null`)\n    - **command**: (string, optional)\n    \n## Example\n\n```yaml\nout:\n  type: sftp\n  host: 127.0.0.1\n  port: 22\n  user: civitaspo\n  secret_key_file: /Users/civitaspo/.ssh/id_rsa\n  secret_key_passphrase: secret_pass\n  user_directory_is_root: false\n  timeout: 600\n  path_prefix: /data/sftp\n  file_ext: _20151020.tsv\n  sequence_format: \".%01d%01d\"\n  temp_file_threshold: 10737418240 # 10GiB\n```\n\nWith proxy\n```yaml\nout:\n  type: sftp\n  host: 127.0.0.1\n  port: 22\n  user: embulk\n  secret_key_file: /Users/embulk/.ssh/id_rsa\n  secret_key_passphrase: secret_pass\n  user_directory_is_root: false\n  timeout: 600\n  path_prefix: /data/sftp\n  proxy:\n    type: http\n    host: proxy_host\n    port: 8080\n    user: proxy_user\n    password: proxy_secret_pass\n    command:\n```\n\n### Secret Keyfile configuration\n\nPlease set path of secret_key_file as follows.\n```yaml\nout:\n  type: sftp\n  ...\n  secret_key_file: /path/to/id_rsa\n  ...\n```\n\nYou can also embed contents of secret_key_file at config.yml.\n```yaml\nout:\n  type: sftp\n  ...\n  secret_key_file:\n    content |\n      -----BEGIN RSA PRIVATE KEY-----\n      ABCDEFG...\n      HIJKLMN...\n      OPQRSTU...\n      -----END RSA PRIVATE KEY-----\n  ...\n```\n\n## Run Example\n\n```\n$ ./gradlew classpath\n```\n\nUse `vagrant` to start a remote sshd server:\n\n```\n$ vagrant up\n```\n\nRun:\n\n\n```\n$ embulk run -Ilib example/sample.yml\n```\n\n## Build\n\n```\n$ ./gradlew gem  # -t to watch change of files and rebuild continuously\n$ ./gradlew bintrayUpload # release embulk-output-sftp to Bintray maven repo\n```\n\n## Note\n\nThis plugin uses \"org.apache.commons:commons-vfs\" and the library uses the logger \"org.apache.commons.logging.Log\". So, this plugin suppress the logger's message except when embulk log level is debug.\n\n## Contributors\n- Satoshi Akama (@sakama)\n- Rudolph Miller (@Rudolph-Miller)\n- Naotoshi Seo (@sonots)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-output-sftp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fembulk%2Fembulk-output-sftp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-output-sftp/lists"}