{"id":18129364,"url":"https://github.com/kamatama41/embulk-input-remote","last_synced_at":"2025-10-20T11:53:44.479Z","repository":{"id":30363311,"uuid":"33915833","full_name":"kamatama41/embulk-input-remote","owner":"kamatama41","description":null,"archived":false,"fork":false,"pushed_at":"2019-03-31T07:53:28.000Z","size":380,"stargazers_count":3,"open_issues_count":0,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-10-11T09:36:37.815Z","etag":null,"topics":["embulk","kotlin"],"latest_commit_sha":null,"homepage":null,"language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kamatama41.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-04-14T06:57:06.000Z","updated_at":"2019-03-31T07:53:30.000Z","dependencies_parsed_at":"2022-08-17T22:00:59.979Z","dependency_job_id":null,"html_url":"https://github.com/kamatama41/embulk-input-remote","commit_stats":null,"previous_names":[],"tags_count":17,"template":false,"template_full_name":null,"purl":"pkg:github/kamatama41/embulk-input-remote","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamatama41%2Fembulk-input-remote","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamatama41%2Fembulk-input-remote/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamatama41%2Fembulk-input-remote/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamatama41%2Fembulk-input-remote/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kamatama41","download_url":"https://codeload.github.com/kamatama41/embulk-input-remote/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kamatama41%2Fembulk-input-remote/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279997170,"owners_count":26256546,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-20T02:00:06.978Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["embulk","kotlin"],"created_at":"2024-11-01T10:09:10.650Z","updated_at":"2025-10-20T11:53:44.461Z","avatar_url":"https://github.com/kamatama41.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![CircleCI](https://circleci.com/gh/kamatama41/embulk-input-remote.svg?style=svg)](https://circleci.com/gh/kamatama41/embulk-input-remote)\n\n# Remote file input plugin for [Embulk](https://github.com/embulk/embulk)\n\nThis plugin load data from Remote hosts by SCP\n\n## Overview\n\n* **Plugin type**: file input\n* **Resume supported**: yes\n* **Cleanup supported**: yes\n\n## Configuration\n\n- **hosts**: Target hosts, its format should be `hostname` or `hostname:port` (its `port` overrides `default_port`) (list, default: [])\n- **hosts_command**: Command to get `hosts` (Windows not supported). If given the option, `hosts` is overwritten. (string, default: null)\n- **hosts_separator**: Separator for `hosts_command` result (string, default: \" \")\n- **default_port**: Port number for SSH (integer, default: 22)\n- **path**: File or directory path of remote host (string, default: \"\")\n- **path_command**: Command to get `path` (Windows not supported). If given the option, `path` is overwritten. (string, default: null)\n- **ignore_not_found_hosts**: If true, hosts that meet the following conditions are skipped, which means they are not included into the the target of resuming.) (boolean, default: false)\n  - Target file (or directory) is not found\n  - An error that is related to SSH connectivity occurred \n- **auth**: SSH authentication setting (hash, default: {})\n    - **user**: SSH username (string, default: current user)\n    - **type**: `public_key` or `password` (string, default: public_key)\n    - **key_path**: Path of your secret key (required when you choose `public_key` type) (string, default: `~/.ssh/id_rsa` or `id_dsa`\")\n    - **password**: Password of the `user` (required when you choose `password` type) (string)\n    - **skip_host_key_verification**: If true, verification of host key will be skipped (boolean, default: false)\n    - **load_known_hosts**: If true, load the known_hosts file (boolean, default: true)\n\n## Example\n\n```yaml\nin:\n  type: remote\n  hosts:\n    - host1\n    - host2:10022\n  # hosts_command: echo 'host1,host2:10022'\n  # hosts_separator: ','\n  path: /some/path/20150414125923\n  # path_command: echo /some/path/`date \"+%Y%m%d%H%M%S\"`\n  ignore_not_found_hosts: true\n  auth:\n    user: a_user\n    type: public_key\n    key_path: /usr/home/.ssh/id_rsa\n    # type: password\n    # password: {password}\n```\n\n## Note\nWhen running Embulk with this plugin on Linux, it might be blocked due to SecureRandom. Please try one of followings to resolve it.\n(Note: it makes SecureRandom more insecure than default)\n\n### set JVM_OPTION \"-Djava.security.egd\"\n\n```bash\n$ export JAVA_TOOL_OPTIONS=\"-Djava.security.egd=file:/dev/./urandom\"\n$ embulk run config.yml\n```\n\n### Update \"securerandom.source\" in ${JAVA_HOME}/jre/lib/security/java.security\n```\n# securerandom.source=file:/dev/random  # before\nsecurerandom.source=file:/dev/./urandom # after\n```\n\n### see also\n\nhttp://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom\n\n## Development on local machine\nInstall Docker and Docker compose so that you can start SSH-able containers and run tests.\n\n```sh\n$ docker-compose up -d\n$ docker-compose ps\n          Name                 Command       State           Ports        \n--------------------------------------------------------------------------\nembulk-input-remote_host1   /entrypoint.sh   Up      0.0.0.0:10022-\u003e22/tcp\nembulk-input-remote_host2   /entrypoint.sh   Up      0.0.0.0:10023-\u003e22/tcp\n\n$ docker cp $(docker inspect --format=\"{{.Id}}\" embulk-input-remote_host1):/home/ubuntu/.ssh/id_rsa_test .\n$ chmod 400 id_rsa_test\n$ ./gradlew test\n```\n\n## Build\n\n```\n$ ./gradlew gem\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkamatama41%2Fembulk-input-remote","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkamatama41%2Fembulk-input-remote","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkamatama41%2Fembulk-input-remote/lists"}