{"id":18142238,"url":"https://github.com/trocco-io/embulk-input-elasticsearch","last_synced_at":"2025-04-22T19:05:17.350Z","repository":{"id":59153334,"uuid":"60320917","full_name":"trocco-io/embulk-input-elasticsearch","owner":"trocco-io","description":"Elasticsearch input plugin for Embulk. parallel query support.","archived":false,"fork":false,"pushed_at":"2019-10-28T13:30:51.000Z","size":30,"stargazers_count":6,"open_issues_count":2,"forks_count":4,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-05T02:35:13.469Z","etag":null,"topics":["elasticsearch","embulk","embulk-input-plugin","jruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trocco-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-03T05:34:58.000Z","updated_at":"2024-07-08T03:30:45.000Z","dependencies_parsed_at":"2022-09-13T11:01:29.148Z","dependency_job_id":null,"html_url":"https://github.com/trocco-io/embulk-input-elasticsearch","commit_stats":null,"previous_names":["toyama0919/embulk-input-elasticsearch"],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trocco-io%2Fembulk-input-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trocco-io%2Fembulk-input-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trocco-io%2Fembulk-input-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trocco-io%2Fembulk-input-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trocco-io","download_url":"https://codeload.github.com/trocco-io/embulk-input-elasticsearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250306619,"owners_count":21408925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["elasticsearch","embulk","embulk-input-plugin","jruby"],"created_at":"2024-11-01T18:06:17.359Z","updated_at":"2025-04-22T19:05:17.291Z","avatar_url":"https://github.com/trocco-io.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Elasticsearch input plugin for Embulk [![Build Status](https://secure.travis-ci.org/toyama0919/embulk-input-elasticsearch.png?branch=master)](http://travis-ci.org/toyama0919/embulk-input-elasticsearch) [![Gem Version](https://badge.fury.io/rb/embulk-input-elasticsearch.svg)](http://badge.fury.io/rb/embulk-input-elasticsearch)\n\n## Overview\n\n* **Plugin type**: input\n* **Resume supported**: yes\n* **Cleanup supported**: yes\n* **Guess supported**: no\n\n## Configuration\n- **nodes**: nodes (array, required)\n  - **host**: host (string, required)\n  - **port**: port (integer, required)\n- **queries**: lucene query array. (array, required)\n- **index**: index (string, required)\n- **index_type**: index_type (string)\n- **request_timeout**: request timeout (integer)\n- **per_size**: per size query. (integer, required, default: `1000`)\n- **limit_size**: limit size unit query. (integer, default: unlimit)\n- **num_threads**: number of threads for queries. (integer, default: 1)\n- **retry_on_failure**: retry on failure. set 0 is retry forever. (integer, default: 5)\n- **sort**: sort order. (hash, default: nil)\n- **scroll**: scroll. to keep the search context. (string, default: '1m')\n- **fields**: fields (array, required)\n  - **name**: name (string, required)\n  - **type**: type (string, required)\n  - **metadata**: metadata (boolean, default: false)\n  - **time_format**: time_format (string)\n\n## Example\n\n```yaml\nin:\n  type: elasticsearch\n  nodes:\n    - {host: localhost, port: 9200}\n  queries:\n    - 'page_type: HP'\n    - 'page_type: GP'\n  index: crawl\n  index_type: m_corporation_page\n  request_timeout: 60\n  per_size: 1000\n  limit_size: 200000\n  num_threads: 2\n  sort:\n    m_corporation_id: desc\n    employee_range: asc\n  fields:\n    - { name: _id, type: string, metadata: true }\n    - { name: _type, type: string, metadata: true }\n    - { name: _index, type: string, metadata: true }\n    - { name: _score, type: double, metadata: true }\n    - { name: page_type, type: string }\n    - { name: corp_name, type: string }\n    - { name: corp_key, type: string }\n    - { name: title, type: string }\n    - { name: body, type: string }\n    - { name: url, type: string }\n    - { name: employee_range, type: long }\n    - { name: m_corporation_id, type: long }\n    - { name: cg_lv1, type: json }\n    - { name: cg_lv2, type: json }\n    - { name: cg_lv3, type: json }\n```\n\n## Support Type\n* string\n* long\n* double\n* timestamp\n* json\n* boolean\n\n## test\n\n### setup\n\n```\ncurl -o embulk.jar --create-dirs -L \"http://dl.embulk.org/embulk-latest.jar\"\nchmod +x embulk.jar\n./embulk.jar gem install bundler\n./embulk.jar bundle install --path vendor/bundle\n```\n\n### run test\n\n```\n./embulk.jar bundle exec rake test\n```\n\n## Build\n\n```\n$ rake\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrocco-io%2Fembulk-input-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrocco-io%2Fembulk-input-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrocco-io%2Fembulk-input-elasticsearch/lists"}