{"id":19651266,"url":"https://github.com/embulk/embulk-output-elasticsearch","last_synced_at":"2025-04-10T02:24:32.648Z","repository":{"id":26457863,"uuid":"29909148","full_name":"embulk/embulk-output-elasticsearch","owner":"embulk","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-26T09:13:21.000Z","size":536,"stargazers_count":32,"open_issues_count":7,"forks_count":26,"subscribers_count":14,"default_branch":"master","last_synced_at":"2025-04-02T23:13:47.587Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/embulk.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2015-01-27T10:27:49.000Z","updated_at":"2024-07-26T09:09:48.000Z","dependencies_parsed_at":"2024-12-20T21:12:47.862Z","dependency_job_id":"ff01837c-0b98-47ee-80ef-9f10e3f042ec","html_url":"https://github.com/embulk/embulk-output-elasticsearch","commit_stats":{"total_commits":187,"total_committers":15,"mean_commits":"12.466666666666667","dds":0.5294117647058824,"last_synced_commit":"65ad90c9f8b60988cfe1c783a790ed0bd900bcdd"},"previous_names":[],"tags_count":23,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/embulk%2Fembulk-output-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/embulk","download_url":"https://codeload.github.com/embulk/embulk-output-elasticsearch/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248143522,"owners_count":21054793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T15:05:49.579Z","updated_at":"2025-04-10T02:24:32.618Z","avatar_url":"https://github.com/embulk.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Elasticsearch output plugin for Embulk\n\n**Notice** This plugin doesn't positively support [Amazon(AWS) Elasticsearch Service](https://aws.amazon.com/elasticsearch-service/).\nActually, AWS Elasticsearch Service supported AWS VPC at Oct 2017 and user is able to access to Es from EC2 instances in VPC subnet without any authentication.\nYou can use this plugin for AWS ES at your own risk.\n\n- *[Amazon Elasticsearch Service Limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html)*\n\n## Overview\n\n* **Plugin type**: output\n* **Rollback supported**: no\n* **Resume supported**: no\n* **Cleanup supported**: no\n\n## Configuration\n\n- **mode**: \"insert\" or \"replace\". See below(string, optional, default is insert)\n- **nodes**: list of nodes. nodes are pairs of host and port (list, required)\n  - NOTE: This plugin uses HTTP/REST Clients and uses TCP:9200 as a default. TCP:9300 is usually used for Transport Client.\n- **use_ssl** Use SSL encryption (boolean, default is false)\n- **auth_method** (string, default is 'none') 'none'/'basic'. See also [Authentication](#authentication).\n- **user** Username for basic authentication (string, default is null)\n- **password** Password for above user (string, default is null)\n- ~~**cluster_name**: name of the cluster (string, default is \"elasticsearch\")~~ Not used now. May use in the future\n- **index**: index name (string, required)\n- **index_type**: index type (string, required)\n- **id**: document id column (string, default is null)\n- **bulk_actions**: Sets when to flush a new bulk request based on the number of actions currently added. (int, default is 1000)\n- **bulk_size**: Sets when to flush a new bulk request based on the size of actions currently added. (long, default is 5242880)\n- **fill_null_for_empty_column**: Fill null value when column value is empty (boolean, optional, default is false)\n- ~~**concurrent_requests**: concurrent_requests (int, default is 5)~~  Not used now. May use in the future\n- **maximum_retries** Number of maximam retry times (int, optional, default is 7)\n- **initial_retry_interval_millis** Initial interval between retries in milliseconds (int, optional, default is 1000)\n- **maximum_retry_interval_millis** Maximum interval between retries in milliseconds (int, optional, default is 120000)\n- **timeout_millis** timeout in milliseconds for each HTTP request(int, optional, default is 60000)\n- **connect_timeout_millis** connection timeout in milliseconds for HTTP client(int, optional, default is 60000)\n- **max_snapshot_waiting_secs** maximam waiting time in second when snapshot is just creating before delete index. works when `mode: replace` (int, optional, default is 1800)\n### Modes\n\n#### insert:\n\ndefault.\nThis mode writes data to existing index.\n\n#### replace:\n\n1. Create new temporary index\n2. Insert data into the new index\n3. replace the alias with the new index. If alias doesn't exists, plugin will create new alias.\n4. Delete existing (old) index if exists\n\nIndex should not exists with the same name as the alias\n\n```yaml\nout:\n  type: elasticsearch\n  mode: replace\n  nodes:\n  - {host: localhost, port: 9200}\n  index: \u003calias name\u003e # plugin generates index name like \u003cindex\u003e_%Y%m%d-%H%M%S\n  index_type: \u003cindex type\u003e\n```\n\n### Authentication\n\nThis plugin supports Basic authentication and works with [Elastic Cloud](https://cloud.elastic.co/) and 'Security'(formally Sield).\n'Security' also supports LDAP and Active Directory. This plugin doesn't supports these auth methods.\n\n```yaml\nuse_ssl: true\nauth_method: basic\nuser: \u003cusername\u003e\npassword: \u003cpassword\u003e\n```\n\n## Example\n\n```yaml\nout:\n  type: elasticsearch\n  mode: insert\n  nodes:\n  - {host: localhost, port: 9200}\n  index: \u003cindex name\u003e\n  index_type: \u003cindex type\u003e\n```\n\n## Test\n\nFirstly install Docker and Docker compose then `docker-compose up -d`,\nso that an ES server will be locally launched then you can run tests with `./gradlew test`.\n\n```sh\n$ docker-compose up -d\nCreating network \"embulk-output-elasticsearch_default\" with the default driver\nCreating embulk-output-elasticsearch_server ... done\n\n$ docker-compose ps\n               Name                             Command               State                        Ports\n------------------------------------------------------------------------------------------------------------------------------\nembulk-output-elasticsearch_server   /docker-entrypoint.sh elas ...   Up      0.0.0.0:19200-\u003e9200/tcp, 0.0.0.0:19300-\u003e9300/tcp\n\n$ ./gradlew test  # -t to watch change of files and rebuild continuously\n```\n\nFor Maintainers\n----------------\n\n### Release\n\nModify `version` in `build.gradle` at a detached commit, and then tag the commit with an annotation.\n\n```\ngit checkout --detach master\n(Edit: Remove \"-SNAPSHOT\" in \"version\" in build.gradle.)\ngit add build.gradle\ngit commit -m \"Release vX.Y.Z\"\ngit tag -a vX.Y.Z\n(Edit: Write a tag annotation in the changelog format.)\n```\n\nSee [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) for the changelog format. We adopt a part of it for Git's tag annotation like below.\n\n```\n## [X.Y.Z] - YYYY-MM-DD\n### Added\n- Added a feature.\n### Changed\n- Changed something.\n### Fixed\n- Fixed a bug.\n```\n\nPush the annotated tag, then. It triggers a release operation on GitHub Actions after approval.\n\n```\ngit push -u origin vX.Y.Z\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-output-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fembulk%2Fembulk-output-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fembulk%2Fembulk-output-elasticsearch/lists"}