{"id":13509091,"url":"https://github.com/sashman/elasticsearch_elixir_bulk_processor","last_synced_at":"2025-03-21T21:31:20.786Z","repository":{"id":56165430,"uuid":"212535951","full_name":"sashman/elasticsearch_elixir_bulk_processor","owner":"sashman","description":"Elasticsearch Elixir Bulk Processor is a configurable manager for efficiently inserting data into Elasticsearch.   This processor uses GenStages (data-exchange steps) for handling backpressure, and various settings to control the bulk payloads being uploaded to Elasticsearch.","archived":false,"fork":false,"pushed_at":"2021-05-06T13:00:15.000Z","size":212,"stargazers_count":14,"open_issues_count":2,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T14:42:22.353Z","etag":null,"topics":["bulk","elasticsearch","elixir","genstage","hacktoberfest","processor"],"latest_commit_sha":null,"homepage":"","language":"Elixir","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sashman.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-10-03T08:55:46.000Z","updated_at":"2024-04-21T15:50:32.000Z","dependencies_parsed_at":"2022-08-15T14:00:17.632Z","dependency_job_id":null,"html_url":"https://github.com/sashman/elasticsearch_elixir_bulk_processor","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sashman%2Felasticsearch_elixir_bulk_processor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sashman%2Felasticsearch_elixir_bulk_processor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sashman%2Felasticsearch_elixir_bulk_processor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sashman%2Felasticsearch_elixir_bulk_processor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sashman","download_url":"https://codeload.github.com/sashman/elasticsearch_elixir_bulk_processor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244164431,"owners_count":20408934,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bulk","elasticsearch","elixir","genstage","hacktoberfest","processor"],"created_at":"2024-08-01T02:01:02.823Z","updated_at":"2025-03-21T21:31:20.239Z","avatar_url":"https://github.com/sashman.png","language":"Elixir","readme":"# ElasticsearchElixirBulkProcessor\n\n[![Hex.pm](https://img.shields.io/hexpm/v/elasticsearch_elixir_bulk_processor)](https://hex.pm/packages/elasticsearch_elixir_bulk_processor)\n[![CircleCI](https://circleci.com/gh/sashman/elasticsearch_elixir_bulk_processor.svg?style=svg)](https://circleci.com/gh/sashman/elasticsearch_elixir_bulk_processor)\n[![Coverage Status](https://coveralls.io/repos/github/sashman/elasticsearch_elixir_bulk_processor/badge.svg?branch=master)](https://coveralls.io/github/sashman/elasticsearch_elixir_bulk_processor?branch=master)\n![Hex.pm](https://img.shields.io/hexpm/l/elasticsearch_elixir_bulk_processor)\n\nElasticsearch Elixir Bulk Processor is a configurable manager for efficiently inserting data into Elasticsearch.\nThis processor uses [GenStages](https://hexdocs.pm/gen_stage/GenStage.html) for handling backpressure, and various settings to control the bulk payloads being uploaded to Elasticsearch.\n\nInspired by the [Java Bulk Processor](https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html). Uses [elasticsearch-elixir](https://github.com/danielberkompas/elasticsearch-elixir) as the client. Featured on the [Elastic Community Contributed Clients page](https://www.elastic.co/guide/en/elasticsearch/client/community/current/index.html#erlang).\n\n## Installation\n\nIf [available in Hex](https://hex.pm/packages/elasticsearch_elixir_bulk_processor), the package can be installed\nby adding `elasticsearch_elixir_bulk_processor` to your list of dependencies in `mix.exs`:\n\n```elixir\ndef deps do\n  [\n    {:elasticsearch_elixir_bulk_processor, \"~\u003e 0.1\"}\n  ]\nend\n```\n\n## Sending data\n\n```elixir\nElasticsearchElixirBulkProcessor.send_requests(list_of_items)\n```\n\nTo send a list of request items to Elasticsearch. This mechanism uses GenStages for back pressure.\nNOTE: It should be completely reasonable to use this function by passing single element lists, the mechanism aggregates the items together prior to sending them.\n\nIf you wish to bypass the GenStage mechanism and send the data synchronously you can use:\n\n```elixir\nElasticsearchElixirBulkProcessor.Bulk.DirectUpload.add_requests(list_of_items)\n```\n\nThe list elements must be [structs](https://github.com/sashman/elasticsearch_elixir_bulk_processor/tree/master/lib/elasticsearch_elixir_bulk_processor/items):\n\n- `ElasticsearchElixirBulkProcessor.Items.Index`\n- `ElasticsearchElixirBulkProcessor.Items.Create`\n- `ElasticsearchElixirBulkProcessor.Items.Update`\n- `ElasticsearchElixirBulkProcessor.Items.Delete`\n\n#### Examples\n\n```elixir\n    iex\u003e alias ElasticsearchElixirBulkProcessor.Items.Index\n    ...\u003e [\n    ...\u003e  %Index{index: \"test_index\", source: %{\"field\" =\u003e \"value1\"}},\n    ...\u003e  %Index{index: \"test_index\", source: %{\"field\" =\u003e \"value2\"}},\n    ...\u003e  %Index{index: \"test_index\", source: %{\"field\" =\u003e \"value3\"}}\n    ...\u003e ]\n    ...\u003e |\u003e ElasticsearchElixirBulkProcessor.send_requests()\n    :ok\n```\n\n## Configuration\n\n### Elasticsearch endpoint\n\nCan be configurate via the `ELASTICSEARCH_URL` environment variable, defaults to: `\"http://localhost:9200\"`. \n\n```elixir\nconfig :elasticsearch_elixir_bulk_processor,\n       ElasticsearchElixirBulkProcessor.ElasticsearchCluster,\n       url: {:system, \"ELASTICSEARCH_URL\"},\n       api: Elasticsearch.API.HTTP\n```\n\n\nAlternatively:\n\n```elixir\nconfig :elasticsearch_elixir_bulk_processor,\n       ElasticsearchElixirBulkProcessor.ElasticsearchCluster,\n       url: \"http://localhost:9200\",\n       api: Elasticsearch.API.HTTP\n```\n\nSee the [client configuration](https://github.com/danielberkompas/elasticsearch-elixir#configuration) for more.\n\n### Action count\n\nNumber of actions/items to send per bulk (can be changed at run time), deault is `nil` (unlimited):\n\n```elixir\nElasticsearchElixirBulkProcessor.set_event_count_threshold(100)\n```\n\n### Byte size\n\nMax number of bytes to send per bulk (can be changed at run time), default is `62_914_560` (60mb):\n\n```elixir\nElasticsearchElixirBulkProcessor.set_byte_threshold(100)\n```\n\n### Action order\n\nPreservation of order of actions/items\n\n```elixir\nconfig :elasticsearch_elixir_bulk_processor, preserve_event_order: false\n```\n\n### Retries\n\nRetry policy, this uses the [ElixirRetry](https://github.com/safwank/ElixirRetry) DSL. See [`ElasticsearchElixirBulkProcessor.Bulk.Retry.policy`](https://github.com/sashman/elasticsearch_elixir_bulk_processor/blob/0d015282315c016db07334824c7b98c858d43658/lib/elasticsearch_elixir_bulk_processor/bulk/retry.ex#L29).\n\n```elixir\nconfig :elasticsearch_elixir_bulk_processor, retry_function: \u0026MyApp.Retry.policy/0\n```\n\nDefault:\n\n```elixir\ndef default do\n  constant_backoff(100) |\u003e Stream.take(5)\nend\n```\n\n### Success and error handlers\n\nThe callbacks on a successful upload or in case of failed items or failed request can bet set through the config.\nOn success, the handler is called with the Elasticsearch bulk request. On failure, the hanlder is called with`%{data: any, error: any}`, `data` being the original payload and `error` being the response or HTTP error.\nSee [`ElasticsearchElixirBulkProcessor.Bulk.Handlers`](https://github.com/sashman/elasticsearch_elixir_bulk_processor/blob/master/lib/elasticsearch_elixir_bulk_processor/bulk/handlers.ex).\n\n```elixir\nconfig :elasticsearch_elixir_bulk_processor,\n  success_function: \u0026MyApp.success_handler/1,\n  error_function: \u0026MyApp.error_handler/1\n```\n\nIt's highly recommended you add an error handler to make sure your data is uploaded succesfully, for example you can use the logger:\n\n```elixir\n  require Logger\n  ...\n\n  def error_handler(%{data: _, error: {_, error}}) do\n    error\n    |\u003e inspect\n    |\u003e Logger.error()\n  end\n```\n\nDocumentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)\nand published on [HexDocs](https://hexdocs.pm). Once published, the docs can\nbe found at [https://hexdocs.pm/elasticsearch_elixir_bulk_processor](https://hexdocs.pm/elasticsearch_elixir_bulk_processor).\n\n## Testing script\n\nThe testing script is used to compare insertion using direct upload vs using a GenStage based approach. Run Elasticsearch set up with:\n\n```bash\ndocker-compose up\n```\n\nRun test:\n\n```bash\nmix insert_test \u003cINSERT_COUNT\u003e \u003cBULK_SIZE\u003e staged|direct\n```\n\n- `staged` - uses a GenStage mechanism to aggregate and insert.\n- `direct` - iterates and inserts bulk sequentially as given.\n","funding_links":[],"categories":["Search"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsashman%2Felasticsearch_elixir_bulk_processor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsashman%2Felasticsearch_elixir_bulk_processor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsashman%2Felasticsearch_elixir_bulk_processor/lists"}