{"id":15204016,"url":"https://github.com/gjbae1212/go-esworker","last_synced_at":"2025-07-12T08:03:14.890Z","repository":{"id":57490097,"uuid":"199614101","full_name":"gjbae1212/go-esworker","owner":"gjbae1212","description":"Async Worker that bulk insert, update, delete to ElasticSearch for Golang. 🚀","archived":false,"fork":false,"pushed_at":"2020-10-28T07:10:09.000Z","size":89,"stargazers_count":9,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-09T03:51:11.136Z","etag":null,"topics":["async","bulk","client","elastic","elasticsearch","go","golang","worker"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gjbae1212.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-30T08:55:51.000Z","updated_at":"2023-08-09T20:48:00.000Z","dependencies_parsed_at":"2022-09-02T12:01:17.902Z","dependency_job_id":null,"html_url":"https://github.com/gjbae1212/go-esworker","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/gjbae1212/go-esworker","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbae1212%2Fgo-esworker","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbae1212%2Fgo-esworker/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbae1212%2Fgo-esworker/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbae1212%2Fgo-esworker/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gjbae1212","download_url":"https://codeload.github.com/gjbae1212/go-esworker/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gjbae1212%2Fgo-esworker/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264958581,"owners_count":23689033,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","bulk","client","elastic","elasticsearch","go","golang","worker"],"created_at":"2024-09-28T05:05:27.938Z","updated_at":"2025-07-12T08:03:14.843Z","avatar_url":"https://github.com/gjbae1212.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# go-esworker\n`go-esworker` is an async worker that documents can bulk insert, update, delete to the elasticsearch using Golang.\nIt is support to an infrastructure on AWS, GCP, Elastic Cloud, and so on.\n\u003cp align=\"left\"\u003e \n   \u003ca href=\"https://hits.seeyoufarm.com\"/\u003e\u003cimg src=\"https://hits.seeyoufarm.com/api/count/incr/badge.svg?url=https%3A%2F%2Fgithub.com%2Fgjbae1212%2Fgo-esworker\"/\u003e\u003c/a\u003e\n   \u003ca href=\"https://circleci.com/gh/gjbae1212/go-esworker\"\u003e\u003cimg src=\"https://circleci.com/gh/gjbae1212/go-esworker.svg?style=svg\"\u003e\u003c/a\u003e   \n   \u003ca href=\"https://goreportcard.com/report/github.com/gjbae1212/go-esworker\"\u003e\u003cimg src=\"https://goreportcard.com/badge/github.com/gjbae1212/go-esworker\"/\u003e\u003c/a\u003e         \n   \u003ca href=\"https://godoc.org/github.com/gjbae1212/go-esworker\"\u003e\u003cimg src=\"https://godoc.org/github.com/gjbae1212/go-esworker?status.svg\"/\u003e\u003c/a\u003e   \n   \u003ca href=\"/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-MIT-GREEN.svg\" alt=\"license\"/\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n## Installation\n```bash\ngo get -u github.com/gjbae1212/go-esworker\n```\n\n\n## Usage\n```go\nimport (\n\t\"context\"\n\t\"log\"\n\n\t\"github.com/gjbae1212/go-esworker\"\n)\n\nfunc main() {\n\n\t// Create dispatcher\n\tdispatcher, err := esworker.NewDispatcher(\n    \t\tesworker.WithESVersionOption(esworker.V6),\n    \t\tesworker.WithAddressesOption([]string{\"http://localhost:9200\"}),\n    \t\tesworker.WithUsernameOption(\"user\"),\n    \t\tesworker.WithPasswordOption(\"password\"),\n    \t\tesworker.WithErrorHandler(func(err error) {\n    \t\t\tlog.Println(err)\n    \t\t}),\n    \t)\n    \tif err != nil {\n    \t\tlog.Panic(err)\n    \t}\n\n\t// Start dispatcher\n\tif err := dispatcher.Start(); err != nil {\n\t\tlog.Panic(err)\n\t}\n\n\t// Process operations in bulk.\n\tctx := context.Background()\n\t// create doc\n\tdispatcher.AddAction(ctx, \u0026esworker.StandardAction{\n\t\top:    esworker.ES_CREATE,\n\t\tindex: \"allan\",\n\t\tid:    \"1\",\n\t\tdoc:   map[string]interface{}{\"field1\": 10},\n\t})\n\n\t// update doc\n\tdispatcher.AddAction(ctx, \u0026esworker.StandardAction{\n\t\top:    esworker.ES_UPDATE,\n\t\tindex: \"allan\",\n\t\tid:    \"1\",\n\t\tdoc:   map[string]interface{}{\"field1\": 20},\n\t})\n\n\t// delete doc\n\tdispatcher.AddAction(ctx, \u0026esworker.StandardAction{\n\t\top:    esworker.ES_DELETE,\n\t\tindex: \"allan\",\n\t\tid:    \"1\",\n\t})\n}\n\n```\n\n\n## Dispatcher Parameters\nIt should pass parameters for dependency injection when you are creating a `go-esworker` dispatcher.  \nA list to support the parameters below.  \n\n| method name | description | value | state |\n|-------------|-------------|-------|-------|\n| **WithESVersionOption** | ElasticSearch Version | esworker.V5, esworker.V6, esworker.V7 | default `V6` |\n| **WithAddressesOption** | ElasticSearch Address | | default `http://localhost:9200` |\n| **WithUsernameOption** | ElasticSearch Username for HTTP basic authentication| | optional |\n| **WithPasswordOption** | ElasticSearch Password for HTTP basic authentication | | optional |\n| **WithCloudIdOption**  | ID for Elastic Cloud | | optional |\n| **WithApiKeyOption**  | Base64-Encoded value for authorization(api-key) | | optional(if set, overrides username and password) |\n| **WithTransportOption** | Http transport | | default `http default transport` |\n| **WithLoggerOption** | Logger | | optional |\n| **WithGlobalQueueSizeOption** | Global queue max size | | default `5000` |\n| **WithWorkerSizeOption** | Worker size | | default `5` |\n| **WithWorkerQueueSizeOption** | Worker max queue size | | default `5` |\n| **WithWorkerWaitInterval** | Deal with data in worker queue after every interval time | | default `2 * time.Second` |\n| **WithErrorHandler** | A function that deals with an error when an error is raised | | optional |  \n\n\n## Action Interface\nTo deal with operation as insert and update and delete to, you would use to the `StandardAction` struct or a struct which is implementing `esworker.Action` interface.\n```go\n// generate and start dispatcher \ndispatcher, _ := esworker.NewDispatcher()\ndispatcher.Start()\n\n// Ex) Standard Action Example\nact := \u0026esworker.StandardAction{\n\tOp: ES_CREATE\n\tIndex: \"sample\",\n\tDocType: \"_doc\",\n\tId: \"test-id\",\n\tDoc: map[string]interface{}{\"field\": 1},\n}\ndispatcher.AddAction(context.Background(), act)\n\n\n// Ex) Custom Action Example\nsampleAction struct {}\n\nfunc (act *sampleAction) GetOperation() esworker.ESOperation {\n\t// return esworker.ES_CREATE\n\t// return esworker.ES_INDEX\n\t// return esworker.ES_UPDATE\n\t// return esworker.ES_DELETE    \n}\n\nfunc (act *sampleAction) GetIndex() string {\n\t// return \"your index name\"\n}\n\nfunc (act *sampleAction) GetDocType() string {\n\t//return \"\"\n\t//return \"doc type\"\t\n}\n\nfunc (act *sampleAction) GetID() string {\n\t//return \"\"\n\t//return \"doc id\"\t\t\n}\n\nfunc (act *sampleAction) GetDoc() map[string]interface{} {\n\t//return map[string]interface{}{}\n}\ndispatcher.AddAction(context.Background(), \u0026sampleAction{})\n\n``` \nIf you will make to a custom struct which is implementing `esworker.Action` interface, it must implement 5 methods.  \n\n| name          | description |\n|---------------|----------------------------------------------|\n| **GetOperation**  |  ES_CREATE, ES_INDEX, ES_UPDATE, ES_DELETE |\n| **GetIndex**      |  index name |\n| **GetDocType**    |  doc type (if it is returned an empty string, default `_doc` or `doc`) |\n| **GetID**         |  doc id (if an operation is ES_INDEX, possible `empty string`) |\n| **GetDoc**        |  doc data |\n\n\n## Elastic Cloud\nIf you use to infrastructure on Elastic Cloud, you could access to ElasticSearch without endpoint and basic authentication.\n[(**How to use API-KEY)**](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html)\n```go\ndispatcher, err := esworker.NewDispatcher(\n\t\tesworker.WithESVersionOption(esworker.V7),\n\t\tesworker.WithCloudIdOption(\"your-cloud-id\"),\n\t\tesworker.WithApiKeyOption(\"api-key\"),\n)\ndispatcher.Start()\n```\n \n### LICENSE\nThis project is following The MIT.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgjbae1212%2Fgo-esworker","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgjbae1212%2Fgo-esworker","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgjbae1212%2Fgo-esworker/lists"}