{"id":20951358,"url":"https://github.com/lgug2z/elasdx","last_synced_at":"2025-05-14T03:33:01.112Z","repository":{"id":37960873,"uuid":"195637287","full_name":"LGUG2Z/elasdx","owner":"LGUG2Z","description":"An ElasticSearch index template updating, reindexing and cleanup tool","archived":false,"fork":false,"pushed_at":"2022-07-15T14:27:53.000Z","size":20,"stargazers_count":21,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-11-16T12:47:39.501Z","etag":null,"topics":["cli","elasticsearch","go","golang"],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LGUG2Z.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-07T10:17:57.000Z","updated_at":"2024-10-01T10:15:02.000Z","dependencies_parsed_at":"2022-09-02T11:30:34.002Z","dependency_job_id":null,"html_url":"https://github.com/LGUG2Z/elasdx","commit_stats":null,"previous_names":[],"tags_count":8,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LGUG2Z%2Felasdx","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LGUG2Z%2Felasdx/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LGUG2Z%2Felasdx/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LGUG2Z%2Felasdx/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LGUG2Z","download_url":"https://codeload.github.com/LGUG2Z/elasdx/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225275675,"owners_count":17448389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","elasticsearch","go","golang"],"created_at":"2024-11-19T00:58:31.040Z","updated_at":"2024-11-19T00:58:31.669Z","avatar_url":"https://github.com/LGUG2Z.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# elasdx\n`elasdx` is a command line tool to make ElasticSearch index template updates, reindexing and cleanup a more simple and\nCI-friendly process.\n\n## Installation\n### Go Get\n```bash\ngo get -u github.com/LGUG2Z/elasdx\ncd ${GOPATH}/src/github.com/LGUG2Z/elasdx\nmake install\n```\n\n### Binary\n```bash\nexport ELASDX_VERSION=x.x.x\ncurl -Lo elasdx.tar.gz https://github.com/LGUG2Z/elasdx/releases/download/v${ELASDX_VERSION}/elasdx_${ELASDX_VERSION}_linux_amd64.tar.gz\nmkdir elasdx \u0026\u0026 tar -xzf elasdx.tar.gz -C elasdx \u0026\u0026 mv elasdx/elasdx /usr/local/bin/elasdx \u0026\u0026 chmod +x /usr/local/bin/elasdx \u0026\u0026 rm -rf elasdx.tar.gz elasdx\n```\n\n### Docker\n```bash\nexport ELASDX_VERSION=x.x.x\ndocker pull lgug2z/elasdx:${ELASDX_VERSION}\ndocker pull lgug2z/elasdx:latest\n```\n\n### Homebrew\n```bash\nbrew tap LGUG2Z/tap\nbrew install LGUG2Z/tap/elasdx\n```\n\n## Background\n[ElasticSearch](https://www.elastic.co/products/elasticsearch) provides a comprehensive API to do just about anything,\nso it's unfortunate that until now I have always been stuck writing `bash` scripts full of `curl` commands to interact\nwith it. One of the most painful things to do with `bash` and `curl` when it comes to ElasticSearch is reindexing.\n\nTypically this process requires the following steps:\n\n* Make sure environment variables are set correctly\n* [Update a template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html):\n```bash\ncurl -X PUT \"localhost:9200/_template/template_1\" -H 'Content-Type: application/json' -d'\n{\n  \"index_patterns\": [\"te*\", \"bar*\"],\n  \"settings\": {\n    \"number_of_shards\": 1\n  },\n  \"mappings\": {\n    \"_source\": {\n      \"enabled\": false\n    },\n    \"properties\": {\n      \"host_name\": {\n        \"type\": \"keyword\"\n      },\n      \"created_at\": {\n        \"type\": \"date\",\n        \"format\": \"EEE MMM dd HH:mm:ss Z yyyy\"\n      }\n    }\n  }\n}\n'\n```\n\n* [Create a new index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html):\n```bash\ncurl -X PUT \"localhost:9200/twitter\"\n```\n\n* Update the settings on the index for\n[bulk reindexing](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html#bulk):\n```bash\ncurl -X PUT \"localhost:9200/twitter/_settings\" -H 'Content-Type: application/json' -d'\n{\n    \"index\" : {\n        \"refresh_interval\" : \"-1\"\n    }\n}\n'\n```\n\n* [Reindex](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) from the old index to the new index:\n```bash\ncurl -X POST \"localhost:9200/_reindex\" -H 'Content-Type: application/json' -d'\n{\n  \"source\": {\n    \"index\": \"twitter\"\n  },\n  \"dest\": {\n    \"index\": \"new_twitter\"\n  }\n}\n'\n```\n\n* Set the desired refresh interval once the reindexing is done:\n```bash\ncurl -X PUT \"localhost:9200/twitter/_settings\" -H 'Content-Type: application/json' -d'\n{\n    \"index\" : {\n        \"refresh_interval\" : \"1s\"\n    }\n}\n'\n```\n\n* [Associate an alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html) with the new index:\n```bash\ncurl -X POST \"localhost:9200/_aliases\" -H 'Content-Type: application/json' -d'\n{\n    \"actions\" : [\n        { \"add\" : { \"index\" : \"test1\", \"alias\" : \"alias1\" } }\n    ]\n}\n'\n```\n\n* Remove the old index from an alias:\n```bash\ncurl -X POST \"localhost:9200/_aliases\" -H 'Content-Type: application/json' -d'\n{\n    \"actions\" : [\n        { \"remove\" : { \"index\" : \"test1\", \"alias\" : \"alias1\" } }\n    ]\n}\n'\n```\n\n* [Clean up old indices](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html):\n```bash\ncurl -X DELETE \"localhost:9200/twitter\"\n```\n\n## Usage\n`elasdx` can operate either on a single index template `json` file or on a directory containing multiple index templates.\nThe ElasticSearch URL can be set using the `ELASDX_URL` environment variable or using the `--url` command line flag.\nThis value defaults to `http://127.0.0.1:9200`. If using basic auth, the username and password can be provided using the\n`ELASDX_USERNAME`, `--username` or `ELASDX_PASSWORD`, `--password` environment variables and flags respectively.\n\nOptionally, `elasdx` can make a connection to an instance or a cluster at a `https://` url without providing a valid\ncertificate by setting either `ELASDX_SKIP_VERIFY` or `--skip-verify`.\n\n### Reindex\nThe `reindex` command assumes either a file or a directory of files named to match the index template and the eventual\nalias desired.\n\nRunning `elasdx reindex twitter.json` would create or update an index template with the name `twitter`\nand create a new index and associate it with the alias `twitter`. Each new index will take the alias name and append a\ntimestamp: `twitter-2019-07-07-10:47:17`.\n\nIf provisioning a fresh instance of ElasticSearch, this command will skip trying to reindex from a matching existing\nindex and simply create a new index and associate it with the correct alias.\n\nIf reindexing, the `--bulk-indexing` flag can optionally be passed to optimise for bulk indexing by setting the refresh\nrate of the new index to `-1` before restoring it to either the default refresh rate once the reindexing is finished, or\nto the refresh rate defined in the index template `json` file.\n\n```text\n❯ elasdx reindex --bulk-indexing twitter.json\nINDEX TEMPLATE   Updated     twitter\nINDEX            Created     twitter-2019-07-07-10:54:40\nDOCUMENTS        Reindexed   100 from twitter-2019-07-07-10:54:33 to twitter-2019-07-07-10:54:40\nALIAS            Removed     twitter-2019-07-07-10:54:33 from twitter\nALIAS            Added       twitter-2019-07-07-10:54:40 to twitter\n```\n```json5\n// ❯ curl -X GET \"localhost:9200/twitter/_settings\" | jq\n{\n  \"twitter-2019-07-07-10:54:40\": {\n    \"settings\": {\n      \"index\": {\n        \"refresh_interval\": \"10s\", // Desired refresh interval from template is set\n        \"number_of_shards\": \"5\",\n        \"provided_name\": \"twitter-2019-07-07-10:54:40\",\n        \"creation_date\": \"1562492836239\",\n        \"number_of_replicas\": \"1\",\n        \"uuid\": \"d9EkiyzwTMGt5-vaChULcQ\",\n        \"version\": {\n          \"created\": \"6020299\"\n        }\n      }\n    }\n  }\n}\n```\n### Cleanup\nThe `cleanup` command is intended to work on the same file or directory structure used for the `reindex` command.\n\nThe maximum history of indices to keep is set using the `--max-history` flag, and defaults to `2`, meaning that the\ncurrent index associated with the alias and the previous index will be kept, and all other previous indices will be\ndeleted. If `--max-history` is set to `0`, all indices including the current index associated with the alias will be\ndeleted; this can be useful during development.\n\n```text\n❯ elasdx cleanup twitter.json\nINDEX            Deleted     twitter-2019-07-07-10:15:31\nINDEX            Deleted     twitter-2019-07-07-10:15:39\nINDEX            Deleted     twitter-2019-07-07-10:47:15\n```\n\n```text\n❯ elasdx cleanup --max-history 0 twitter.json\nINDEX            Deleted     twitter-2019-07-07-10:54:33\nINDEX            Deleted     twitter-2019-07-07-10:54:40\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgug2z%2Felasdx","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flgug2z%2Felasdx","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flgug2z%2Felasdx/lists"}