{"id":16179293,"url":"https://github.com/jonashackt/elasticsearch-cheatsheet","last_synced_at":"2025-03-19T01:30:58.238Z","repository":{"id":147269614,"uuid":"69020833","full_name":"jonashackt/elasticsearch-cheatsheet","owner":"jonashackt","description":"Some helpful CURLs for interacting directly with es","archived":false,"fork":false,"pushed_at":"2017-12-16T11:05:33.000Z","size":10,"stargazers_count":6,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-02-28T13:47:04.251Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jonashackt.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-09-23T11:59:26.000Z","updated_at":"2018-05-03T12:41:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"89b90e12-cb00-427b-8877-1e9a293da96d","html_url":"https://github.com/jonashackt/elasticsearch-cheatsheet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Felasticsearch-cheatsheet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Felasticsearch-cheatsheet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Felasticsearch-cheatsheet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jonashackt%2Felasticsearch-cheatsheet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jonashackt","download_url":"https://codeload.github.com/jonashackt/elasticsearch-cheatsheet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243958083,"owners_count":20374792,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-10T05:26:39.273Z","updated_at":"2025-03-19T01:30:57.975Z","avatar_url":"https://github.com/jonashackt.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# elasticsearch-cheatsheet\n\n## How to delete logstash incides, if your disk space is low\n\n[This is a good answer on so](https://stackoverflow.com/questions/33430055/removing-old-indices-in-elasticsearch), but i´ll give you a complete guide here:\n\nIf you´re on elasticsearch 2.x like me, the [4.x docs of curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/4.3/index.html) are a good choice :)\n\n#### Install the corresponding elasticsearch-curator version:\n\n* install 5.x, if you have elasticsearch 5.x via `sudo pip install -Iv elasticsearch-curator`\n* install 4.3.1, if you have elasticsearch 2.x via `sudo pip install -Iv elasticsearch-curator==4.3.1`\n* install 3.5.1, if you have elasticsearch 1.x via `sudo pip install -Iv elasticsearch-curator==3.5.1`\n\n\u003e If curator isn´t working (e.g. if you have a already installed but corrupt installation of curator, remove the package with a `sudo rm -rf /usr/local/lib/python2.7/dist-packages/curator`, https://stackoverflow.com/a/14572899/4964553)\n\n#### Create a curator-configfile.yml (or download from here)\n\nYou can copy the contents from here - you only need to change the `unit_count: 14` to the quantity of days you don´t want to delete.\n\n[curator-configfile.yml](https://github.com/jonashackt/elasticsearch-cheatsheet/blob/master/curator-configfile.yml):\n\n```\n---\n# Remember, leave a key empty if there is no value.  None will be a string,\n# not a Python \"NoneType\"\nclient:\n  hosts:\n    - 127.0.0.1\n  port: 9200\n  url_prefix:\n  use_ssl: False\n  ssl_no_validate: False\n  timeout: 30\n  master_only: False\n\nlogging:\n  loglevel: INFO\n  logfile:\n  logformat: default\n```\n\n#### Create a curator-actionfile.yml (or download from here)\n\n[curator-actionfile.yml](https://github.com/jonashackt/elasticsearch-cheatsheet/blob/master/curator-actionfile.yml):\n\n```\n---\n# Remember, leave a key empty if there is no value.  None will be a string,\n# not a Python \"NoneType\"\n#\n# Also remember that all examples have 'disable_action' set to True.  If you\n# want to use this action as a template, be sure to set this to False after\n# copying it.\nactions:\n  1:\n    action: delete_indices\n    description: \u003e-\n      Delete indices older than 45 days (based on index name), for logstash-\n      prefixed indices. Ignore the error if the filter does not result in an\n      actionable list of indices (ignore_empty_list) and exit cleanly.\n    options:\n      ignore_empty_list: True\n      timeout_override:\n      continue_if_exception: False\n    filters:\n    - filtertype: pattern\n      kind: prefix\n      value: logstash-\n      exclude:\n    - filtertype: age\n      source: name\n      direction: older\n      timestring: '%Y.%m.%d'\n      unit: days\n      unit_count: 14\n      exclude:\n```\n\n#### Copy both files to your linux box that runs elaticsearch\n\ne.g. to folder `/home/userName/.curator`\n\n#### Run curator\n\nStart with a dry-run:\n\n`curator --dry-run --config curator-configfile.yml curator-actionfile.yml`\n\nIf that looks good, delete your Indices with:\n\n`curator --config curator-configfile.yml curator-actionfile.yml`\n\n#### Optional: Setup regularly schedule to run the deletion\n\nPut delete-logstash-indices bash script into `/etc/cron.daily` and you´re done with that issue!\n\n\n## Some helpful CURLs for interacting directly with elasticsearch (mostly 2.x tested)\n\n#### show indices settings\n```\ncurl -XGET 'http://localhost:9200/_all/_settings?pretty=true'\n```\n\n#### get elasticsearch version:\n```\ncurl -XGET 'localhost:9200'\n```\n\n#### show cluster health status:\n```\ncurl -XGET 'http://localhost:9200/_cluster/health?pretty=true'\n```\n\n#### show nodes:\n```\ncurl -XGET 'http://localhost:9200/_cat/nodes?pretty=true'\n```\n\n#### show shards:\n```\ncurl -XGET http://localhost:9200/_cat/shards\n```\n\n#### show indices:\n```\ncurl -XGET 'http://localhost:9200/_cat/indices?v'\n```\n\n#### allocate UNASSIGNED shards\n\nOne specific:\n```\ncurl -XPOST 'localhost:9200/_cluster/reroute' -d '{\n        \"commands\" : [ {\n              \"allocate\" : {\n                  \"index\" : \"logstash-2017.12.09\",\n                  \"shard\" : 0,\n                  \"node\" : \"Human Torch II\",\n                  \"allow_primary\" : true\n              }\n            }\n        ]\n    }'\n```\n\nAll of them:\n```\ncurl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print 1,2}' | while read var_index var_shard; do\n    curl -XPOST 'localhost:9200/_cluster/reroute' -d '{\n        \"commands\" : [ {\n              \"allocate\" : {\n                  \"index\" : \"$var_index\",\n                  \"shard\" : $var_shard,\n                  \"node\" : \"Human Torch II\",\n                  \"allow_primary\" : true\n              }\n            }\n        ]\n    }'\n    sleep 5;\ndone\n```\n\n\n\n#### deactivate shard allocation:\n```\ncurl -XPUT http://localhost:9200/_cluster/settings -d '{\n  \"persistent\": {\n    \"cluster.routing.allocation.enable\": \"none\"\n  }\n}'\n```\n\n#### activate shard allocation:\n```\ncurl -XPUT http://localhost:9200/_cluster/settings -d '{\n  \"persistent\": {\n    \"cluster.routing.allocation.enable\": \"all\"\n  }\n}'\n```\n\n#### reallocate an index\n```\ncurl -XPOST 'localhost:9200/_cluster/reroute' -d '{\n        \"commands\" : [ {\n              \"allocate\" : {\n                  \"index\" : \"logstash-2016.09.21\", \n                  \"shard\" : 5, \n                  \"node\" : \"Dormammu\", \n                  \"allow_primary\" : true\n              }\n            }\n        ]\n    }'\n```\n\n####  reallocate all unallocated indices\n```\nfor shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do\n    curl -XPOST 'localhost:9200/_cluster/reroute' -d '{\n        \"commands\" : [ {\n              \"allocate\" : {\n                  \"index\" : \"logstash-2016.06.02\", \n                  \"shard\" : $shard, \n                  \"node\" : \"Assassin\", \n                  \"allow_primary\" : true\n              }\n            }\n        ]\n    }'\n    sleep 5\ndone\n```\n\n\n#### flush for rolling cluster restart\n```\ncurl -XPOST \"http://localhost:9200/elasticsearch/_flush/synced\"\n```\n\n#### set all indices number of replicas to 0 (if you only have on node!)\n```\ncurl -XPUT localhost:9200/_settings -d '{\n    \"index\" : {\n        \"number_of_replicas\" : 0\n    }\n}'\n```\n\n#### create an index, e.g. when kibana says: \"unable to fetch mapping\"\n```\ncurl -XPUT 'http://localhost:9200/logstash-2016.09.24/' -d '{\n    \"settings\" : {\n        \"index\" : {\n            \"number_of_shards\" : 3, \n            \"number_of_replicas\" : 0 \n        }\n    }\n}'\n```\n\n#### get logstash template from es\n```\n curl -XGET localhost:9200/_template/logstash?pretty=true\n```\n\n#### activate elasticsearch logging\n\n```\ncurl -XPUT 'http://localhost:9200/_cluster/settings/' -d '{\n    \"transient\" : {\n        \"logger.discovery\" : \"DEBUG\"\n    }\n}'\n```\n\n\n\n\n\n## Upgrade elasticsearch from 1.x to 2.x\n\n\nOnly start the upgrade-process, if status is green! check with:\n```\ncurl -XGET 'http://localhost:9200/_cluster/health?pretty=true'\n```\n\nWhen not green, first make sure it get´s there! Then start.\n\n\n#### 1. deactivate shard allocation:\n```\ncurl -XPUT http://localhost:9200/_cluster/settings -d '{\n  \"persistent\": {\n    \"cluster.routing.allocation.enable\": \"none\"\n  }\n}'\n```\n\n#### 2. flush for rolling cluster restart\n```\ncurl -XPOST \"http://localhost:9200/elasticsearch/_flush/synced\"\n```\n\n#### 3. stop elasticsearch\n```\nsudo service elasticsearch stop\n```\n\n#### 4. install new elasticsearch version\n```\ndeb http://packages.elasticsearch.org/elasticsearch/{{ elk_elasticsearch.version }}/debian stable main\napt-get install elasticsearch\n```\n\n#### 5. wait for yellow\n```\ncurl -XGET 'http://localhost:9200/_cluster/health?pretty=true'\n```\n\n#### 6. activate shard allocation again:\n```\ncurl -XPUT http://localhost:9200/_cluster/settings -d '{\n  \"persistent\": {\n    \"cluster.routing.allocation.enable\": \"all\"\n  }\n}'\n```\n\n## and don´t forget logstash upgrade from 1.x to 2.x\n\nhttps://www.elastic.co/guide/en/logstash/current/_upgrading_logstash_and_elasticsearch_to_2_0.html\n\n#### 1. change logstash elasticsearch template\n\nBecause of a known issue (http://stackoverflow.com/questions/32761038/elk-unable-to-fetch-mapping-do-you-have-indices-matching-the-pattern, https://discuss.elastic.co/t/elasticseach-2-geoip-problem/33424/4), you have to manually change the logstash elasticsearch template. Therefor look into your logstash template with:\n\n```\ncurl -XGET localhost:9200/_template/logstash?pretty=true\n```\n\nif it has a line  __\"path\" : \"full\",__ in it like:\n```\n{\n    \"order\" : 0,\n    \"template\" : \"logstash-*\",\n    \"settings\" : {\n      \"index\" : {\n        \"refresh_interval\" : \"5s\"\n      }\n    },\n    \"mappings\" : {\n      \"_default_\" : {\n        \"dynamic_templates\" : [ {\n          \"string_fields\" : {\n            \"mapping\" : {\n              \"index\" : \"analyzed\",\n              \"omit_norms\" : true,\n              \"type\" : \"string\",\n              \"fields\" : {\n                \"raw\" : {\n                  \"ignore_above\" : 256,\n                  \"index\" : \"not_analyzed\",\n                  \"type\" : \"string\"\n                }\n              }\n            },\n            \"match_mapping_type\" : \"string\",\n            \"match\" : \"*\"\n          }\n        } ],\n        \"_all\" : {\n          \"enabled\" : true\n        },\n        \"properties\" : {\n          \"geoip\" : {\n            \"path\" : \"full\",\n            \"dynamic\" : true,\n            \"type\" : \"object\",\n            \"properties\" : {\n              \"location\" : {\n                \"type\" : \"geo_point\"\n              }\n            }\n          },\n          \"@version\" : {\n            \"index\" : \"not_analyzed\",\n            \"type\" : \"string\"\n          }\n        }\n      }\n    },\n    \"aliases\" : { }\n  }\n```\n\nthen we have to delete this line with:\n\n```\ncurl -XPUT http://localhost:9200/_template/logstash -d '{\n    \"order\" : 0,\n    \"template\" : \"logstash-*\",\n    \"settings\" : {\n      \"index\" : {\n        \"refresh_interval\" : \"5s\"\n      }\n    },\n    \"mappings\" : {\n      \"_default_\" : {\n        \"dynamic_templates\" : [ {\n          \"string_fields\" : {\n            \"mapping\" : {\n              \"index\" : \"analyzed\",\n              \"omit_norms\" : true,\n              \"type\" : \"string\",\n              \"fields\" : {\n                \"raw\" : {\n                  \"ignore_above\" : 256,\n                  \"index\" : \"not_analyzed\",\n                  \"type\" : \"string\"\n                }\n              }\n            },\n            \"match_mapping_type\" : \"string\",\n            \"match\" : \"*\"\n          }\n        } ],\n        \"_all\" : {\n          \"enabled\" : true\n        },\n        \"properties\" : {\n          \"geoip\" : {\n            \"dynamic\" : true,\n            \"type\" : \"object\",\n            \"properties\" : {\n              \"location\" : {\n                \"type\" : \"geo_point\"\n              }\n            }\n          },\n          \"@version\" : {\n            \"index\" : \"not_analyzed\",\n            \"type\" : \"string\"\n          }\n        }\n      }\n    },\n    \"aliases\" : { }\n}'\n```\n\n#### 2. update logstash config:\n\nadd the line __template_overwrite =\u003e true__ to the output-section of your __logstash.conf__:\n\n```\noutput {\n\telasticsearch {\n\t\thosts =\u003e [ \"localhost:9200\" ]\n\t\ttemplate_overwrite =\u003e true\n\t}\n}\n```\n\n#### 3. see logstash logs\n\n```\ntail -f /var/log/logstash/logstash.log\n```\n\n#### 4. need more details in logstash logging while having problems with json filter?\n\nAdd the following to your logstash output section:\n```\noutput {\n  file {\n        path =\u003e \"/var/log/logstash/jsonparsefailure.debug.log\"\n        codec =\u003e \"rubydebug\"\n    }\n}\n```\n\n#### 5. Fix Parsed JSON object/hash requires a target configuration option\n\nIf you get {:timestamp=\u003e\"2016-09-29T11:10:05.559000+0200\", :message=\u003e\"Parsed JSON object/hash requires a target configuration option\", :source=\u003e\"message\", :raw=\u003e\"\", :level=\u003e:warn}\n\n\n\n## and finally: kibana also want´s to be updated:\n\nIf kibana says: \"unable to fetch mapping\", when you want to create an index, then you have to manually create an logstash-index in elasticsearch:\n```\ncurl -XPUT 'http://localhost:9200/logstash-2016.09.24/' -d '{\n    \"settings\" : {\n        \"index\" : {\n            \"number_of_shards\" : 3, \n            \"number_of_replicas\" : 0 \n        }\n    }\n}'\n```\n\nIf the Settings/Indices Setup-Page has an empty __Time-field name__ dropdownbox, do these steps: http://stackoverflow.com/a/29535262/4964553\n\n\n## Helpful Links\n\nhttps://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html\n\nhttps://www.elastic.co/guide/en/elasticsearch/reference/1.4/cluster-nodes-shutdown.html\n\nhttp://blog.florian-hopf.de/2015/02/fixing-elasticsearch-allocation-issues.html\n\nhttp://blog.kiyanpro.com/2016/03/06/elasticsearch/reroute-unassigned-shards/\n\nhttps://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Felasticsearch-cheatsheet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjonashackt%2Felasticsearch-cheatsheet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjonashackt%2Felasticsearch-cheatsheet/lists"}