{"id":20285469,"url":"https://github.com/mramshaw/elastic","last_synced_at":"2026-05-12T14:01:49.318Z","repository":{"id":92905231,"uuid":"205558825","full_name":"mramshaw/Elastic","owner":"mramshaw","description":"Diving into Elasticsearch","archived":false,"fork":false,"pushed_at":"2019-11-01T13:01:42.000Z","size":1338,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-14T08:12:13.151Z","etag":null,"topics":["amazon-elasticsearch-service","docker","elasticsearch","elk","elk-stack","kibana","luxon","serverless"],"latest_commit_sha":null,"homepage":null,"language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mramshaw.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-31T15:01:32.000Z","updated_at":"2019-11-01T13:01:44.000Z","dependencies_parsed_at":"2023-04-29T00:55:10.818Z","dependency_job_id":null,"html_url":"https://github.com/mramshaw/Elastic","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2FElastic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2FElastic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2FElastic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mramshaw%2FElastic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mramshaw","download_url":"https://codeload.github.com/mramshaw/Elastic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241780512,"owners_count":20019061,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amazon-elasticsearch-service","docker","elasticsearch","elk","elk-stack","kibana","luxon","serverless"],"created_at":"2024-11-14T14:26:56.742Z","updated_at":"2025-11-30T17:04:10.350Z","avatar_url":"https://github.com/mramshaw.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Elastic\n\n![Elasticsearch graphic](images/apple-icon-144x144.png)\n\nGetting up-to-speed with the __Elastic Stack__ (formerly the __ELK stack__) which consists of\n [Elasticsearch](http://www.elastic.co/products/elasticsearch),\n [Logstash](http://www.elastic.co/products/logstash), and\n [Kibana](http://www.elastic.co/products/kibana).\n\n![Elasticsearch graphic](images/logo-elastic-search-color-64.svg)\n![Logstash graphic](images/logo-logstash-64-color.svg)\n![Kibana graphic](images/logo-kibana-64-color.svg)\n\nAnd now [Beats](http://www.elastic.co/products/beats).\n\n![Beats graphic](images/logo-beats-64-color.svg)\n\n## Contents\n\nThe content are as follows:\n\n* [Motivation](#motivation)\n* [Elasticsearch overview](#elasticsearch-overview)\n* [Use Cases](#use-cases)\n    * [Product Catalogues](#products-catalogues)\n    * [Application or Site Search](#application-or-site-search)\n    * [Data Visualization](#data-visualization)\n    * [Log ingestion and analysis](#log-ingestion-and-analysis)\n    * [Data assembly, standardization and enrichment](#data-assembly-standardization-and-enrichment)\n    * [Network, System and Performance monitoring](#network-system-and-performance-monitoring)\n    * [Data Science](#data-science)\n    * [Machine Learning](#machine-learning)\n* [Creating a cluster](#creating-a-cluster)\n    * [AWS](#aws)\n    * [GCP](#gcp)\n    * [Docker](#docker)\n    * [Kubernetes](#kubernetes)\n* [Version](#version)\n* [Elasticsearch and Semver](#elasticsearch-and-semver)\n* [Health](#health)\n* [PUT](#put)\n* [Indices](#indices)\n* [Aliases](#aliases)\n* [POSTs](#posts)\n* [GET](#get)\n    * [Specific query](#specific-query)\n* [Kibana](#kibana)\n    * [Useful commands](#useful-commands)\n    * [Leading slashes](#leading-slashes)\n    * [Human-readable output](#human-readable-output)\n    * [Intellisense](#intellisense)\n    * [Kibana tools](#kibana-tools)\n    * [Ctrl-F is your friend](#ctrl-f-is-your-friend)\n    * [Debugging](#debugging)\n* [Searches](#searches)\n    * [How to override the 10,000 items query limit](#how-to-override-the-10000-items-query-limit)\n    * [Case](#case)\n* [Leaf query clauses](#leaf-query-clauses)\n    * [match](#match)\n    * [term](#term)\n    * [range](#range)\n* [DELETE](#delete)\n    * [DELETE individual item](#delete-individual-item)\n    * [DELETE items](#delete-items)\n* [Bulk loading](#bulk-loading)\n* [Index/Alias problems](#indexalias-problems)\n* [Aggregates](#aggregates)\n* [Beats and Logstash](#beats-and-logstash)\n    * [Beats](#beats)\n    * [Logstash](#logstash)\n* [X-Pack](#x-pack)\n* [Podcasts](#podcasts)\n    * [SE-Radio](#se-radio)\n    * [SE-Daily](#se-daily)\n    * [The Changelog](#the-changelog)\n* [Reference](#reference)\n    * [Glossary](#glossary)\n    * [Mappings](#mappings)\n    * [Mappings and Amazon Elasticsearch Service](#mappings-and-amazon-elasticsearch-service)\n    * [Logstash plugins](#logstash-plugins)\n    * [Dates](#dates)\n    * [Number formatting](#number-formatting)\n    * [Indices and Aliases](#indices-and-aliases)\n    * [Bulk API](#bulk-api)\n    * [Update documentation](#update-documentation)\n    * [Reindex API](#reindex-api)\n    * [Deprecation logging](#deprecation-logging)\n    * [Upgrading Elasticsearch](#upgrading-elasticsearch)\n    * [AWS Developer Guide](#aws-developer-guide)\n    * [PUT and POST](#put-and-post)\n* [To Do](#to-do)\n* [Credits](#credits)\n\n## Motivation\n\nHaving looked at [Cassandra with Python](http://github.com/mramshaw/Python_Cassandra), [Couchbase](http://github.com/mramshaw/RESTful-Couchbase) and [DynamoDB](http://github.com/mramshaw/DynamoDB),\n this time Elasticsearch is in the cross-hairs.\n\nUnder the covers Elasticsearch uses [Apache Lucene](http://lucene.apache.org/).\n\nElasticsearch is very similiar to [Apache Solr](http://lucene.apache.org/solr/).\nBoth Elasticsearch and Solr are built on top of Lucene, and seem to offer roughly\nsimiliar features. It's worth remembering that Elasticsearch, while open-source,\nis backed by a commercial (for-profit) company.\n\nAmazon offers both as services: Elasticsearch is marketed as\n [Amazon Elasticsearch Service](http://aws.amazon.com/elasticsearch-service/)\n while Solr is marketed as [Amazon CloudSearch](http://aws.amazon.com/cloudsearch/).\nAmazon Elasticsearch Service is based on a cluster of managed servers (where scaling\nneeds to be managed) while Amazon CloudSearch is a managed service which autoscales.\n\nThere are a number of options for Elasticsearch (on premise, etc), but I will be looking at\n[Amazon Elasticsearch Service](#amazon-elasticsearch-service).\n\n#### Amazon Elasticsearch Service\n\nAmazon describes it as follows:\n\n\u003e Amazon Elasticsearch Service is a fully managed service that makes it easy for you to deploy, secure, and operate Elasticsearch at scale with zero down time.\n\n[__at scale__, __zero down time__]\n\n\u003e The service offers open-source Elasticsearch APIs, managed [Kibana](http://aws.amazon.com/elasticsearch-service/the-elk-stack/kibana/), and integrations with\n\u003e [Logstash](http://aws.amazon.com/elasticsearch-service/the-elk-stack/logstash/) and other AWS Services, enabling you to securely ingest data from any source\n\u003e and search, analyze, and visualize it in real time.\n\n[So a full ELK stack. This stack is popular with developers.]\n\n\u003e Amazon Elasticsearch Service lets you pay only for what you use – there are no upfront costs or usage requirements.\n\n[So a good match for __serverless__ computing.]\n\nAll of the above quotes are from: http://aws.amazon.com/elasticsearch-service/\n\n## Elasticsearch Overview\n\nElasticsearch is a __distributed__, __sharded__ database with __no required indices__.\n\nIt is a __NoSQL__ database, so theoretically __schema-less__ (although the recommended practice is to define a schema).\n\nIt is primarily oriented towards ___full-text search___.\n\nIn terms of NoSQL categories, it seems to usually be defined as a __real-time search and analytics engine__ first,\nand a __Document store__ second. Of course, many relational databases also offer full-text search capabilities,\nhowever it is probably its real-time streaming characteristics that make Elasticsearch attractive.\n\n[As a document-store, it seems to offer a strong challenge to MongoDB. And it can also handle __search__.]\n\nElasticsearch seems to be very similiar to [Apache Cassandra](http://cassandra.apache.org/).\n\nCassandra has its own __CQL__ while Couchbase has __N1QL__; on the other hand Elasticsearch is __RESTful__.\n\n\u003e Scales good for reads, ok’ish for writes\n\nFrom: http://vlkan.com/blog/post/2018/11/14/elasticsearch-primary-data-store/cncml-vienna-2019.pdf\n\nWrites are handled via __shards__ while reads are handled by __replicas__. Replicas may be scaled up\ndynamically (for instance, for Black Friday).\n\nIts terminology is a little weird - for instance it refers to what might normally be called __databases__ as __indices__.\n\n[It seems these actually were just indices - just with the data included (this enables re-indexing, which is a big plus).]\n\nIntegrity is maintained with a [CAS](http://en.wikipedia.org/wiki/Compare-and-swap) (compare and swap) versioning mechanism.\n\nIt's a little unusual (for a database anyway) in that it offers __fuzzy search__ options.\n\n## Use Cases\n\nAs noted above, Elasticsearch is optimized for __search__, and more specifically - ___full-text search___.\n\nEven so, it can be put to a surprisingly varied number of uses, some of which are listed below.\n\n#### Product Catalogues\n\nAs a result of its ___full-text search___ capabilities, Elasticsearch is ideal for Product Catalogues.\n\n[Although probably not for all languages (Latin-based languages should be okay). And as noted below,\n ES does not handle frequent updates particularly well, so may not be up-to-the-minute accurate for\n pricing data.]\n\n#### Application or Site Search\n\nTo quickly add search to an app or website, there is [Search UI](http://github.com/elastic/search-ui).\n\n#### Data Visualization\n\nAt the enterprise level, it's very useful for the types of data visualization and dashboarding\n offered by say, [Tableau](http://www.tableau.com/) (among others).\n\n#### Log ingestion and analysis\n\nFor log ingestion and analysis, offerings from __DataDog__ and __Splunk__ may be more feature-rich,\nstill this is a very common use case for Elasticsearch.\n\nThere are quick and easy ingestion options for many common log formats such as:\n\n1. Apache logs\n1. Cloudwatch logs\n1. Elasticsearch logs\n1. Kafka logs\n1. Logstash logs\n1. MySQL logs\n1. Nats logs\n1. Nginx logs\n1. PostgreSQL logs\n1. Syslogs\n1. Traefik logs\n\nIn addition, there are numerous other ingestion options available via [Logstash plugins](#logstash-plugins).\n\n[Elastic appear to have taken note of the value of standard log formats, and with Elastic 7.0 have\n introduced the [Elastic Common Schema](http://www.elastic.co/guide/en/ecs/1.1/index.html), to which\n all Beats now conform.]\n\n#### Data assembly, standardization and enrichment\n\nFor assembling data from disparate sources, standardization and enrichment (if needed) are critical.\nElastic has recognized this - and offer [Beats](http://www.elastic.co/products/beats) to cover most\neventualities.\n\n#### Network, System and Performance monitoring\n\nThere are specialized Beats for monitoring Network, System and Performance data.\n\nFor these purposes it can be used for the type of dashboarding provided by\n [Grafana](http://grafana.com/) (among others).\n\n#### Data Science\n\nDue to it's ability to ingest and scan documents, it is becoming very useful for __data science__.\n\n#### Machine Learning\n\nElasticsearch offers easy integration for [Machine Learning](http://www.elastic.co/what-is/elasticsearch-machine-learning).\n\n## Creating a cluster\n\nThere are a number of options for creating an Elasticsearch cluster.\n\n#### AWS\n\nIt is possible to create an AWS test cluster from http://www.elastic.co/ but it is also possible to do this from AWS itself.\n\nFollow the link to see the steps involved with the [AWS option](01-Amazon_Elasticsearch_Service.md).\n\n#### GCP\n\nOf course, Elasticsearch is also available on [GCP](http://www.elastic.co/about/partners/google-cloud-platform),\nwhich may in fact be a stronger option than the AWS offering (which tends to lag behind the ES releases).\n\n#### Docker\n\nFollow the link to see the steps involved with the [Docker option](02-Docker.md).\n\n#### Kubernetes\n\nAnd of course, if ES is available on Docker, it's no surprise to see it available on\n[Kubernetes](http://www.elastic.co/products/elastic-cloud-kubernetes) - where it can\nalso be used as an alternative to the default Stackdriver logging.\n\n## Version\n\nLet's see what version of Elasticsearch we are dealing with first.\n\nClick the wrench icon (LHS) to navigate to the Kibana __Dev Tools__ console.\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nGET /\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"name\" : \"b7f9cb7ad431\",\n  \"cluster_name\" : \"docker-cluster\",\n  \"cluster_uuid\" : \"uXuvtXXnQpSJ3I-F9VqxBg\",\n  \"version\" : {\n    \"number\" : \"7.3.1\",\n    \"build_flavor\" : \"default\",\n    \"build_type\" : \"docker\",\n    \"build_hash\" : \"4749ba6\",\n    \"build_date\" : \"2019-08-19T20:19:25.651794Z\",\n    \"build_snapshot\" : false,\n    \"lucene_version\" : \"8.1.0\",\n    \"minimum_wire_compatibility_version\" : \"6.8.0\",\n    \"minimum_index_compatibility_version\" : \"6.0.0-beta1\"\n  },\n  \"tagline\" : \"You Know, for Search\"\n}\n```\n\n[This should exactly match what we got from our ES cluster via our http://localhost:9200 URL.]\n\nNote the version number (affects API calls, etc) which is __7.3.1__.\n\n## Elasticsearch and Semver\n\nTHIS POINT CANNOT BE STRESSED ENOUGH. YOU HAVE BEEN WARNED!\n\nElastic does not follow the standard practice (usually referred to as __Semver__ or [Semantic Versioning](http://semver.org/))\nof only releasing breaking changes with major releases.\n\n[This is based upon my own personal experience, but feel free to disregard my comments.]\n\nWhile it should be expected there *might* be breaking changes going from a 6.y.z version to a 7.y.z version,\nit seems there are often breaking changes going from a x.1.z version to an x.2.z version. As an example, for\ncreating aggregate histograms in Elasticsearch __7.1.1__ versus __7.2.1__, the ___interval___ keyword was\ndeprecated and replaced with ___calendar_interval___ and/or ___fixed_interval___.\n\nCompare:\n\n    https://www.elastic.co/guide/en/elasticsearch/reference/7.1/search-aggregations-bucket-datehistogram-aggregation.html\n\nAnd:\n\n    https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-aggregations-bucket-datehistogram-aggregation.html\n\nWhile it may be that the new terms have clearer semantics than the old term, and while I applaud Elastic for continuing to\nsupport and expand their software, I think a good argument could be made that this type of change requires a __major release__\nand not simply a __minor release__.\n\n[I have not tested this so it may well be that ___interval___ will not actually be removed until the next major release,\n however no developer really likes breaking changes and there seem to have been far too many of these during the brief\n time that I have been working with Elasticsearch. The problem with breaking changes is not so much that code needs to\n be changed - _every developer expects and understands this_ - but that it then needs to be __tested__ afterwards.\n Testing code that may have been running happily for some time can often turn out to be a non-trivial problem.]\n\n## Health\n\nLet's check the health of our cluster:\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nGET /_cat/health?v\n```\n\nWhich should look something like the following:\n\n![Kibana running](images/Kibana_running.png)\n\n## PUT\n\nLet's create an index (database instance) first.\n\n[This step is actually optional as Elasticsearch will *auto-magically* create indices and types that\n do not exist, but it is a good step to try in terms of getting familiar with Elasticsearch and Kibana.]\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nPUT school\n```\n\n[Creates index - Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"acknowledged\" : true,\n  \"shards_acknowledged\" : true,\n  \"index\" : \"school\"\n}\n```\n\nTo delete the index (database instance), there is [DELETE](#delete).\n\n## Indices\n\nHaving created an index, let's see what indices we have:\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nGET /_cat/indices\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\ngreen  open .kibana_task_manager Bw7bdH5oTBOPIwB7JrzXaQ 1 0 2 0 45.5kb 45.5kb\nyellow open school               HOGuhBS5TKCjpP84f-CnjQ 1 1 0 0   230b   230b\ngreen  open .kibana_1            DuPECNdXSJu0vaUeOnmwxA 1 0 3 0 11.7kb 11.7kb\n```\n\n[The `yellow` status for our schools index indicates we do not have the recommended number of replicas\n for our school index, which is fine for a testing cluster. This is a warning status, where `red`\n indicates an error condition and `green` signifies `all systems go`. So a standard traffic light.]\n\nNote that `/_cat/indices` will not show [Aliases](#aliases).\n\n## Aliases\n\nA definition:\n\n\u003e aliases are like soft links or shortcuts to actual indexes\n\u003e\n\u003e the advantage is to be able to have an alias pointing to index1a while building or\n\u003e re-indexing on index2b and the moment of swapping them is atomic thanks to the alias\n\nAs:\n\n\u003e Renaming an alias is a simple remove then add operation within the same API. This operation is atomic,\n\u003e no need to worry about a short period of time where the alias does not point to an index\n\nAlso:\n\n\u003e Multiple indices can be specified for an action\n\nAnd:\n\n\u003e We will talk more about the other uses for aliases later in the book. For now we will explain how to\n\u003e use them to switch from an old index to a new index with zero downtime.\n\nAll from: http://stackoverflow.com/questions/48907041/what-are-aliases-in-elasticsearch-for\n\nSo let's create an alias.\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nPOST /_aliases\n{\n    \"actions\" : [\n        { \"add\" : { \"index\" : \"school\", \"alias\" : \"academia\" } }\n    ]\n}\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"acknowledged\" : true\n}\n```\n\nAnd check it exists (200 for yes, 404 for no):\n\n```\nHEAD academia\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n200 - OK\n```\n\nAnd check out the statistics for this alias:\n\n```\nGET academia/_stats\n```\n\nAnd just for fun, \u003ckbd\u003eCtrl-Enter\u003c/kbd\u003e (on OS/X, \u003ckbd\u003eCommand-Enter\u003c/kbd\u003e) to execute:\n\n```\n{\n  \"_shards\" : {\n    \"total\" : 2,\n    \"successful\" : 1,\n    \"failed\" : 0\n  },\n  \"_all\" : {\n    \"primaries\" : {\n      \"docs\" : {\n\n    ...\n\n      }\n    },\n    \"total\" : {\n      \"docs\" : {\n\n    ...\n\n      }\n    }\n  },\n  \"indices\" : {\n    \"school\" : {\n      \"uuid\" : \"JN_oqm2fTfCyjSMzWskbfA\",\n      \"primaries\" : {\n        \"docs\" : {\n\n    ...\n\n        }\n      },\n      \"total\" : {\n        \"docs\" : {\n\n    ...\n\n        },\n        \"fielddata\" : {\n          \"memory_size_in_bytes\" : 0,\n          \"evictions\" : 0\n\n    ...\n\n        }\n      }\n    }\n  }\n}\n```\n\n[In general, use of `fielddata` is probably to be avoided.]\n\n## POSTs\n\nNow let's create some entries.\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nPOST school/_doc/10\n{\n   \"name\":\"Saint Paul School\", \"description\":\"ICSE Afiliation\",\n   \"street\":\"Dawarka\", \"city\":\"Delhi\", \"state\":\"Delhi\", \"zip\":\"110075\",\n   \"location\":[28.5733056, 77.0122136], \"fees\":5000,\n   \"tags\":[\"Good Faculty\", \"Great Sports\"], \"rating\":\"4.5\"\n}\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"_index\" : \"school\",\n  \"_type\" : \"_doc\",\n  \"_id\" : \"10\",\n  \"_version\" : 1,\n  \"result\" : \"created\",\n  \"_shards\" : {\n    \"total\" : 2,\n    \"successful\" : 1,\n    \"failed\" : 0\n  },\n  \"_seq_no\" : 0,\n  \"_primary_term\" : 1\n}\n```\n\n[Looks like anything starting with an underscore is metadata.]\n\nAnd:\n\n```\nPOST school/_doc/16\n{\n   \"name\":\"Crescent School\", \"description\":\"State Board Affiliation\",\n   \"street\":\"Tonk Road\",\n   \"city\":\"Jaipur\", \"state\":\"RJ\", \"zip\":\"176114\",\"location\":[26.8535922,75.7923988],\n   \"fees\":2500, \"tags\":[\"Well equipped labs\"], \"rating\":\"4.5\"\n}\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"_index\" : \"school\",\n  \"_type\" : \"_doc\",\n  \"_id\" : \"16\",\n  \"_version\" : 1,\n  \"result\" : \"created\",\n  \"_shards\" : {\n    \"total\" : 2,\n    \"successful\" : 1,\n    \"failed\" : 0\n  },\n  \"_seq_no\" : 0,\n  \"_primary_term\" : 1\n}\n```\n\n## GET\n\nLet's get a description (definition) of our index.\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nGET school\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"school\" : {\n    \"aliases\" : { },\n    \"mappings\" : {\n      \"_doc\" : {\n        \"properties\" : {\n          \"city\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"description\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"fees\" : {\n            \"type\" : \"long\"\n          },\n          \"location\" : {\n            \"type\" : \"float\"\n          },\n          \"name\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"rating\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"state\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"street\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"tags\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          },\n          \"zip\" : {\n            \"type\" : \"text\",\n            \"fields\" : {\n              \"keyword\" : {\n                \"type\" : \"keyword\",\n                \"ignore_above\" : 256\n              }\n            }\n          }\n        }\n      }\n    },\n    \"settings\" : {\n      \"index\" : {\n        \"creation_date\" : \"1566930221735\",\n        \"number_of_shards\" : \"5\",\n        \"number_of_replicas\" : \"1\",\n        \"uuid\" : \"xxxxxxxxxxxxxxxxxxxxxx\",\n        \"version\" : {\n          \"created\" : \"6050499\"\n        },\n        \"provided_name\" : \"school\"\n      }\n    }\n  }\n}\n```\n\n[fees is a ___long___, zip is a ___float___; others are text fields.]\n\n#### Specific Query\n\nNow let's try retrieving a specific entry.\n\nFrom the Kibana __Dev Tools__ console:\n\n```\nGET /school/_search?q=name:\"Saint Paul School\"\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"took\" : 46,\n  \"timed_out\" : false,\n  \"_shards\" : {\n    \"total\" : 5,\n    \"successful\" : 5,\n    \"skipped\" : 0,\n    \"failed\" : 0\n  },\n  \"hits\" : {\n    \"total\" : 1,\n    \"max_score\" : 0.8630463,\n    \"hits\" : [\n      {\n        \"_index\" : \"school\",\n        \"_type\" : \"_doc\",\n        \"_id\" : \"10\",\n        \"_score\" : 0.8630463,\n        \"_source\" : {\n          \"name\" : \"Saint Paul School\",\n          \"description\" : \"ICSE Afiliation\",\n          \"street\" : \"Dawarka\",\n          \"city\" : \"Delhi\",\n          \"state\" : \"Delhi\",\n          \"zip\" : \"110075\",\n          \"location\" : [\n            28.5733056,\n            77.0122136\n          ],\n          \"fees\" : 5000,\n          \"tags\" : [\n            \"Good Faculty\",\n            \"Great Sports\"\n          ],\n          \"rating\" : \"4.5\"\n        }\n      }\n    ]\n  }\n}\n```\n\n## Kibana\n\nKibana is an open-source data visualization plugin for Elasticsearch.\n\n![Kibana tools](images/Kibana_tools.png)\n\nClick the wrench icon (LHS) to navigate to the Kibana __Dev Tools__ console.\n\nYour last few Kibana Dev Tools sessions will be cached, which is a very useful feature.\n\n#### Useful commands\n\nCheck cluster is running:\n\n    GET /\n\nCheck cluster health:\n\n    GET /_cat/health?v\n\nList all indices:\n\n    GET /_cat/indices?v\n\nList all aliases:\n\n    GET /_cat/aliases?v\n\nList an index's aliases, mappings and settings:\n\n    GET /some_index\n\nList _only_ an index's mappings:\n\n    GET /some_index/_mapping\n\n    GET /some_alias/_mapping\n\nHow to check a field's mapping:\n\n    GET /some_index/_mapping/field/some_field\n\n    GET /some_alias/_mapping/field/some_field\n\n#### Leading slashes\n\nIn general leading slashes may be omitted. However, as they are *sometimes* needed,\nperhaps a better practice is to get into the habit of ___always___ using a leading slash.\n\n#### Human-readable output\n\nFor human-readable output add __?v__ to the end of the initial line:\n\n    GET /_cat/indices\n\nGives:\n\n```\nyellow open some_index           JDyFX-XjQMu95f3KXZnPpQ 1 1 1 0    5kb    5kb\ngreen  open .kibana_task_manager 4wLH-SPCRTKreXpzmjmhoA 1 0 2 0 79.1kb 79.1kb\ngreen  open .kibana_1            Un0M5PpqTlWDlib_qmsXGw 1 0 4 1 23.9kb 23.9kb\n```\n\nWhile:\n\n    GET /_cat/indices?v\n\nGives:\n\n```\nhealth status index                uuid                   pri rep docs.count docs.deleted store.size pri.store.size\nyellow open   some_index           JDyFX-XjQMu95f3KXZnPpQ   1   1          1            0        5kb            5kb\ngreen  open   .kibana_task_manager 4wLH-SPCRTKreXpzmjmhoA   1   0          2            0     79.1kb         79.1kb\ngreen  open   .kibana_1            Un0M5PpqTlWDlib_qmsXGw   1   0          4            1     23.9kb         23.9kb\n```\n\n#### Intellisense\n\nKibana has a really useful `Intellisense` auto-complete feature:\n\n![Intellisense](images/Intellisense.png)\n\n#### Kibana tools\n\nIn addition to executing Elasticsearch queries (green Play button),\nthe Dev Tools console in Kibana also has some nice tools (diagonal wrench icon).\n\ncURL (from Spanner - 'Copy as cURL'):\n\n\tcurl -XGET \"http://localhost:9200/school/_search?q=rating:4.5\"\n\nThe 'Auto indent' feature is also useful - it toggles between expanded and compact formats.\nIt will only work for correctly-formatted JSON, which can be a quick and convenient way to\ncheck for correctly-formed JSON.\n\n#### Ctrl-F is your friend\n\nThe workspaces in Kibana can get pretty crowded very quickly, so being able to find things\nvia ___searching___ is a very nice feature. It is possible to search in both the __query__\nand the __results__ panels, as the following screencap shows:\n\n![Ctrl-F is your friend](images/Ctrl-F_is_your_friend.png)\n\n#### Debugging\n\nAlthough the Kibana workspace can include dozens of queries, the results panel will always\nrefer to the current Elasticsearch command as if it had been executed in isolation.\n\nSpacing out your query so it starts on a line ending in a __1__ should make debugging and\nproblems slightly easier, as the following annotated screencap shows:\n\n![Dealing in tens](images/Dealing_in_tens.png)\n\n## Searches\n\nIt is possible to search without any criteria (a 'match all query', the equivalent of `head` or `tail` in *nix) to explore data.\n\n```\nGET /shakespeare/_search\n```\n\n#### How to override the 10,000 items query limit\n\nThere is a soft limit of 10,000 items on queries. This is to prevent runaway queries from killing the cluster.\n\n[The 10,000 items soft limit for queries was introduced with Elasticsearch 7.0+]\n\nTo report on more than 10,000 items, add the following line:\n\n`\"track_total_hits\": true,`\n\nAs in:\n\n```\nGET some_index/_search\n{\n  \"track_total_hits\": true,\n  \"query\": {\n    \"bool\": {\n      \"must_not\": {\n        \"exists\": {\n          \"field\": \"some_field\"\n        }\n      }\n    }\n  }\n}\n```\n\n[This is a query for all documents that do not contain the term 'some_field'.]\n\n#### Case\n\nCase is important - in some cases only lower-case will match (presumably text fields are canonicalized,\nso \"USD\" will only match if \"usd\" is requested), whereas in other situations only the exact text will match:\n\n```\nGET /_search\n{\n    \"query\": {\n        \"query_string\" : {\n            \"default_field\" : \"city\",\n            \"query\" : \"jaipur\"\n        }\n    }\n}\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"took\" : 53,\n  \"timed_out\" : false,\n  \"_shards\" : {\n    \"total\" : 324,\n    \"successful\" : 324,\n    \"skipped\" : 0,\n    \"failed\" : 0\n  },\n  \"hits\" : {\n    \"total\" : 1,\n    \"max_score\" : 0.2876821,\n    \"hits\" : [\n      {\n        \"_index\" : \"school\",\n        \"_type\" : \"_doc\",\n        \"_id\" : \"16\",\n        \"_score\" : 0.2876821,\n        \"_source\" : {\n          \"name\" : \"Crescent School\",\n          \"description\" : \"State Board Affiliation\",\n          \"street\" : \"Tonk Road\",\n          \"city\" : \"Jaipur\",\n          \"state\" : \"RJ\",\n          \"zip\" : \"176114\",\n          \"location\" : [\n            26.8535922,\n            75.7923988\n          ],\n          \"fees\" : 2500,\n          \"tags\" : [\n            \"Well equipped labs\"\n          ],\n          \"rating\" : \"4.5\"\n        }\n      }\n    ]\n  }\n}\n```\n\n## Leaf query clauses\n\nhttp://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html\n\n#### match\n\nSeems to be used for exact matches (EQ).\n\n#### term\n\nSeems to be used for text matching - contains the specified term (LIKE, etc).\n\n#### range\n\nUsed for numeric matching (LT, LTE, GT, GTE).\n\n```\nGET /_search\n{\n  \"query\": {\n    \"range\": {\n      \"fees\": {\n        \"gt\": \"2500\"\n      }\n    }\n  }\n}\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"took\" : 9,\n  \"timed_out\" : false,\n  \"_shards\" : {\n    \"total\" : 324,\n    \"successful\" : 324,\n    \"skipped\" : 319,\n    \"failed\" : 0\n  },\n  \"hits\" : {\n    \"total\" : 1,\n    \"max_score\" : 1.0,\n    \"hits\" : [\n      {\n        \"_index\" : \"school\",\n        \"_type\" : \"_doc\",\n        \"_id\" : \"10\",\n        \"_score\" : 1.0,\n        \"_source\" : {\n          \"name\" : \"Saint Paul School\",\n          \"description\" : \"ICSE Afiliation\",\n          \"street\" : \"Dawarka\",\n          \"city\" : \"Delhi\",\n          \"state\" : \"Delhi\",\n          \"zip\" : \"110075\",\n          \"location\" : [\n            28.5733056,\n            77.0122136\n          ],\n          \"fees\" : 5000,\n          \"tags\" : [\n            \"Good Faculty\",\n            \"Great Sports\"\n          ],\n          \"rating\" : \"4.5\"\n        }\n      }\n    ]\n  }\n}\n```\n\nIt's pretty error-tolerant, even the following worked:\n\n```\nGET /_search\n{\n  \"query\": {\n    \"range\": {\n      \"eventAt\": {\n        \"gte\": \"2500\"\n      }\n    }\n  }\n}\n```\n\n## term query may return poor results when searching text fields\n\nhttp://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#avoid-term-query-text-fields\n\n## DELETE\n\nDeletes the specified index (and all of the indexed data as well):\n\n```\nDELETE school\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"acknowledged\" : true\n}\n```\n\n#### DELETE individual item\n\n```\nDELETE school/_doc/10\n```\n\n[Click the green Play button to execute]\n\nResponse:\n\n```\n{\n  \"_index\" : \"school\",\n  \"_type\" : \"_doc\",\n  \"_id\" : \"10\",\n  \"_version\" : 2,\n  \"result\" : \"deleted\",\n  \"_shards\" : {\n    \"total\" : 2,\n    \"successful\" : 1,\n    \"failed\" : 0\n  },\n  \"_seq_no\" : 1,\n  \"_primary_term\" : 1\n}\n```\n\n[Note that __result__ is __deleted__; also __version__ is __2__ as index has been deleted and then re-created.]\n\n#### DELETE items\n\nIf you are brave, things like this usually work:\n\n```\nDELETE some_index\n{\n \"query\": {\n   \"exists\": {\n     \"field\": \"some_field\"\n   }\n }\n}\n```\n\n[The risk of deleting more documents than you intended hardly needs to be stated.]\n\n## Bulk loading\n\nBulk loading can be accessed at `/_bulk`.\n\nUsing `curl` [more schools](./more_schools) can be loaded as follows:\n\n```bash\n$ curl -H \"Content-Type: application/x-ndjson\" -XPOST localhost:9200/school/_bulk?pretty --data-binary \"@more_schools\"; echo\n```\n\n[The __?pretty__ option means pretty-print the output. This can be omitted. Can optionally add the `--silent` option to curl.\n The `; echo` part at the end is simply to get a newline so that our console output is readable.]\n\n## Index/Alias problems\n\nIt is quite easy to get into a chicken \u0026 egg situation with indices and aliases. Apparently this is a common enough\nproblem that there is a pretty easy fix:\n\n![Index/Alias remapping](images/Index_Alias.png)\n\nFrom: http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html\n\n## Aggregates\n\nAggregates are kind of a fancy way to say ___summaries___.\n\nIt's possible to specify that you only want the big picture (no details) by specifying `\"size\": 0`:\n\n```\nGET some_index/_search\n{\n  \"query\": {...},\n  \"size\": 0,\n  \"aggregations\": {\n    \"dateHistogram\": {\n      \"date_histogram\": {\n        \"field\": \"@timestamp\",\n        \"fixed_interval\": \"10s\"\n      }\n    }\n  }\n}\n```\n\n## Beats and Logstash\n\nBoth [Beats](http://www.elastic.co/products/beats) and [Logstash](http://www.elastic.co/products/logstash)\nare aggregation agents. Logstash can ship to Elasticsearch while Beats can ship to either Logstash or to\nElasticsearch directly.\n\nI have yet to determine how they differ, but at first glance it seems that Beats are a \"push\" arrangement\nwhile Logstash is more of a \"pull\" arrangement. However this is merely a guess on my part. Presumably\nLogstash itself can either poll or receive, as it can be used to aggregate/enrich data from Beats.\n\nFor more details about Beats and Logstash, refer to [Logstash plugins](#logstash-plugins).\n\n#### Beats\n\nThese seem to be small modules that are designed for specific uses, such as network and/or packet monitors\nand the like.\n\nFor instance, there is [Functionbeat](http://www.elastic.co/products/beats/functionbeat) which can monitor\n__FaaS__ modules, such as __serverless__ components. [The Elastic diagram shows Cloudwatch Logs, Kinesis,\nand SQS. It is not entirely clear whether Functionbeat is meant to be an adjunct or an alternative to\n[AWS X-Ray](http://aws.amazon.com/xray/).]\n\nThere is also [Heartbeat](http://www.elastic.co/products/beats/heartbeat) which can be used to monitor\nsystem uptime data.\n\nIn addition to the Beats provided by Elastic, there are also Beats supported by the Elastic community.\n\n#### Logstash\n\nAs far as I can tell, this is more of a traditional log ingestion tool.\n\n## X-Pack\n\nFor additional requirements such as ___security___, ___access logging___ and ___alerting___ (to comply with\nthe GDPR, perhaps) there is the __X-Pack__ - which has now been [opened up](http://www.elastic.co/what-is/open-x-pack)\n(for more details of this, refer to the [The Changelog podcast](#the-changelog)).\n\n## Podcasts\n\nSome of the following podcast episodes may be helpful for getting an overview of Elasticsearch.\n\n#### SE-Radio\n\n\u003e Software Engineering Radio is a podcast targeted at the professional software developer.\n\nA good overview of the Elastic Stack:\n\n    http://www.se-radio.net/2017/05/se-radio-episode-292-philipp-krenn-on-elasticsearch/\n\nJeff Meyerson, the interviewer, went on to found [Software Engineering Daily](http://softwareengineeringdaily.com/).\n\nAs the stack has evolved, Logstash seems to have spilt into [Beats](http://www.elastic.co/products/beats) (which are\naggregation agents that can ship to either Elasticsearch or Logstash) and [Logstash](http://www.elastic.co/products/logstash)\n(which is used for data ingestion and parsing - and also enrichment [for instance, geolocating IP addresses]).\n\n[Apparently geolocation has evolved to become a strength of Elasticsearch. And __types__ are on their way out.]\n\nPresumably well-structured data can be shipped to Elasticsearch directly by Beats, but less well-structured data\n(that may need to be parsed, processed, aggregated, transformed or enriched) must be passed to Logstash instead.\n\n#### SE-Daily\n\n\u003e Software Engineering Daily features daily interviews about technical software topics.\n\nAn interesting chat about using Elasticsearch at Scale:\n\n    http://softwareengineeringdaily.com/2019/03/20/elasticsearch-at-scale-with-volkan-yazici/\n\nJeff Meyerson interviews Volkan Yazici about his blog post \"Using Elasticsearch as the Primary Data Store\":\n\n    http://vlkan.com/blog/post/2018/11/14/elasticsearch-primary-data-store/\n\nSummary points:\n\n* Elasticsearch does not handle frequent updates particularly well\n* Dynamic fields such as ___prices___ are not normally completely up-to-date but need to be indexed so as to be searchable\n* Nested fields (or objects) can be problematic\n* Great for real-time streaming (which probably correlates with eventual consistency)\n\nAnd a quote:\n\n\u003e Cassandra is really difficult to operate if you want to have it in your own premises. But compared with that Elasticsearch, you can get it right without breaking a sweat.\n\n[Interesting and well worth a listen.]\n\n#### The Changelog\n\n\u003e News and podcasts for developers\n\n[The Changelog](http://changelog.com/) is a podcast in its own right, as well as an umbrella organization for various other interesting podcasts.\n\nJerod and Adam talk with Philipp Krenn about Elasticsearch and doubling down on \"open\"\n\n    http://changelog.com/podcast/292\n\nSome interesting discussions about software licenses and Elasticsearch as a replacement for PostgreSQL's full-text search\n\n[Some good background about search in general and Elasticsearch in particular.]\n\n## Reference\n\nSome useful references follow.\n\n#### Glossary\n\nProbably the place to start:\n\n\thttp://www.elastic.co/guide/en/elasticsearch/reference/current/glossary.html\n\nFor instance, about __shards__:\n\n\u003e A shard is a single Lucene instance\n\nAmazon layers their branding on top of Elastic's, so in AWS a cluster is known as an Amazon ES Domain.\nProbably other terms are similiarly translated.\n\n#### Mappings\n\nIt is impossible to do much in ES without understanding mappings. Read this page:\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html\n\nIndividual mappings (fees, in this case) may be checked with `GET /school/_mapping/field/fees`.\n\n#### Mappings and Amazon Elasticsearch Service\n\nIf using Amazon Elasticsearch Service, their console can be used to view index mappings:\n\n![AWS Kibana mappings](images/Kibana_mappings.png)\n\n#### Logstash plugins\n\nThere are plugins to cover many eventualities and use cases: http://www.elastic.co/guide/en/logstash/current/input-plugins.html\n\n[This page lists an extensive list of plugins that may be used for such things as ingesting logs, metrics and events.\n And once they are in Elasticsearch, Kibana can then provide many templated monitoring and dashboarding options.]\n\nElastic also offers a pretty useful Support Matrix: http://www.elastic.co/support/matrix#show_logstash_plugins\n\n#### Dates\n\nCareful consideration needs to be paid to dates, as most queries will involve time-related processing or date ranges:\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/date.html\n\nTime processing in Elasticsearch is generally processed in either seconds or milliseconds, however __Elasticsearch 7.0__\nintroduced nanosecond processing as an option (internally, Elasticsearch 7.0+ processing will use nanoseconds).\n\nInterestingly, Elasticsearch can support __multiple date formats__ at the same time:\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/date.html#multiple-date-formats\n\nFor date processing with Javascript, there is [Luxon](http://moment.github.io/luxon/).\n\n[Luxon operates in milliseconds.]\n\n#### Number formatting\n\nIt is possible to format numbers:\n\n    http://numeraljs.com/\n\n#### Indices and Aliases\n\nThe index APIs:\n\n    http//www.elastic.co/guide/en/elasticsearch/reference/current/indices.html\n\nA little long, but worth reading:\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html\n\n#### Bulk API\n\n\u003e The bulk API makes it possible to perform many index/delete operations in a single API call. This can greatly increase the indexing speed.\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html\n\nNote that a successful __create__ returns a __201__ HTTP status code while an unsuccessful __delete__ returns a __404__ HTTP status code.\n\nThe __update__ (usually an __upsert__ I think) should normally return a __200__ HTTP status code.\n\n\u003e When using the `update` action, `retry_on_conflict` can be used as a field in the action itself\n\u003e (not in the extra payload line), to specify how many times an update should be retried in the case of a version conflict.\n\nAnd:\n\n\u003e The `update` action payload supports the following options: `doc` (partial document), `upsert`, `doc_as_upsert`, `script`, `params` (for script),\n\u003e `lang` (for script), and `_source`. See update documentation for details on the options.\n\n[Both quotes are from the article linked above.]\n\n#### Update documentation\n\nThe update documentation is available here:\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html\n\n#### Reindex API\n\nIndices (database instances) are - for all practical purposes - immutable.\n\n[It's possible to add fields, and change the display formats of existing fields,\n but that's about it.]\n\nIn order to make any changes to the field mappings, it is pretty much required to reindex.\n\nThis can actually be pretty fast: with a medium-sized cluster, millions of documents can\n be reindexed in a matter of minutes.\n\nNote that the `destination` index must be set up before the reindex:\n\n    http://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html\n\n#### Deprecation logging\n\nThis should be enabled, especially if you are planning on upgrading Elasticsearch in the near future.\n\n\u003e In addition to regular logging, Elasticsearch allows you to enable logging of deprecated actions.\n\nfrom: http://www.elastic.co/guide/en/elasticsearch/reference/7.0/logging.html#deprecation-logging\n\n#### Upgrading Elasticsearch\n\nAWS has a pretty good synopsis of their process for upgrading Elasticsearch:\n\n    http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-version-migration.html\n\nElastic has a short summary also:\n\n    http://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html\n\n[It's not particularly helpful, although it does recommend taking a snapshot first.]\n\n#### AWS Developer Guide\n\nProbably definitive when working with [Amazon Elasticsearch Service](http://aws.amazon.com/elasticsearch-service/):\n\n    http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html\n\n#### PUT and POST\n\n\thttp://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html\n\n__Versioning__ is complicated; can also specify __timeouts__.\n\n## To Do\n\n- [x] Investigate [Beats](http://www.elastic.co/products/beats)\n- [ ] Investigate [Beats](http://www.elastic.co/products/beats) \u0026 [Logstash](http://www.elastic.co/products/logstash) (and how they differ)\n- [x] Investigate the X-Pack\n\n## Credits\n\nWhile most of this is from my painful experiences with wrangling Elasticsearch, I started from:\n\n\thttp://www.tutorialspoint.com/elasticsearch/elasticsearch_populate.htm\n\nAll of my testing was from the Kibana __Dev Tools__ console.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmramshaw%2Felastic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmramshaw%2Felastic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmramshaw%2Felastic/lists"}