{"id":13736290,"url":"https://github.com/ropensci/elastic","last_synced_at":"2025-05-16T14:04:43.628Z","repository":{"id":11832792,"uuid":"14388493","full_name":"ropensci/elastic","owner":"ropensci","description":"R client for the Elasticsearch HTTP API","archived":false,"fork":false,"pushed_at":"2023-11-05T19:56:01.000Z","size":3406,"stargazers_count":247,"open_issues_count":10,"forks_count":58,"subscribers_count":25,"default_branch":"master","last_synced_at":"2025-04-12T10:58:39.165Z","etag":null,"topics":["data-science","database","database-wrapper","elasticsearch","etl","http","json","r","r-package","rstats"],"latest_commit_sha":null,"homepage":"https://docs.ropensci.org/elastic","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ropensci.png","metadata":{"files":{"readme":"README.Rmd","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2013-11-14T07:48:08.000Z","updated_at":"2025-01-31T00:22:15.000Z","dependencies_parsed_at":"2024-01-03T01:20:25.961Z","dependency_job_id":"5109527f-a45f-400a-828f-62987ff48cd9","html_url":"https://github.com/ropensci/elastic","commit_stats":{"total_commits":832,"total_committers":20,"mean_commits":41.6,"dds":0.03605769230769229,"last_synced_commit":"89013defadae40e1e5ee7fea0a70109962d66a4d"},"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Felastic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Felastic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Felastic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ropensci%2Felastic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ropensci","download_url":"https://codeload.github.com/ropensci/elastic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253914289,"owners_count":21983461,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","database","database-wrapper","elasticsearch","etl","http","json","r","r-package","rstats"],"created_at":"2024-08-03T03:01:18.936Z","updated_at":"2025-05-16T14:04:43.595Z","avatar_url":"https://github.com/ropensci.png","language":"R","readme":"elastic\n=======\n\n```{r echo=FALSE}\nknitr::opts_chunk$set(\n  comment = \"#\u003e\",\n  collapse = TRUE,\n  warning = FALSE,\n  message = FALSE\n)\n```\n\n[![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)\n[![R-check](https://github.com/ropensci/elastic/workflows/R-check/badge.svg)](https://github.com/ropensci/elastic/actions?query=workflow%3AR-check)\n[![cran checks](https://cranchecks.info/badges/worst/elastic)](https://cranchecks.info/pkgs/elastic)\n[![rstudio mirror downloads](https://cranlogs.r-pkg.org/badges/elastic?color=E664A4)](https://github.com/r-hub/cranlogs.app)\n[![cran version](https://www.r-pkg.org/badges/version/elastic)](https://cran.r-project.org/package=elastic)\n\u003c!-- [![codecov.io](https://codecov.io/github/ropensci/elastic/coverage.svg?branch=master)](https://codecov.io/github/ropensci/elastic?branch=master) --\u003e\n\n**A general purpose R interface to [Elasticsearch](https://www.elastic.co/elasticsearch/)**\n\n\n## Elasticsearch info\n\n* [Elasticsearch home page](https://www.elastic.co/elasticsearch/)\n* [API docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)\n\n\n## Compatibility\n\nThis client is developed following the latest stable releases, currently `v7.10.0`. It is generally compatible with older versions of Elasticsearch. Unlike the [Python client](https://github.com/elastic/elasticsearch-py#compatibility), we try to keep as much compatibility as possible within a single version of this client, as that's an easier setup in R world.\n\n## Security\n\nYou're fine running ES locally on your machine, but be careful just throwing up ES on a server with a public IP address - make sure to think about security.\n\n* Elastic has paid products - but probably only applicable to enterprise users\n* DIY security - there are a variety of techniques for securing your Elasticsearch installation. A number of resources are collected in a [blog post](https://recology.info/2015/02/secure-elasticsearch/) - tools include putting your ES behind something like Nginx, putting basic auth on top of it, using https, etc.\n\n## Installation\n\nStable version from CRAN\n\n```{r eval=FALSE}\ninstall.packages(\"elastic\")\n```\n\nDevelopment version from GitHub\n\n```{r eval=FALSE}\nremotes::install_github(\"ropensci/elastic\")\n```\n\n```{r}\nlibrary('elastic')\n```\n\n## Install Elasticsearch\n\n* [Elasticsearch installation help](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html)\n\n__w/ Docker__\n\nPull the official elasticsearch image\n\n```\n# elasticsearch needs to have a version tag. We're pulling 7.10.1 here\ndocker pull elasticsearch:7.10.1\n```\n\nThen start up a container\n\n```\ndocker run -d -p 9200:9200 elasticsearch:7.10.1\n```\n\nThen elasticsearch should be available on port 9200, try `curl localhost:9200` and you should get the familiar message indicating ES is on.\n\nIf you're using boot2docker, you'll need to use the IP address in place of localhost. Get it by doing `boot2docker ip`.\n\n__on OSX__\n\n+ Download zip or tar file from Elasticsearch [see here for download](https://www.elastic.co/downloads), e.g., `curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.10.0-darwin-x86_64.tar.gz`\n+ Extract: `tar -zxvf elasticsearch-7.10.0-darwin-x86_64.tar.gz`\n+ Move it: `sudo mv elasticsearch-7.10.0 /usr/local`\n+ Navigate to /usr/local: `cd /usr/local`\n+ Delete symlinked `elasticsearch` directory: `rm -rf elasticsearch`\n+ Add shortcut: `sudo ln -s elasticsearch-7.10.0 elasticsearch` (replace version with your version)\n\nYou can also install via Homebrew: `brew install elasticsearch`\n\n\u003e Note: for the 1.6 and greater upgrades of Elasticsearch, they want you to have java 8 or greater. I downloaded Java 8 from here http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and it seemed to work great.\n\n## Upgrading Elasticsearch\n\nI am not totally clear on best practice here, but from what I understand, when you upgrade to a new version of Elasticsearch, place old `elasticsearch/data` and `elasticsearch/config` directories into the new installation (`elasticsearch/` dir). The new elasticsearch instance with replaced data and config directories should automatically update data to the new version and start working. Maybe if you use homebrew on a Mac to upgrade it takes care of this for you - not sure.\n\nObviously, upgrading Elasticsearch while keeping it running is a different thing ([some help here from Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html)).\n\n## Start Elasticsearch\n\n* Navigate to elasticsearch: `cd /usr/local/elasticsearch`\n* Start elasticsearch: `bin/elasticsearch`\n\nI create a little bash shortcut called `es` that does both of the above commands in one step (`cd /usr/local/elasticsearch \u0026\u0026 bin/elasticsearch`).\n\n## Initialization\n\nThe function `connect()` is used before doing anything else to set the connection details to your remote or local elasticsearch store. The details created by `connect()` are written to your options for the current session, and are used by `elastic` functions.\n\n```{r}\nx \u003c- connect(port = 9200)\n```\n\n\u003e If you're following along here with a local instance of Elasticsearch, you'll use `x` below to \ndo more stuff.\n\nFor AWS hosted elasticsearch, make sure to specify path = \"\" and the correct port - transport schema pair.\n\n```{r eval=FALSE}\nconnect(host = \u003caws_es_endpoint\u003e, path = \"\", port = 80, transport_schema  = \"http\")\n  # or\nconnect(host = \u003caws_es_endpoint\u003e, path = \"\", port = 443, transport_schema  = \"https\")\n```\n\nIf you are using Elastic Cloud or an installation with authentication (X-pack), make sure to specify path = \"\", user = \"\", pwd = \"\" and the correct port - transport schema pair.\n\n\n```r\nconnect(host = \u003cec_endpoint\u003e, path = \"\", user=\"test\", pwd = \"1234\", port = 9243, transport_schema  = \"https\")\n```\n\n\u003cbr\u003e\n\n## Get some data\n\nElasticsearch has a bulk load API to load data in fast. The format is pretty weird though. It's sort of JSON, but would pass no JSON linter. I include a few data sets in `elastic` so it's easy to get up and running, and so when you run examples in this package they'll actually run the same way (hopefully).\n\nI have prepare a non-exported function useful for preparing the weird format that Elasticsearch wants for bulk data loads, that is somewhat specific to PLOS data (See below), but you could modify for your purposes. See `make_bulk_plos()` and `make_bulk_gbif()` [here](https://github.com/ropensci/elastic/blob/master/R/docs_bulk.r).\n\n### Shakespeare data\n\nElasticsearch provides some data on Shakespeare plays. I've provided a subset of this data in this package. Get the path for the file specific to your machine:\n\n```{r echo=FALSE}\nlibrary(elastic)\nx \u003c- connect()\nif (x$es_ver() \u003c 600) {\n  shakespeare \u003c- system.file(\"examples\", \"shakespeare_data.json\", package = \"elastic\")\n} else {\n  shakespeare \u003c- system.file(\"examples\", \"shakespeare_data_.json\", package = \"elastic\")\n  shakespeare \u003c- type_remover(shakespeare)\n}\n```\n\n```{r eval=FALSE}\nshakespeare \u003c- system.file(\"examples\", \"shakespeare_data.json\", package = \"elastic\")\n# If you're on Elastic v6 or greater, use this one\nshakespeare \u003c- system.file(\"examples\", \"shakespeare_data_.json\", package = \"elastic\")\nshakespeare \u003c- type_remover(shakespeare)\n```\n\nThen load the data into Elasticsearch:\n\n\u003e make sure to create your connection object with `connect()`\n\n```{r eval=FALSE}\n# x \u003c- connect()  # do this now if you didn't do this above\ninvisible(docs_bulk(x, shakespeare))\n```\n\nIf you need some big data to play with, the shakespeare dataset is a good one to start with. You can get the whole thing and pop it into Elasticsearch (beware, may take up to 10 minutes or so.):\n\n```sh\ncurl -XGET https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json \u003e shakespeare.json\ncurl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json\n```\n\n### Public Library of Science (PLOS) data\n\nA dataset inluded in the `elastic` package is metadata for PLOS scholarly articles. Get the file path, then load:\n\n```{r}\nif (index_exists(x, \"plos\")) index_delete(x, \"plos\")\nplosdat \u003c- system.file(\"examples\", \"plos_data.json\", package = \"elastic\")\nplosdat \u003c- type_remover(plosdat)\ninvisible(docs_bulk(x, plosdat))\n```\n\n### Global Biodiversity Information Facility (GBIF) data\n\nA dataset inluded in the `elastic` package is data for GBIF species occurrence records. Get the file path, then load:\n\n```{r}\nif (index_exists(x, \"gbif\")) index_delete(x, \"gbif\")\ngbifdat \u003c- system.file(\"examples\", \"gbif_data.json\", package = \"elastic\")\ngbifdat \u003c- type_remover(gbifdat)\ninvisible(docs_bulk(x, gbifdat))\n```\n\nGBIF geo data with a coordinates element to allow `geo_shape` queries\n\n```{r}\nif (index_exists(x, \"gbifgeo\")) index_delete(x, \"gbifgeo\")\ngbifgeo \u003c- system.file(\"examples\", \"gbif_geo.json\", package = \"elastic\")\ngbifgeo \u003c- type_remover(gbifgeo)\ninvisible(docs_bulk(x, gbifgeo))\n```\n\n### More data sets\n\nThere are more datasets formatted for bulk loading in the `sckott/elastic_data` GitHub repository. Find it at \u003chttps://github.com/sckott/elastic_data\u003e\n\n\n\u003cbr\u003e\n\n## Search\n\nSearch the `plos` index and only return 1 result\n\n```{r}\nSearch(x, index = \"plos\", size = 1)$hits$hits\n```\n\nSearch the `plos` index, and query for _antibody_, limit to 1 result\n\n```{r}\nSearch(x, index = \"plos\", q = \"antibody\", size = 1)$hits$hits\n```\n\n## Get documents\n\nGet document with id=4\n\n```{r}\ndocs_get(x, index = 'plos', id = 4)\n```\n\nGet certain fields\n\n```{r}\ndocs_get(x, index = 'plos', id = 4, fields = 'id')\n```\n\n\n## Get multiple documents via the multiget API\n\nSame index and different document ids\n\n```{r}\ndocs_mget(x, index = \"plos\", id = 1:2)\n```\n\n## Parsing\n\nYou can optionally get back raw `json` from `Search()`, `docs_get()`, and `docs_mget()` setting parameter `raw=TRUE`.\n\nFor example:\n\n```{r}\n(out \u003c- docs_mget(x, index = \"plos\", id = 1:2, raw = TRUE))\n```\n\nThen parse\n\n```{r}\njsonlite::fromJSON(out)\n```\n\n## Known pain points\n\n* On secure Elasticsearch servers:\n  * `HEAD` requests don't seem to work, not sure why\n  * If you allow only `GET` requests, a number of functions that require\n  `POST` requests obviously then won't work. A big one is `Search()`, but\n  you can use `Search_uri()` to get around this, which uses `GET` instead\n  of `POST`, but you can't pass a more complicated query via the body\n\n## Screencast\n\nA screencast introducing the package: \u003ca href=\"https://vimeo.com/124659179\"\u003evimeo.com/124659179\u003c/a\u003e\n\n## Meta\n\n* Please [report any issues or bugs](https://github.com/ropensci/elastic/issues)\n* License: MIT\n* Get citation information for `elastic` in R doing `citation(package = 'elastic')`\n* Please note that this package is released with a [Contributor Code of Conduct](https://ropensci.org/code-of-conduct/). By contributing to this project, you agree to abide by its terms.\n","funding_links":[],"categories":["R","Database Management","Backend"],"sub_categories":["Database"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fropensci%2Felastic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fropensci%2Felastic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fropensci%2Felastic/lists"}