{"id":21558104,"url":"https://github.com/koopjs/koop-provider-elasticsearch","last_synced_at":"2025-07-02T15:34:04.622Z","repository":{"id":30967283,"uuid":"126526180","full_name":"koopjs/koop-provider-elasticsearch","owner":"koopjs","description":"A provider for koop that can connect to one or more elastic search instances and turn indices/aliases into individual feature services.","archived":false,"fork":false,"pushed_at":"2024-06-11T17:56:14.000Z","size":863,"stargazers_count":12,"open_issues_count":27,"forks_count":5,"subscribers_count":18,"default_branch":"master","last_synced_at":"2025-06-13T03:47:25.378Z","etag":null,"topics":["koop-provider"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koopjs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-03-23T18:43:26.000Z","updated_at":"2024-12-09T17:51:04.000Z","dependencies_parsed_at":"2024-06-11T21:28:38.430Z","dependency_job_id":"971863e6-d9fc-4eea-bb4d-75264fc45042","html_url":"https://github.com/koopjs/koop-provider-elasticsearch","commit_stats":{"total_commits":102,"total_committers":4,"mean_commits":25.5,"dds":"0.11764705882352944","last_synced_commit":"b6cb3395da219147f0bc0e26dcb7187ede3cd72b"},"previous_names":[],"tags_count":28,"template":false,"template_full_name":null,"purl":"pkg:github/koopjs/koop-provider-elasticsearch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elasticsearch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elasticsearch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elasticsearch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elasticsearch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koopjs","download_url":"https://codeload.github.com/koopjs/koop-provider-elasticsearch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elasticsearch/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263166327,"owners_count":23424135,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["koop-provider"],"created_at":"2024-11-24T08:13:58.934Z","updated_at":"2025-07-02T15:34:04.573Z","avatar_url":"https://github.com/koopjs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Koop Provider for Elastic Search\nThis provider allow Koop to fetch and transform data from an Elasticsearch instance.\n\n# Install\n\nFrom command line in the folder containing the project run:\n\n`npm install @koopjs/provider-elasticsearch --save`\n\nIn your Koop instance, register the provider like:\n\n```js\n  const esProvider = require('@koopjs/provider-elasticsearch')\n  koop.register(esProvider)\n```\n\nThe latest version of this provider can be installed with the Koop CLI.  See the [Koop CLI docs](https://github.com/koopjs/koop-cli) for details on setting up a Koop instance and add providers with the CLI.\n\n# Running\nTo suppress KoopJS warnings from the console output, run with an environment variable of KOOP_WARNINGS=\"suppress\". In powershell, this will look like: $env:KOOP_WARNINGS=\"suppress\" ; node main.js\n\n## Elastic Search Connect Environment Variable\nSetting an environment variable of `KOOP_ENV_KEYS` allows saving of usernames and passwords to connect to elastic search \nwithout having to store them in the configuration file.  The format for this variable is \n`esId,user,password||esId2,user2,password2`. The _esId_ is the id given to the connection in the configuration file as\nshown below in the *Basic Config File Structure* section.\n\n## Command Line\n`npm start`\n\n### Sub-Layers\nBy default, all services only have a single layer, as defined in the index configuration. It is possible to configure\none or more sub-layers in the subLayers section of the configuration. Each of these sub layers can be a predefined\nsub layer or you can create your own.\n\n\n\n#### Basic Config File Structure\nThe following is a sample configuration file showing most capabilities\n```json\n{\n  \"esConnections\": {\n    \"esNameForServiceURLs\": {\n      \"id\": \"esNameForServiceURLs\",\n      \"protocol\": \"http://\",\n      \"port\": 9200,\n      \"hosts\": [\n        \"localhost\"\n      ],\n      \"shapeIndices\": {\n        \"states\": {\n          \"geometryField\": \"geometry\",\n          \"geometryType\": \"Polygon\",\n          \"joinField\": \"NAME\"\n        }\n      },\n      \"indices\": {\n        \"myService1\": {\n          \"index\": \"indexName\",\n          \"allowMultiPoint\": false,\n          \"caching\": {\n            \"enabled\": true,\n            \"seconds\": 600\n          },\n          \"geometryField\": \"geometry.coordinates\",\n          \"geometryType\": \"geo_point\",\n          \"returnFields\": [\n            \"lastUpdate\",\n            \"createdAt\",\n            \"name\"\n          ],\n          \"dateFields\": [\n            \"lastUpdate\",\n            \"createdAt\"\n          ],\n          \"idField\": \"OBJECTID\",\n          \"subLayers\": [\n            {\"name\": \"geohash\"}\n          ],\n          \"maxResults\": 1000,\n          \"maxLayerInfoResults\": 1\n        },\n        \"tableService\": {\n          \"index\": \"indexNoShape\",\n          \"allowMultiPoint\": false,\n          \"isTable\": true,\n          \"returnFields\": [\n            \"state\",\n            \"county\",\n            \"date\"\n          ],\n          \"dateFields\": [\n            \"date\"\n          ],\n          \"maxResults\": 1000\n        },\n        \"joinService\": {\n          \"index\": \"indexToJoin\",\n          \"allowMultiPoint\": false,\n          \"returnFields\": [\n            \"date\",\n            \"country\",\n            \"state.name\"\n          ],\n          \"dateFields\": [\n            \"date\"\n          ],\n          \"subLayers\": [],\n          \"shapeIndex\": {\n            \"name\": \"states\",\n            \"joinField\": \"state.name\"\n          },\n          \"maxResults\": 1000\n        },\n        \"polyService\": {\n          \"index\": \"polygonIndex\",\n          \"allowMultiPoint\": false,\n          \"geometryField\": \"geometry\",\n          \"geometryType\": \"MultiPolygon\",\n          \"reversePolygons\": true,\n          \"returnFields\": [\n            \"properties.date\",\n            \"properties.count\",\n            \"properties.state_name\"\n          ],\n          \"dateFields\": [\n            \"properties.date\"\n          ],\n          \"subLayers\": [],\n          \"maxResults\": 1000\n        }\n      }\n    }\n  }\n}\n```\n\n##### Configuration Options\n* `shapeIndices` includes all indices that will be used for their shapes on other services. All fields are mandatory.\n* `indices` this object has a property for every service to be created.  The name of the property will be the service \nname.\n\n_Index Properties_\n* `index` is the name of the ElasticSearch index\n* `isTable` treat this service as a table, ignoring geometry\n* `allowMultiPoint` is only important for point services and allows more than one point per feature\n* `caching` allows local in-memory caching. Only use this for services that have no more documents than your _maxResults_ setting\n* `geometryField` is the full location of the geometry\n* `geometryType` can be geo_point, Point, MultiPoint, Polyline, MultiLineString, Polygon and MultiPolygon\n* `reversePolygons` if the stored polygons do not follow the right-hand rule setting this to true will fix this. \nPolygons that do not follow the right-hand rule will not be displayed as feature services without setting this to true.\n* `returnFields` includes all fields that will be returned by the feature service\n* `dateFields` includes any return fields that should be treated as dates\n* `idField` if the index includes a field that can be treated as the OBJECTID field, this should be set\n* `subLayers` any sub-layers to be used. Provided sub layers are `geohash_aggregation` and `geotile_aggregation` but custom subLayers may be used here as well.\n* `maxResults` the maximum features returned from a single request\n* `maxLayerInfoResults` maximum number of features to return for a layer info request (Normally 1) _NOTE:_ Do not use this on a layer with caching enabled.\n* `shapeIndex` if the service will join to a shapeIndex for geometry list the name of the index (defined in \n`shapeIndices`) and the joinField from this index.\n* `vectorLayerID` the 0 based index of the subLayer to use for vector output\n* `vectorStyle` an optional map-box style to use for the vector layer\n\n#### Sub-Layer Configurations\nAll subLayer configurations must include a `name` to map to a registered subLayer class. Other than that an `options` object contains\nany other needed information. \n\n* `offset` which is used by both provided subLayers is the offset value passed in by esri clients. This value is in meters and is larger for higher zoom levels.\n* `aggregationFields` can be used to pass in sub-aggregations to elastic search. The fields will be added to the output for display in pop-ups and/or for use in symbology.\n\n_GeoHash_\n\nprecision is between 1-12 for geohash.\n\n```json\n{\n  \"name\": \"geohash_aggregation\",\n  \"options\": {\n    \"tileConfig\": [\n      { \"precision\": 8, \"offset\": 16 },\n      { \"precision\": 6, \"offset\": 10000 },\n      { \"precision\": 4, \"offset\": 32000 },\n      { \"precision\": 2, \"offset\": 640000 }\n    ],\n    \"aggregationFields\": {\n      \"speed_avg\": {\n        \"avg\": { \"field\":  \"speed\"}\n      },\n      \"speed_min\": {\n        \"min\": { \"field\": \"speed\"}\n      }\n    }\n  }\n}\n```\n\n_GeoTile_\n\nprecision is between 0-29 for geotile.\n```json\n{\n  \"name\": \"geotile_aggregation\",\n  \"options\": {\n    \"tileConfig\": [\n      { \"precision\": 22, \"offset\": 16 },\n      { \"precision\": 16, \"offset\": 10000 },\n      { \"precision\": 10, \"offset\": 32000 },\n      { \"precision\": 4, \"offset\": 640000 }\n    ],\n    \"aggregationFields\": {\n      \"speed_avg\": {\n        \"avg\": { \"field\":  \"speed\"}\n      },\n      \"speed_min\": {\n        \"min\": { \"field\": \"speed\"}\n      }\n    }\n  }\n}\n```\n\n### BETA - OpenSearch Support\nSupport for connections to Amazon OpenSearch clusters has been added. This functionality is new and had not been\nthoroughly tested. Please log any issues encountered when using/testing.\n\n#### OpenSearch Configuration\nThe OpenSearch configurations follow the same pattern as elasticsearch connections and can even be included in the same\nconfiguration file. Create an `osConnections` object sibling to the `esConnections` object within the configuration\nfile. From there the configuration is the same as with elasticsearch connections except for cluster connection\ninformation.  All OpenSearch connection information is stored within a `connectConfig` object. An example is given below.\n\n```json\n{\n  \"osConnections\": {\n    \"osCluster1\": {\n      \"id\": \"osCluster1\",\n      \"connectConfig\": {\n        \"node\": \"http://localhost:9200\"\n      },\n      \"indices\": {\n        // same as with elastic configuration\n      }\n    }\n  }\n}\n```\n\n#### Additional Index Configurations\n`mapReturnValues` is an object that can contain keys that are field names that in turn have their own keys equal to field \nvalues mapped to object values. Example below. `__defaultmapping` is not required.\n\n``` js\n{\n    \"mapReturnValues\": {\n        \"fieldName\": {\n            \"returnedValue1\": \"mappedValue\",\n            \"__defaultmapping\": \"defaultValue\"\n        }\n    }\n}\n```\n\n*IN DEVELOPMENT*\n\n`mapFieldNames` can be used to specify a different return field than what is specified within returnFields\n```` json\n{\n    \"mapFieldNames\": {\n        \"fieldName\": \"mappedFieldName\"\n    }\n}\n````\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoopjs%2Fkoop-provider-elasticsearch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoopjs%2Fkoop-provider-elasticsearch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoopjs%2Fkoop-provider-elasticsearch/lists"}