{"id":21558140,"url":"https://github.com/koopjs/koop-provider-elastic-sql","last_synced_at":"2025-09-15T07:39:07.384Z","repository":{"id":199719724,"uuid":"703579882","full_name":"koopjs/koop-provider-elastic-sql","owner":"koopjs","description":"Koop provider to fetch data in elastic/opensearch datastores through the SQL API","archived":false,"fork":false,"pushed_at":"2023-10-19T13:58:23.000Z","size":706,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-08-05T06:31:56.082Z","etag":null,"topics":["koop","koop-provider"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koopjs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-11T14:02:44.000Z","updated_at":"2024-08-09T23:37:38.000Z","dependencies_parsed_at":"2023-10-11T23:18:52.822Z","dependency_job_id":"4a370629-2ad6-47ff-ac4e-dae44fb2ab51","html_url":"https://github.com/koopjs/koop-provider-elastic-sql","commit_stats":null,"previous_names":["koopjs/koop-provider-elastic-sql"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/koopjs/koop-provider-elastic-sql","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elastic-sql","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elastic-sql/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elastic-sql/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elastic-sql/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koopjs","download_url":"https://codeload.github.com/koopjs/koop-provider-elastic-sql/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-elastic-sql/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":275225862,"owners_count":25427000,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-15T02:00:09.272Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["koop","koop-provider"],"created_at":"2024-11-24T08:14:05.792Z","updated_at":"2025-09-15T07:39:07.338Z","avatar_url":"https://github.com/koopjs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# koop-provider-elastic-sql\nProvider to fetch data in  elastic/opensearch datastores through the SQL API.  This provider leverages the SQL API exclusively, and as such, it is limited by the capabilities of that API.\n\n## Data considerations\nIn order to gain the greatest efficiency from this provider, each document/record in _your dataset should have a unique-indentifier that is a property of the document/record_. This is important, because even though Elastic will create its own `_id` field as part of a document's metadata, _metadata fields are not accessible via the SQL API_.\n\nIf your data has a geometry and you want leverage Elastic's geo-filter capability (and you should), the geometry field needs to be mapped in your Elasticsearch Index as a [geopoint](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html) or [geoshape](https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-shape.html).\n\nA key aspect of working with Elastic is that its API will only return a maximum of 10000 records in any request. If your dataset has more than 10000 records and you request doesn't result in a set that is less than 10000 records, the results will be truncated. This limits the ability to paginate.  To reduce the risk of truncation you should always use a unique-identifier property, a mapped geometry field, and registered the provider with the index mappings for these fields (more on this below).\n\n## Usage\n\nRegister the provider with Koop:\n\n```js\nconst Koop = require('@koopjs/koop-core');\nconst koop = new Koop({ logLevel: 'info'});\nconst elasticSqlProvider = require('@koopjs/provider-elastic-sql');\n\nkoop.register(elasticSqlProvider, { conn, idFieldMap, geometryFieldMap });\n```\n\n### Registration parameters\n\n#### `conn` (required)\nAn object that contains connection details for the target Elastic cluster.  See the [Elastic documentation](https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/client-connecting.html) for details. \n\n#### `idFieldMap` (optional)\nAn key/value populated object that serves as a lookup/dictionary for the unique-identifier field of a given Elastic index.  For example, if your index is named \"my-index\", and the documents in the that index have a field called \"some-id-prop\" that can be used as a unique-identifier, then the `idFieldMap` would look like:\n\n```js\nconst idFieldMap = {\n  'my-index': 'some-id-prop'\n};\n```\n\nThe `idFieldMap` can have multiple key/value pairs if your Elastic cluster has multiple indicies.\n\nIf you omit the `idFieldMap`, filtering requested with the GeoService `objectsIds` parameter will not be executed by the Elastic SQL API.\n\n#### `geometryFieldMap` (optional)\nAn key/value populated object that serves as a lookup/dictionary for the geopoint/geoshape field of a given Elastic index.  For example, if your index is named \"my-index\", and the geometry of a document is stored in a geoshape field called \"the-geo-prop\", then the `geometryFieldMap` would look like:\n\n```js\nconst idFieldMap = {\n  'my-index': 'the-geo-prop'\n};\n```\n\nWhile elastic has no limit on the number of geo-fields per document, conversion of a document to GeoJSON requires that we identify a single field as the definitive geometry. The `geometryFieldMap` can have multiple key/value pairs if your Elastic cluster has multiple indicies.\n\nIf you omit the `geometryFieldMap`, the GeoJSON produced by the provider will not include a geometry.  In addition, geometry filtering will not be executed by the Elastic instance.\n\n### Route parameters\n\n#### `id`\nOnce registered with Koop, the provide will expose routes with an `id` parameter. For example:\n\n```sh\n/elastic-sql/rest/services/:id/FeatureServer\n```\n\nThe `id` parameter should be filled with the name of the Elastic index you are targeting.  So if you wished to query the `fires` document-index on you Elastic instance, you would make a request to:\n\n```sh\n/elastic-sql/rest/services/fires/FeatureServer/0/query\n```\n\n\n## Demo\n\nThe repository includes a demonstration project.  To run the demo you will need Docker installed on your computer. Once installed you can following the steps below to create a local Elastic instance and load it with sample data:\n\n```sh\n\u003e npm install\n\n\u003e cd demo\n\n# use Docker to run Elastic/Kibana\n\u003e docker-compose up -d\n\n# load sample data; this will create an Elastc index named \"fires\"\n\u003e node loader.js \n\n# start the Koop application\n\u003e node demo.js\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoopjs%2Fkoop-provider-elastic-sql","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoopjs%2Fkoop-provider-elastic-sql","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoopjs%2Fkoop-provider-elastic-sql/lists"}