{"id":21558137,"url":"https://github.com/koopjs/koop-provider-s3-select","last_synced_at":"2025-03-18T03:28:10.623Z","repository":{"id":34203886,"uuid":"167218106","full_name":"koopjs/koop-provider-s3-select","owner":"koopjs","description":"Koop provider for S3 data","archived":false,"fork":false,"pushed_at":"2022-12-09T21:46:41.000Z","size":67,"stargazers_count":2,"open_issues_count":3,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-03-05T09:48:51.908Z","etag":null,"topics":["koop-provider"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/koopjs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-01-23T16:48:18.000Z","updated_at":"2022-07-18T13:29:37.000Z","dependencies_parsed_at":"2023-01-15T05:16:07.484Z","dependency_job_id":null,"html_url":"https://github.com/koopjs/koop-provider-s3-select","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-s3-select","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-s3-select/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-s3-select/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/koopjs%2Fkoop-provider-s3-select/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/koopjs","download_url":"https://codeload.github.com/koopjs/koop-provider-s3-select/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244149192,"owners_count":20406329,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["koop-provider"],"created_at":"2024-11-24T08:14:05.703Z","updated_at":"2025-03-18T03:28:10.603Z","avatar_url":"https://github.com/koopjs.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# @koopjs/provider-s3-select\n\nKoop provider for fetching data stored S3 with S3 Select. Files can be JSON, JSON lines, or CSVs. Note that CSV source files currently result in features with empty geometries.\n\n## Usage\n\n### Install\n\n```bash\nnpm install @koopjs/provider-s3-select\n```\n\n### Register\n\n```js\n// initialize Koop\nconst Koop = require('koop')\nconst koop = new Koop()\n\n// register\nconst s3SelectProvider = require('@koopjs/provider-s3-select')\nkoop.register(s3SelectProvider)\n```\n\n### Configuration\n\nThe S3 Select provider uses the [config](http://lorenwest.github.io/node-config/) module to store configuration settings in an object scoped to the `koopProviderS3Select` key.  The following JSON provides an example configuration:\n\n```javascript\n{\n  \"koopProviderS3Select\": {\n    \"s3Path\": \"my-default-bucket\", // default path\n    \"ttl\": 300, // The TTL option is measured in seconds, it will be used to set the cache expiration entry\n    \"stores\": {\n      \"my-store-key\": {\n        \"s3Path\": \"my-other-bucket/a-folder\", // specific path for this store\n        \"serialization\": {\n          \"JSON\": {\n            \"Type\": \"LINES\"\n          }  \n        }\n      }\n    }\n  }\n}\n```\n\n| key | type | description | required |\n| -- | -- | -- | -- |\n| `s3Path` | String | Default path-prefix that will be prepended to all requested files. Begins with the S3 bucket name and may include any sub-directory path. | No |\n| `ttl` | Integer | Number of seconds to cache the data | No |\n| `stores` | Object |key-value lookup used by the provider to fetch specific configuration information for a request. Each key in `stores` defines a valid value of the `:host` route parameter | Yes |\n| `stores[\u003ckey\u003e].s3Path`| String | Path-prefix that will be prepended to all file requests when `:host` is equal to `\u003ckey\u003e` | No |\n| `stores[\u003ckey\u003e].serialization`| Object | S3 input serialization object. Informs S3 how files requested with a give `:host` value should be deserialized. See [S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html).| No |\n\nAs noted above, `s3Path` is always optional, but if set,it should begin with the S3 bucket name and include any sub-directory path along which you want to restrict requests for fetching files. If `s3Path` is not set, the full S3 path to the file, including the bucket name, needs to be included in the request's `:id` parameter.  \n\n### AWS Credentials\n\nIn order to use the provider (and its S3 sdk), you will need to set your AWS credentials as environment variables.  See the [AWS docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) for details.\n\n### Request Parameters\nOnce the provider is registered with Koop, it will be available for all output services and will include the `:host` and `:id` route parameters.\n\n| parameter | type | desription |\n| -- | -- | -- |\n| `:host` | String | Points the provider to an entry in the `koopProviderS3Select.stores` config object. |\n| `:id` | String | The S3 path to a S3 file, with adjustments for any path prefix set in configuration with `s3Path`. Any forward slashes in the path should be replaced with `::` to avoid routing errors. |\n\nThe Koop output service provides an example of the usage of these parameters.  The pattern for the query service is:\n\n`/s3/:host/:id/FeatureServer/:layer/:method`\n\nAn example request using this pattern is below:\n\n`/s3/json-lines/path::to::json::file.json/FeatureServer/0/query`\n\nIn the above request, `json-lines` is the `:host` parameter and refers to a key in the configurations `stores` object. `path::to::json::file.json` is the `:id` parameter and is a partial path to an S3 file with all `/` replaced with `::`. Using the `s3Path` from the `json-lines` entry of the configuration object, the noted request would fetch the file with the following path `/my-other-bucket/a-folder/path/to/json/file.json`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoopjs%2Fkoop-provider-s3-select","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkoopjs%2Fkoop-provider-s3-select","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkoopjs%2Fkoop-provider-s3-select/lists"}