Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/koopjs/koop-provider-s3-select
Koop provider for S3 data
https://github.com/koopjs/koop-provider-s3-select
koop-provider
Last synced: about 1 month ago
JSON representation
Koop provider for S3 data
- Host: GitHub
- URL: https://github.com/koopjs/koop-provider-s3-select
- Owner: koopjs
- Created: 2019-01-23T16:48:18.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2022-12-09T21:46:41.000Z (about 2 years ago)
- Last Synced: 2024-04-09T14:59:05.274Z (9 months ago)
- Topics: koop-provider
- Language: JavaScript
- Size: 65.4 KB
- Stars: 2
- Watchers: 12
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
Awesome Lists containing this project
README
# @koopjs/provider-s3-select
Koop provider for fetching data stored S3 with S3 Select. Files can be JSON, JSON lines, or CSVs. Note that CSV source files currently result in features with empty geometries.
## Usage
### Install
```bash
npm install @koopjs/provider-s3-select
```
### Register
```js
// initialize Koop
const Koop = require('koop')
const koop = new Koop()
// register
const s3SelectProvider = require('@koopjs/provider-s3-select')
koop.register(s3SelectProvider)
```
### Configuration
The S3 Select provider uses the [config](http://lorenwest.github.io/node-config/) module to store configuration settings in an object scoped to the `koopProviderS3Select` key. The following JSON provides an example configuration:
```javascript
{
"koopProviderS3Select": {
"s3Path": "my-default-bucket", // default path
"ttl": 300, // The TTL option is measured in seconds, it will be used to set the cache expiration entry
"stores": {
"my-store-key": {
"s3Path": "my-other-bucket/a-folder", // specific path for this store
"serialization": {
"JSON": {
"Type": "LINES"
}
}
}
}
}
}
```
| key | type | description | required |
| -- | -- | -- | -- |
| `s3Path` | String | Default path-prefix that will be prepended to all requested files. Begins with the S3 bucket name and may include any sub-directory path. | No |
| `ttl` | Integer | Number of seconds to cache the data | No |
| `stores` | Object |key-value lookup used by the provider to fetch specific configuration information for a request. Each key in `stores` defines a valid value of the `:host` route parameter | Yes |
| `stores[].s3Path`| String | Path-prefix that will be prepended to all file requests when `:host` is equal to `` | No |
| `stores[].serialization`| Object | S3 input serialization object. Informs S3 how files requested with a give `:host` value should be deserialized. See [S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html).| No |
As noted above, `s3Path` is always optional, but if set,it should begin with the S3 bucket name and include any sub-directory path along which you want to restrict requests for fetching files. If `s3Path` is not set, the full S3 path to the file, including the bucket name, needs to be included in the request's `:id` parameter.
### AWS Credentials
In order to use the provider (and its S3 sdk), you will need to set your AWS credentials as environment variables. See the [AWS docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) for details.
### Request Parameters
Once the provider is registered with Koop, it will be available for all output services and will include the `:host` and `:id` route parameters.
| parameter | type | desription |
| -- | -- | -- |
| `:host` | String | Points the provider to an entry in the `koopProviderS3Select.stores` config object. |
| `:id` | String | The S3 path to a S3 file, with adjustments for any path prefix set in configuration with `s3Path`. Any forward slashes in the path should be replaced with `::` to avoid routing errors. |
The Koop output service provides an example of the usage of these parameters. The pattern for the query service is:
`/s3/:host/:id/FeatureServer/:layer/:method`
An example request using this pattern is below:
`/s3/json-lines/path::to::json::file.json/FeatureServer/0/query`
In the above request, `json-lines` is the `:host` parameter and refers to a key in the configurations `stores` object. `path::to::json::file.json` is the `:id` parameter and is a partial path to an S3 file with all `/` replaced with `::`. Using the `s3Path` from the `json-lines` entry of the configuration object, the noted request would fetch the file with the following path `/my-other-bucket/a-folder/path/to/json/file.json`.