{"id":13735207,"url":"https://github.com/stac-utils/stac-server","last_synced_at":"2025-06-23T17:12:31.824Z","repository":{"id":38191357,"uuid":"226420934","full_name":"stac-utils/stac-server","owner":"stac-utils","description":"A Node-based STAC API, AWS Serverless, OpenSearch","archived":false,"fork":false,"pushed_at":"2024-05-14T13:47:33.000Z","size":8306,"stargazers_count":64,"open_issues_count":50,"forks_count":28,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-05-15T17:13:34.892Z","etag":null,"topics":["aws","opensearch","serverless"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/stac-utils.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-12-06T22:22:28.000Z","updated_at":"2024-05-29T22:28:56.591Z","dependencies_parsed_at":"2023-11-09T01:46:04.659Z","dependency_job_id":"5042b5d8-2861-483c-829a-2ad383bb488b","html_url":"https://github.com/stac-utils/stac-server","commit_stats":{"total_commits":1272,"total_committers":36,"mean_commits":"35.333333333333336","dds":0.529874213836478,"last_synced_commit":"6d41ad8d1b22f64a9c84f9f738faefb0aec6e68d"},"previous_names":[],"tags_count":50,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stac-utils%2Fstac-server","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stac-utils%2Fstac-server/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stac-utils%2Fstac-server/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/stac-utils%2Fstac-server/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/stac-utils","download_url":"https://codeload.github.com/stac-utils/stac-server/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224727173,"owners_count":17359532,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aws","opensearch","serverless"],"created_at":"2024-08-03T03:01:04.173Z","updated_at":"2025-05-08T11:32:38.264Z","avatar_url":"https://github.com/stac-utils.png","language":"JavaScript","funding_links":[],"categories":["`Python` processing of optical imagery (non deep learning)","Data catalogs"],"sub_categories":["Cloud Native Geospatial","Geodata catalogs"],"readme":"\u003c!-- omit from toc --\u003e\n\n# stac-server\n\n![push event badge](https://github.com/stac-utils/stac-server/workflows/Push%20Event/badge.svg)\n\n- [stac-server](#stac-server)\n - [Overview](#overview)\n - [Architecture](#architecture)\n - [Migration](#migration)\n - [Warnings](#warnings)\n - [4.1.0](#410)\n - [Thumbnails feature disabled by default](#thumbnails-feature-disabled-by-default)\n - [4.0.0](#400)\n - [Context Extension disabled by default](#context-extension-disabled-by-default)\n - [Node 22 update](#node-22-update)\n - [Hidden collections filter](#hidden-collections-filter)\n - [3.10.0](#3100)\n - [Node 20 update](#node-20-update)\n - [3.1.0](#310)\n - [OpenSearch Version 2.11](#opensearch-version-211)\n - [3.0.0](#300)\n - [Node 18 update](#node-18-update)\n - [2.4.0](#240)\n - [OpenSearch Version 2.9](#opensearch-version-29)\n - [2.3.0](#230)\n - [OpenSearch Version 2.7](#opensearch-version-27)\n - [0.x or 1.x -\\\u003e 2.x](#0x-or-1x---2x)\n - [Fine-grained Access Control](#fine-grained-access-control)\n - [Enabling Post-ingest SNS publishing](#enabling-post-ingest-sns-publishing)\n - [0.4.x -\\\u003e 0.5.x](#04x---05x)\n - [Elasticsearch to OpenSearch Migration](#elasticsearch-to-opensearch-migration)\n - [Preferred Elasticsearch to OpenSearch Migration Process](#preferred-elasticsearch-to-opensearch-migration-process)\n - [Granting Access for Thumbnails](#granting-access-for-thumbnails)\n - [0.3.x -\\\u003e 0.4.x](#03x---04x)\n - [Elasticsearch upgrade from 7.9 to 7.10](#elasticsearch-upgrade-from-79-to-710)\n - [Disable automatic index creation](#disable-automatic-index-creation)\n - [Validate index mappings](#validate-index-mappings)\n - [Usage](#usage)\n - [Deployment](#deployment)\n - [OpenSearch Configuration](#opensearch-configuration)\n - [Disable automatic index creation](#disable-automatic-index-creation-1)\n - [OpenSearch fine-grained access control](#opensearch-fine-grained-access-control)\n - [Option 1 - API method](#option-1---api-method)\n - [Option 2 - Dashboard method](#option-2---dashboard-method)\n - [Populating and accessing credentials](#populating-and-accessing-credentials)\n - [Create collection index](#create-collection-index)\n - [Proxying stac-server through CloudFront](#proxying-stac-server-through-cloudfront)\n - [Locking down transaction endpoints](#locking-down-transaction-endpoints)\n - [AWS WAF Rule Conflicts](#aws-waf-rule-conflicts)\n - [API Gateway Logging](#api-gateway-logging)\n - [Queryables](#queryables)\n - [Filter Extension](#filter-extension)\n - [Query Extension](#query-extension)\n - [Aggregation](#aggregation)\n - [Hidden collections filter for authorization](#hidden-collections-filter-for-authorization)\n - [Ingesting Data](#ingesting-data)\n - [Ingest actions](#ingest-actions)\n - [Ingesting large items](#ingesting-large-items)\n - [Subscribing to SNS Topics](#subscribing-to-sns-topics)\n - [Ingest Errors](#ingest-errors)\n - [Supporting Cross-cluster Search and Replication](#supporting-cross-cluster-search-and-replication)\n - [Cross-cluster Search](#cross-cluster-search)\n - [Cross-cluster Replication](#cross-cluster-replication)\n - [Pre- and Post-Hooks](#pre--and-post-hooks)\n - [Pre-Hook](#pre-hook)\n - [Post-Hook](#post-hook)\n - [Request Flow](#request-flow)\n - [Notes](#notes)\n - [Development](#development)\n - [Running Locally](#running-locally)\n - [Running Unit Tests](#running-unit-tests)\n - [Running System and Integration Tests](#running-system-and-integration-tests)\n - [Updating the OpenAPI specification](#updating-the-openapi-specification)\n - [About](#about)\n - [License](#license)\n\n## Overview\n\nStac-server is an implementation of the [STAC API specification](https://github.com/radiantearth/stac-api-spec) for searching and serving metadata for geospatial data, including but not limited to satellite imagery). The STAC and STAC API versions supported by a given version of stac-server are shown in the table below. Additional information can be found in the [CHANGELOG](CHANGELOG.md)\n\n| stac-server Version(s) | STAC Version | STAC API Foundation Version |\n| ---------------------- | ------------ | --------------------------- |\n| 0.1.x | 0.9.x | 0.9.x |\n| 0.2.x | \u003c1.0.0-rc.1 | 0.9.x |\n| 0.3.x | 1.0.0 | 1.0.0-beta.2 |\n| 0.4.x | 1.0.0 | 1.0.0-beta.5 |\n| 0.5.x-0.8.x | 1.0.0 | 1.0.0-rc.2 |\n| \u003e=1.0.0 | 1.0.0 | 1.0.0 |\n| \u003e=3.10.0 | 1.1.0 | 1.0.0 |\n\nCurrently, stac-server supports the following specifications:\n\n- STAC API - Core\n- STAC API - Features\n- STAC API - Collections\n- STAC API - Item Search\n- Context Extension (deprecated, disabled by default)\n- Sort Extension\n- Fields Extension\n- Query Extension\n- Filter Extension (conformance classes \"Basic CQL2\", \"CQL2 JSON\", \"Basic Spatial Functions\", and\n \"Basic Spatial Functions with additional Spatial Literals\", and\n the \"in\" and \"between\" predicates from \"Advanced Comparison Operators\")\n- Transaction Extension (disabled by default)\n- Aggregation Extension (experimental)\n\nThe following APIs are deployed instances of stac-server:\n\n| Name | STAC Version | STAC API Version | Description |\n| -------------------------------------------------------------------- | ------------ | ---------------- | ---------------------------------------- |\n| [Earth Search v1](https://earth-search.aws.element84.com/v1) | 1.0.0 | 1.0.0 | Catalog (v1) of some AWS Public Datasets |\n| [USGS Astrogeology STAC API](https://stac.astrogeology.usgs.gov/api) | 1.0.0 | 1.0.0 | A STAC API for planetary data |\n| [Earth Search v0](https://earth-search.aws.element84.com/v0) | 1.0.0-beta.2 | 0.9.0 | Catalog (v0) of some AWS Public Datasets |\n| [Landsat Look](https://landsatlook.usgs.gov/stac-server) | 1.0.0 | 0.9.0 | |\n\n## Architecture\n\n```mermaid\nflowchart LR\n\nitemsForIngest[Items for ingest]\n\nsubgraph ingest[Ingest]\n ingestSnsTopic[Ingest SNS Topic]\n ingestQueue[Ingest SQS Queue]\n ingestLambda[Ingest Lambda]\n postIngestSnsTopic[Post-Ingest SNS Topic]\n\n ingestDeadLetterQueue[Ingest Dead Letter Queue]\nend\n\nusers[Users]\n\nsubgraph api[STAC API]\n apiGateway[API Gateway]\n apiLambda[API Lambda]\nend\n\nopensearch[(OpenSearch)]\n\n%% Ingest workflow\n\nitemsForIngest --\u003e ingestSnsTopic\ningestSnsTopic --\u003e ingestQueue\ningestQueue --\u003e ingestLambda\ningestQueue --\u003e ingestDeadLetterQueue\ningestLambda --\u003e opensearch\ningestLambda --\u003e postIngestSnsTopic\n\n\n%% API workflow\n\nusers --\u003e api\napiGateway --\u003e apiLambda\napiLambda --\u003e opensearch\n\n```\n\n## Migration\n\n### Warnings\n\n- When upgrading to at least OpenSearch 2.7, there seems to be some low-level problem\n in the Lucene data storage that is a problem with indicies created in some but not all\n versions older\n than 2.7. Indicies created on the latest version in Fall of 2023 were not affected, but\n indices created is some previous version or versions are.\n After upgrading to 2.7, items may fail with the message reason \"cannot\n change field \\\"geometry\\\" from doc values type=NONE to inconsistent doc values\n type=BINARY\". There is no publicly-available information about this being a problem.\n The solution is to create a new index by creating a new collection with a different\n name, reindex the existing index into the newly-created index, delete and re-created\n the existing index by creating a collection, and reindex back into the index.\n\n### 4.1.0\n\n#### Thumbnails feature disabled by default\n\nThe thumbnails behavior is now disabled by default, and can be enabled with\n`ENABLE_THUMBNAILS` = `true`.\n\n### 4.0.0\n\n#### Context Extension disabled by default\n\nContext Extension is now disabled by default, and can be enabled with ENABLE_CONTEXT_EXTENSION\nenv var. Usage of the \"context\" object \"limit\", \"matched\", and \"returned\" fields can be replaced\nwith the root-level fields \"numberMatched\" and \"numberReturned\".\n\n#### Node 22 update\n\nThe default Lambda deployment environment is now Node 22.\n\nTo update the deployment to use Node 22, modify the serverless config file value\n`provider.runtime` to be `nodejs22.x` and the application re-deployed.\n\n#### Hidden collections filter\n\nTo all endpoints that depend on collections, there is now support for a query parameter\n(GET) or body field (POST) `_collections` that will filter to only those collections, but\nwill not reveal that in link contents. This is useful for the application of permissions\nto only certain collections.\n\n### 3.10.0\n\n#### Node 20 update\n\nThe default Lambda deployment environment is now Node 20. The major difference between\nthe Node 18 and Node 20 Lambda environment is the update of the underlying Linux version\nfrom Amazon Linux 2 to Amazon Linux 2023.\n\nTo update the deployment to use Node 20, modify the serverless config file value\n`provider.runtime` to be `nodejs20.x` and the application re-deployed.\n\n### 3.1.0\n\n#### OpenSearch Version 2.11\n\n- Update the `EngineVersion` setting in the serverless config file to `OpenSearch_2.11`\n and re-deploy\n\n### 3.0.0\n\n#### Node 18 update\n\nThe default Lambda deployment environment is now Node 18. The major difference between\nthe Node 16 and Node 18 Lambda environment is that the Node 16 env includes AWS SDK\nfor JS v2, and Node 18 includes v3. This code has been updated to use v3, so the\nNode 18 environment must be used, or the build must be modified to install the v3 libraries.\n\nTo update the deployment to use Node 18, modify the serverless config file value\n`provider.runtime` to be `nodejs18.x` and the application re-deployed.\n\n### 2.4.0\n\n#### OpenSearch Version 2.9\n\n- Update the `EngineVersion` setting in the serverless config file to `OpenSearch_2.9`\n and re-deploy\n\n### 2.3.0\n\n#### OpenSearch Version 2.7\n\n- Update the `EngineVersion` setting in the serverless config file to `OpenSearch_2.7`\n and re-deploy\n\n### 0.x or 1.x -\u003e 2.x\n\n#### Fine-grained Access Control\n\nAs of 2.0.0, only OpenSearch is supported and only using fine-grained access control.\nIt is recommended to follow the migration path to upgrade to fine-grained access control\nfirst and then upgrade to stac-server 2.x.\n\n#### Enabling Post-ingest SNS publishing\n\nstac-server now has the ability to publish all ingested entities (Items and Collections)\nto an SNS topic. Follow these steps to add this to an existing deployment. These\nconfigurations are also in the serverless.example.yml file, so reference that if it is\nunclear exactly where to add this in your config.\n\nThe following changes should be added to the serverless.yml file.\n\nExplicitly set the provider/environment setting for STAC_API_URL so the ingested entities\npublished to the topic will have their link hrefs set correctly. If this is not set,\nthe entities will still be published, with with incorrect link hrefs.\n\n```text\nSTAC_API_URL: \"https://some-stac-server.com\"\n```\n\nAdd the SNS topic resource:\n\n```text\npostIngestTopic:\n Type: AWS::SNS::Topic\n Properties:\n TopicName: ${self:service}-${self:provider.stage}-post-ingest\n```\n\nFor the `ingest` Lambda resource definition, configure the ARN to publish to by adding:\n\n```text\nenvironment:\n POST_INGEST_TOPIC_ARN: !Ref postIngestTopic\n```\n\nAdd IAM permissions with the statement:\n\n```text\n- Effect: Allow\n Action:\n - sns:Publish\n Resource:\n Fn::GetAtt: [postIngestTopic, TopicArn]\n```\n\n### 0.4.x -\u003e 0.5.x\n\n#### Elasticsearch to OpenSearch Migration\n\nBy default, a new deployment of 0.5.x will use OpenSearch instead of Elasticsearch. There\nare three options if you have an existing deployment that uses Elasticsearch:\n\n1. Use stac-server in compatibility mode\n 1. Add to serverless.yml environment variables `ES_COMPAT_MODE: \"true\"` and retain the\n existing Elasticsearch 7.10 resource description.\n2. Manage the Elasticsearch/OpenSearch domain outside the stac-server serverless deployment.\n 1. With the 0.4.x stac-server code, add `DeletionPolicy: Retain` to the `AWS::Elasticsearch::Domain` resource\n 2. Deploy the stack to update this property in the deployed CloudFormation Stack.\n 3. Remove the `AWS::Elasticsearch::Domain` resource from serverless.yml, modify all of the variables that were previously dynamically populated by the Elasticsearch resource values to be hard-coded, and re-deploy.\n 4. The Elasticsearch domain is now independent of the CF Stack.\n 5. With the 0.5.x stac-server code, update the serverless.yml environment variable `ES_COMPAT_MODE: \"true\"`\n 6. Deploy the 0.5.x stac-server code with the updated serverless.yml file\n 7. Through the AWS Console, upgrade the OpenSearch Service domain from Elasticsearch 7.10\n to OpenSearch 1.3, retaining the compatibilty mode enabled configuration.\n 8. Upgrade the OpenSearch 1.3 domain to OpenSearch 2.5.\n 9. Re-deploy the stack without the ES_COMPAT_MODE environment variable set.\n3. (Preferred) Disconnect the Elasticsearch domain from the stac-server CF Stack, deploy a new stac-server CF Stack,\n upgrade the Elasticsearch domain to OpenSearch, and connect the domain to the new CF Stack.\n This is described below.\n\nAdditionally, the `ES_HOST` variable used in the serverless.yml file has been\nrenamed `OPENSEARCH_HOST`.\n\n#### Preferred Elasticsearch to OpenSearch Migration Process\n\n**Note! The migration must be done carefully to avoid losing the database!**\n\nThe major part of this migration is the use of OpenSearch 2.5 instead of Elasticsearch\n7.10. Confusingly, both of these are options in the AWS OpenSearch Service, but the Elasticsearch option\nis no longer being updated by AWS in favor of OpenSearch.\n\nThe migration generally follows the outline in [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-opensearchservice-domain.html#aws-resource-opensearchservice-domain--remarks). The underlying problem being solved\nhere is that the CloudFormation resource AWS::Elasticsearch::Domain is used for Elasticsearch,\nbut AWS::OpenSearchService::Domain is used for OpenSearch, and a CloudFormation update\ncan't \"migrate\" between these resource types. So, the approach is to upgrade the domain\nto OpenSearch in compatibility mode, then clone the CloudFormation Stack, and import\nthe OpenSearch domain into it.\n\n1. With the 0.4.x codebase, change the serverless.yml file to add `DeletionPolicy: Retain` and `UpdateReplacePolicy: Retain` to the `AWS::Elasticsearch::Domain` definition at the same level as the `Type` and deploy. See instructions for deploying [here](https://github.com/stac-utils/stac-server/blob/main/README.md#deployment).\n\n```yaml\nType: AWS::Elasticsearch::Domain\nDeletionPolicy: Retain\nUpdateReplacePolicy: Retain\nProperties: . . .\n```\n\n2. The existing Elasticsearch domain must be manually migrated to OpenSearch. Prior to\n re-deploying the stack, use the AWS Console to manually upgrade the\n Elasticsearch domain (`Actions-\u003eUpgrade`) to OpenSearch 1.3. Select \"Enable\n compatibility mode\" to support the existing stac-server 0.4.x code using the Elasticsearch\n JavaScript client library (@elastic/elasticsearch version 7.9.0). After this upgrade to\n OpenSearch 1.3, then upgrade the domain to OpenSearch 2.5.\n\n3. Create a clone of the stac-server 0.5.x code. Copy and update the serverless.yml file used for the 0.4.0 deployment with these changes:\n\n- `ElasticSearchInstance` should be renamed to `OpenSearchInstance`\n - The `Type` of this resource should be changed from `AWS::Elasticsearch::Domain` to\n `AWS::OpenSearchService::Domain`\n - `ElasticsearchClusterConfig` is now `ClusterConfig`\n - `InstanceType` values have changed, e.g., t3.small.elasticsearch is now t3.small.search\n - `ElasticsearchVersion` is replaced with `EngineVersion` and set to `OpenSearch_2.5`\n- `EsEndpoint` should be renamed to `OpenSearchEndpoint` and the exported name suffixed\n with `-os-endpoint` instead of `-es-endpoint`\n- Environment variable `STAC_API_VERSION` should be removed to instead defer to the current default version\n\n- The `DomainName` value\n **must** remain the same as it is for the current deployment so\n the CloudFormation deployment will import the existing resource. Instead of a parameterized\n value of `${self:service}-${self:provider.stage}` as in the example serverless.yml file,\n it would have a hard-coded service name and `-es` suffix, e.g., `my-stac-server-${self:provider.stage}-es`.\n\n- Note: these changes can be checked against the [serverless.example.yml](https://github.com/stac-utils/stac-server/blob/main/serverless.example.yml) file.\n\n4. Run `npm run package` to generate the CloudFormation templates in the `.serverless` directory.\n Extract from the file `.serverless/cloudformation-template-update-stack.json` a template\n that only has the OpenSearchInstance resource in it. For example:\n\n```json\n{\n \"AWSTemplateFormatVersion\": \"2010-09-09\",\n \"Description\": \"A STAC API running on stac-server\",\n \"Resources\": {\n \"OpenSearchInstance\": {\n \"Type\": \"AWS::OpenSearchService::Domain\",\n \"DeletionPolicy\": \"Retain\",\n \"UpdateReplacePolicy\": \"Retain\",\n \"UpdatePolicy\": {\n \"EnableVersionUpgrade\": true\n },\n \"Properties\": {\n \"DomainName\": \"my-stac-server-dev-es\",\n \"EBSOptions\": {\n \"EBSEnabled\": true,\n \"VolumeType\": \"gp2\",\n \"VolumeSize\": 35\n },\n \"ClusterConfig\": {\n \"InstanceType\": \"t3.small.search\",\n \"InstanceCount\": 2,\n \"DedicatedMasterEnabled\": false,\n \"ZoneAwarenessEnabled\": true\n },\n \"EngineVersion\": \"OpenSearch_2.3\",\n \"DomainEndpointOptions\": {\n \"EnforceHTTPS\": true\n }\n }\n }\n }\n}\n```\n\n5. Within CloudFormation, choose `Create stack` and `With existing resources (import resources)`.\n Upload the template that contains only the OpenSearch resource. Choose a new stack name for this similar to the old one, e.g., `my-stac-server-2-{deploy-stage}` and update `service` name in the serverless.yml file with this name without the deploy stage e.g., `my-stac-server-2`. When prompted for the name of the OpenSearch Domain, put in the name of the existing one, e.g., `my-stac-server-dev-es`.\n\n6. Deploy the new stack with `npm run deploy -- --stage {deploy-stage}`. This should appear as an update to the CloudFormation stack that was just created manually, and should use the existing OpenSearch domain.\n\n7. Switch the DNS entry for the domain name to the API Gateway endpoint for the new Stack. See instructions [here](https://github.com/stac-utils/stac-server/blob/main/README.md#proxying-stac-server-through-cloudfront).\n\n8. Double-check that the `DeletionPolicy: Retain` is set on the old Stack for the Elasticsearch/OpenSearch resource, and then delete the old Stack.\n\n#### Granting Access for Thumbnails\n\nThe new experimental endpoint `/collections/{c_id}/items/{item_id}/thumbnail` will\nredirect to a URL providing a thumbnail as determined by the assets in an item. This is\nenabled only if `ENABLE_THUMBNAILS` is set to `true`. If the\nhref for this is an AWS S3 ARN, IAM permissions must be granted for the API Lambda to\ngenerate a pre-signed HTTP URL instead. For example:\n\n```yaml\n- Effect: Allow\n Action: s3:GetObject\n Resource: 'arn:aws:s3:::usgs-landsat/*'\n```\n\n### 0.3.x -\u003e 0.4.x\n\nCreate a new deployment, copy the elasticsearch database, and rename indexes.\n\n#### Elasticsearch upgrade from 7.9 to 7.10\n\nThe Serverless Framework supports provisioning AWS resources, but it does not support updating existing resources. In 0.4, the default Elasticsearch version has been updated from 7.9 to 7.10. Continuing to use 7.9 should not cause any problems, but it recommended that you manually upgrade to 7.10 by going to [AWS Console - Amazon OpenSearch Service](https://console.aws.amazon.com/esv3/home), choosing the Elasticsearch domain used by your stac-server deployment (e.g., stac-server-{stage}-es), choose Upgrade from the Actions menu, and then upgrade to Elasticsearch 7.10.\n\n#### Disable automatic index creation\n\nIt is now recommended to [disable automatic index creation](#disable-automatic-index-creation-1).\n\n#### Validate index mappings\n\nElasticsearch indices each have a mapping applied that determines how the data is indexed and searched over.\nThese mappings do not change the document data, but can change search behavior. One relevant mapping\nbehavior is that by default, string fields are analyzed for full-text search. In most cases with STAC Items,\nvalues such as those in the `id` and `collection` fields should not be analyzed and should instead be searchable only\nby exact matches. In Elasticsearch, this is known as a `keyword` field type. Importantly, sorting may only be done over `keyword` typed fields. As of 0.4.0, the default sort is now by `properties.datetime`, then `id`, then `collection`, and results will not be returnd if any indicies have the `id` or `collection` fields mapped as `text` instead of `keyword`.\n\nFor each index (other than `collections`), use GET to retrieve the endpoint `GET /{collectionId}/_mapping`, and\nvalidate that `properties.datetime` type is `date`, and `id` and `collection` mappings are `keyword` (not `text` with a `keyword` subfield). For an AWS Opensearch Service instance, this can be done with a script similar to the one [here](#disable-automatic-index-creation-1).\n\nThe results should look simliar to this:\n\n```json\n{\n \"my_collection_name\": {\n \"mappings\": {\n \"dynamic_templates\": [\n ...\n {\n \"strings\": {\n \"match_mapping_type\": \"string\",\n \"mapping\": {\n \"type\": \"keyword\"\n }\n }\n },\n ...\n ],\n \"properties\": {\n ....\n \"id\": {\n \"type\": \"keyword\"\n },\n \"collection\": {\n \"type\": \"keyword\"\n },\n ....\n \"properties\": {\n \"properties\": {\n ...\n \"datetime\": {\n \"type\": \"date\"\n },\n ...\n }\n },\n ...\n }\n }\n }\n}\n```\n\nIf this is not the case, the easiest solution to fix it is to:\n\n1. Deploy a 0.4.0 instance.\n2. Backup and restore the 0.3.0 instance's Elasticsearch indicies to the 0.4.0 instances's\n Elasticsearch database.\n3. Create a collection via ingest with a new collection name similar to the existing one (e.g., if index foo exists, create foo_new).\n4. Reindex from the the existing index (foo) to the the new one (foo_new).\n5. Delete the exiting index and rename the new one to the name of the formerly-existing one (e.g. foo_new -\u003e foo).\n\n## Usage\n\nStac-server is a web API that returns JSON, see the [documentation](http://stac-utils.github.io/stac-server), or the /api endpoint which is a self-documenting OpenAPI document. [STAC Index](https://stacindex.org) collects information on a number of [client tools](https://stacindex.org/ecosystem?category=Client).\n\nstac-server supports both GET and POST Search requests.\n\nAn Item Search with GET:\n\n```shell\ncurl \"${HOST}/search?collections=sentinel-2-l2a,sentinel-2-l1c\u0026bbox=10,10,15,15\u0026query=%7B%22eo%3Acloud_cover%22%3A%7B%22gte%22%3A0%2C%22lte%22%3A5%7D%7D\u0026filter=%7B%22op%22%3A%22%3C%22%2C%22args%22%3A%5B%7B%22property%22%3A%22view%3Asun_elevation%22%7D%2C50%5D%7D\u0026sortby=-properties.datetime\"\n```\n\nNotice that the `query` and `filter` parameters are URL-encoded JSON values.\n\nAn Item Search with POST:\n\n```shell\ncurl -X \"POST\" \"${HOST}/search\" \\\n -H 'Content-Type: application/json; charset=utf-8' \\\n -d $'{\n \"collections\": [\n \"sentinel-2-l2a\",\n \"sentinel-2-l1c\"\n ],\n \"bbox\": [\n 10,\n 10,\n 15,\n 15\n ],\n \"query\": {\n \"eo:cloud_cover\": {\n \"gte\": 0,\n \"lte\": 5\n }\n },\n \"filter\": {\n \"op\": \"\u003c\",\n \"args\": [\n \"property\": \"view:sun_elevation\"\n ],\n 50\n }\n \"sortby\": {\n \"field\": \"properties.datetime\",\n \"direction\": \"desc\"\n }\n}'\n```\n\n## Deployment\n\nThis repository contains Node libraries for running the API, along with a [serverless](https://serverless.com/) configuration file for deployment to AWS.\n\nTo create your own deployment of stac-server, first clone the repository:\n\n```shell\ngit clone https://github.com/stac-utils/stac-server.git\ncd stac-server\n```\n\nCopy the [example serverless config file](serverless.example.yml) to a file named `serverless.yml`:\n\n```shell\ncp serverless.example.yml serverless.yml\n```\n\nThere are some settings that should be reviewed and updated as needeed in the serverless config file, under provider-\u003eenvironment:\n\n| Name | Description | Default Value |\n| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |\n| OPENSEARCH_HOST | The URL of the OpenSearch cluster. | |\n| STAC_ID | ID of this catalog | stac-server |\n| STAC_TITLE | Title of this catalog | STAC API |\n| STAC_DESCRIPTION | Description of this catalog | A STAC API |\n| STAC_DOCS_URL | URL to documentation | [https://stac-utils.github.io/stac-server](https://stac-utils.github.io/stac-server) |\n| LOG_LEVEL | Level for logging (error, warn, info, http, verbose, debug, silly) | warn |\n| REQUEST_LOGGING_ENABLED | Express request logging enabled. String 'false' disables. | enabled |\n| REQUEST_LOGGING_FORMAT | Express request logging format to use. Any of the [Morgan predefined formats](https://github.com/expressjs/morgan#predefined-formats). | tiny |\n| STAC_API_URL | The root endpoint of this API | Inferred from request |\n| ENABLE_TRANSACTIONS_EXTENSION | Boolean specifying if the [Transaction Extension](https://github.com/radiantearth/stac-api-spec/tree/master/ogcapi-features/extensions/transaction) should be activated | false |\n| ENABLE_CONTEXT_EXTENSION | Boolean specifying if the [Context Extension](https://github.com/stac-api-extensions/context) should be activated | false |\n| STAC_API_ROOTPATH | The path to append to URLs if this is not deployed at the server root. For example, if the server is deployed without a custom domain name, it will have the stage name (e.g., dev) in the path. | \"\" |\n| PRE_HOOK | The name of a Lambda function to be called as the pre-hook. | none |\n| POST_HOOK | The name of a Lambda function to be called as the post-hook. | none |\n| ES_COMPAT_MODE | Enable Elasticsearch 7.10 compatibility mdoe within the server. | false |\n| OPENSEARCH_CREDENTIALS_SECRET_ID | The AWS Secrets Manager secret use for the username and password to authenticate to OpenSearch. | |\n| OPENSEARCH_USERNAME | The username to authenticate to OpenSearch when AWS Secrets Manager is not used. | |\n| OPENSEARCH_PASSWORD | The password to authenticate to OpenSearch when AWS Secrets Manager is not used. | |\n| COLLECTION_TO_INDEX_MAPPINGS | A JSON object representing collection id to index name mappings if they do not have the same names. | |\n| ITEMS_INDICIES_NUM_OF_SHARDS | Configure the number of shards for the indices that contain Items. | none |\n| ITEMS_INDICIES_NUM_OF_REPLICAS | Configure the number of replicas for the indices that contain Items. | none |\n| CORS_ORIGIN | Configure the value to send for the `Access-Control-Allow-Origin` CORS header. Should be set to the domain name of the UI if Basic Authentication is enable (e.g., `https://ui.example.com`). | `*` |\n| CORS_CREDENTIALS | Configure whether or not to send the `Access-Control-Allow-Credentials` CORS header. Header will be sent if set to `true`. | none |\n| CORS_METHODS | Configure whether or not to send the `Access-Control-Allow-Methods` CORS header. Expects a comma-delimited string, e.g., `GET,PUT,POST`. | `GET,HEAD,PUT,PATCH,POST,DELETE` |\n| CORS_HEADERS | Configure whether or not to send the `Access-Control-Allow-Headers` CORS header. Expects a comma-delimited string, e.g., `Content-Type,Authorization`. If not specified, defaults to reflecting the headers specified in the request’s `Access-Control-Request-Headers` header. | none |\n| ENABLE_COLLECTIONS_AUTHX | Enables support for hidden `_collections` query parameter / field when set to `true`. | none (not enabled) |\n| ENABLE_THUMBNAILS | Enables support for presigned thumbnails. | none (not enabled) |\n| ENABLE_INGEST_ACTION_TRUNCATE | Enables support for ingest action \"truncate\". | none (not enabled) |\n\nAdditionally, the credential for OpenSearch must be configured, as decribed in the\nsection [Populating and accessing credentials](#populating-and-accessing-credentials).\n\nAfter reviewing the settings, build and deploy:\n\n```shell\nnpm install\nnpm run build\nOPENSEARCH_MASTER_USER_PASSWORD='some-password' npm run deploy\n```\n\nThis will use the file `serverless.yml` and create a CloudFormation stack in the\n`us-west-2` region called `stac-server-dev`.\n\nAfter the initial deployment, the `MasterUserOptions` option in the serverless.yml file\ncan be commented out so that OPENSEARCH_MASTER_USER_PASSWORD does not need to be passed\nat every deployment.\n\nTo change the region or the stage name (from `dev`) provide arguments to the deploy command\n(note the additional `--` in the command, required by `npm` to provide arguments):\n\n```shell\nOPENSEARCH_MASTER_USER_PASSWORD='some-password' npm run deploy -- --stage mystage --region eu-central-1\n```\n\nMultiple deployments can be managed with multiple serverless config files and specified\nto the deploy command with:\n\n```shell\nnpm run deploy -- --config serverless.some-name.yml\n```\n\nOnce deployed, there are a few steps to configure OpenSearch.\n\n### OpenSearch Configuration\n\n#### Disable automatic index creation\n\nIt is recommended to disable the automatic index creation. This prevents the situation where\na group of Items are bulk indexed before the Collection in which they are contained has\nbeen created, and an OpenSearch index is created without the appropriate mappings.\n\nThis can either be done by calling the `/_cluster/settings` endpoint directly:\n\n```shell\ncurl -X \"PUT\" \"${HOST}/_cluster/settings\" \\\n -H 'Content-Type: application/json; charset=utf-8' \\\n -u \"admin:${OPENSEARCH_MASTER_USER_PASSWORD}\" \\\n -d '{\"persistent\": {\"action.auto_create_index\": \"false\"}}'\n```\n\nor setting that configuration via the OpenSearch Dashboard.\n\n#### OpenSearch fine-grained access control\n\nstac-server supports either fine-grained access control or AWS IAM authentication to\nOpenSearch. This section describes how to configure fine-grained access control.\n\n**Warning**: Unfortunately, fine-grained access control cannot be enabled on an\nexisting OpenSearch\ncluster through the serverless deploy, as this is a restriction of CloudFormation\nwhich serverless uses. A migration process between the clusters must be performed similar\nto the Elasticsearch -\u003e OpenSearch migration process.\n\nThe AccessPolicies Statement will restrict the OpenSearch instance to only being accessible\nwithin AWS. This requires the user creation steps below be either executed from or proxied\nthrough an EC2 instance, or that the Access Policy be changed temporarily through the\nconsole in the domain's Security configuration to be \"Only use fine-grained access control\".\n\nThe next step is to create the OpenSearch user and role to use for stac-server. This can\neither be done through the OpenSearch API or Dashboard.\n\n##### Option 1 - API method\n\nThis assumes the master username is `admin` and creates a user with the name `stac_server`.\nEnvironment variables `HOST` and `OPENSEARCH_MASTER_USER_PASSWORD` should be set in the\nshell environment.\n\nCreate the Role:\n\n```shell\ncurl -X \"PUT\" \"${HOST}/_plugins/_security/api/roles/stac_server_role\" \\\n -H 'Content-Type: application/json; charset=utf-8' \\\n -u \"admin:${OPENSEARCH_MASTER_USER_PASSWORD}\" \\\n -d $'{\n \"cluster_permissions\": [\n \"cluster_composite_ops\",\n \"cluster:monitor/health\"\n ],\n \"index_permissions\": [\n {\n \"index_patterns\": [\n \"*\"\n ],\n \"allowed_actions\": [\n \"indices_all\"\n ]\n }\n ],\n \"tenant_permissions\": [\n {\n \"tenant_patterns\": [\n \"global_tenant\"\n ],\n \"allowed_actions\": [\n \"kibana_all_read\"\n ]\n }\n ]\n}'\n\n```\n\nCreate the User:\n\n```shell\ncurl -X \"PUT\" \"${HOST}/_plugins/_security/api/internalusers/stac_server\" \\\n -H 'Content-Type: application/json; charset=utf-8' \\\n -u \"admin:${OPENSEARCH_MASTER_USER_PASSWORD}\" \\\n -d $'{ \"password\": \"xxx\" }'\n```\n\nDouble-check the response to ensure that the user was actually created!\n\nMap the Role to the User:\n\n```shell\ncurl -X \"PUT\" \"${HOST}/_plugins/_security/api/rolesmapping/stac_server_role\" \\\n -H 'Content-Type: application/json; charset=utf-8' \\\n -u \"admin:${OPENSEARCH_MASTER_USER_PASSWORD}\" \\\n -d $'{\n \"users\": [\n \"stac_server\"\n ]\n}'\n```\n\n##### Option 2 - Dashboard method\n\nLogin to the OpenSearch Dashboard with the master username (e.g. `admin`) and password.\nFrom the left sidebar menu, select \"Security\". Select \"Internal users\", and then \"Create\ninternal user\". Create the user with the name `stac_server`.\n\nClick \"Create New Role\". Create a new Role with name `stac_server_role` with:\n\n- Cluster permissions: `cluster:monitor/health`, `cluster_composite_ops`\n- Index permissions: `indices_all` on `*`\n- Tenant permissions: `global_tenant` Read only\n\nNote that several of the indices permissions in `cluster_composite_ops` action group\nare required to\nbe applyed to the Cluster permissions. Confusingly, the `cluster_all` action group does\nnot have those permissions in it because they are `indices` permissions rather than\n`cluster` permissions. This is all very confusing! [This issue](https://github.com/opensearch-project/security/issues/2336) has been filed against\nthe OpenSearch Security Plugin to request improvements to the documentation.\n\nAdd the user `stac_server` as a mapped user to this role.\n\n##### Populating and accessing credentials\n\nAfter you've created the users, you'll need to populate the credentials for the user\nso that stac-server can access them.\n\nThe preferred mechanism for populating the OpenSearch credentials to stac-server is to\ncreate a secret in AWS Secret Manager that contains the username and password. The\nrecommended name for this Secret corresponds\nto the stac-server deployment as `${service}-${stage}-opensearch-user-creds`, e.g.,\n`my-stac-server-dev-opensearch-user-creds`.\n\nThe Secret type should be \"Other type of secret\" and\nhave two keys, `username` and `password`, with the appropriate\nvalues, e.g., `stac_server` and whatever you set as the password when creating that user.\n\nAdd the `OPENSEARCH_CREDENTIALS_SECRET_ID` variable to the serverless.yml section\n`environment`:\n\n```yaml\nOPENSEARCH_CREDENTIALS_SECRET_ID: ${self:service}-${self:provider.stage}-opensearch-user-creds\n```\n\nAdd to the IAM Role Statements:\n\n```yaml\n- Effect: Allow\n Resource: arn:aws:secretsmanager:${aws:region}:${aws:accountId}:secret:${self:provider.environment.OPENSEARCH_CREDENTIALS_SECRET_ID}-*\n Action: 'secretsmanager:GetSecretValue'\n```\n\nIf desired, the resource ARN can be replaced with the exact ARN for the Secret instead of\nusing an ARN ending with `*`.\n\nRedeploy to reconfigure OpenSearch and populate the authentication configuration. The server\nshould now be using fine-grained access control.\n\nAlternately, instead of using the preferred mechanism of Secrets Manager,\nthe `OPENSEARCH_USERNAME` and `OPENSEARCH_PASSWORD` values can be set directly\nin the `environment` section:\n\n```yaml\nOPENSEARCH_USERNAME: stac_server\nOPENSEARCH_PASSWORD: xxxxxxxxxxx\n```\n\nSetting these as environment variables can also be useful when running stac-server\nlocally.\n\nstac-server is now ready to ingest data!\n\n#### Create collection index\n\nThe `collection` index must be created, which stores the metadata about each Collection.\nInvoke the `stac-server-\u003cstage\u003e-ingest` Lambda function with a payload of:\n\n```json\n{\n \"create_indices\": true\n}\n```\n\nThis can be done with the [AWS CLI Version 2](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).\n\n```shell\naws lambda invoke \\\n --function-name stac-server-dev-ingest \\\n --cli-binary-format raw-in-base64-out \\\n --payload '{ \"create_indices\": true }' \\\n /dev/stdout\n```\n\n### Proxying stac-server through CloudFront\n\nThe API Gateway URL associated with the deployed stac-server instance may not be the URL that you ultimately wish to expose to your API users. AWS CloudFront can be used to proxy to a more human readable URL. In order to accomplish this:\n\n1. Create a new CloudFront distribution (or use an existing distribution).\n2. Set the origin to the Gateway API URL (obtain in the stage view of the deployed stac-server). The URL is in the form `\u003c##abcde\u003e.execute-api.region.amazonaws.com`.\n3. Set the origin path to the deployed stage name prepended with a `/`, (e.g., /dev or /prod).\n4. Under behaviors, add a new behavior for the desired URL endpoint or subdomain (e.g., /api or /v0.4.0).\n5. Set the 'Origin and origin groups to the URL defined above ('`\u003c##abcde\u003e.execute-api.region.amazonaws.com`').\n6. Set Viewer to HTTPS only and Allowed HTTP Methods to 'GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE'.\n7. Set the Cache Policy to a custom policy that forwards query strings. If one simply disables caching, CloudFront strips the query strings.\n8. Optionally, define a LambdaEdge to perform a URL rewrite. This is necessary if your API URL is appended to the root URL (e.g., mydomain.com/api). The Lambda must rewrite the URL to remove the /api. For example:\n\n```python\nfrom re import sub\n\ndef lambda_handler(event, context):\n request = event['Records'][0]['cf']['request']\n uri = request[\"uri\"]\n\n if uri in [\"/\", \"/index.html\"]:\n response = {\n \"status\": 302,\n \"statusDescription\": \"Found\",\n \"headers\": {\n \"location\": [{\n \"key\": \"Location\",\n \"value\": \"/api/\"\n }]\n }\n }\n return response\n\n request[\"uri\"] = sub(\"^/api\", \"/\", uri)\n print(request)\n return request\n```\n\n### Locking down transaction endpoints\n\nIf you wanted to deploy STAC Server in a way which ensures certain endpoints have restricted access but others don't, you can deploy it into a VPC and add conditions that allow only certain IP addresses to access certain endpoints. Once you deploy STAC Server into a VPC, you can modify the Resource Policy of the API Gateway endpoint that gets deployed to restrict access to certain endpoints. Here is a hypothetical example. Assume that the account into which STAC Server is deployed is numbered 1234-5678-9123, the API ID is ab1c23def, and the region in which it is deployed is us-west-2. You might want to give the general public access to use any GET or POST endpoints with the API such as the \"/search\" endpoint, but lock down access to the transaction endpoints (see \u003chttps://github.com/radiantearth/stac-api-spec/tree/master/ogcapi-features/extensions/transaction\u003e) to only allow certain IP addresses to access them. These IP addresses can be, for example: 94.61.192.106, 204.176.50.129, and 11.27.65.78. In order to do this, you can impose a condition on the API Gateway that only allows API transactions such as adding, updating, and deleting STAC items from the whitelisted endpoints. For example, here is a Resource Policy containing two statements that allow this to happen:\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Statement\": [\n {\n \"Effect\": \"Allow\",\n \"Principal\": \"*\",\n \"Action\": \"execute-api:Invoke\",\n \"Resource\": [\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/POST/search\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/POST/search/*\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/GET/search/*\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23defi/v1/GET/*\"\n ]\n },\n {\n \"Effect\": \"Allow\",\n \"Principal\": \"*\",\n \"Action\": \"execute-api:Invoke\",\n \"Resource\": [\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/POST/collections/*/items\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/PUT/collections/*/items/*\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/PATCH/collections/*/items/*\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/DELETE/collections/*/items/*\"\n ],\n \"Condition\": {\n \"IpAddress\": {\n \"aws:sourceIp\": [\"94.61.192.106\", \"204.176.50.129\", \"11.27.65.78\"]\n }\n }\n }\n ]\n}\n```\n\nThe first statement in the Resource Policy above grants access to STAC API endpoints for use in general operations like searching, and the second statement restricts access to the Transaction endpoints to a set of source IP addresses. According to this policy, POST, PUT, PATCH, and DELETE operations on items within collections are only allowed if the request originates from the IP addresses 94.61.192.106, 204.176.50.129, or 11.27.65.78. The second statement can also be written in another manner, denying access to the Transaction endpoints for all addresses that don’t match a set of source IP addresses. This is shown below.\n\n```json\n{\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": \"execute-api:Invoke\",\n \"Resource\": [\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/POST/collections/*/items\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/PUT/collections/*/items/*\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/PATCH/collections/*/items/*\",\n \"arn:aws:execute-api:us-west-2:123456789123:ab1c23def/v1/DELETE/collections/*/items/*\"\n ],\n \"Condition\": {\n \"NotIpAddress\": {\n \"aws:sourceIp\": [\"94.61.192.106\", \"204.176.50.129\", \"11.27.65.78\"]\n }\n }\n}\n```\n\n### AWS WAF Rule Conflicts\n\nFrequently, stac-server is deployed with AWS WAF protection. When making a POST request\nthat only has the `limit` parameter in the body, a WAF SQL injection protection rule\nincurs a false positive and returns a Forbidden status code. This request is an example:\n\n```shell\ncurl -X POST ${HOST}/search -d '{\"limit\": 1}'\n```\n\nThis is also triggered when using pystac_client with no filtering parameters.\n\nThe fix is to disable the WAF SQL injection rule, which is unnecessary because\nstac-server does not use SQL.\n\n### API Gateway Logging\n\nThe example serverless.yml config contains disabled configuration for setting up\nAPI Gateway logging of API requests. More information about these configuration can be\nfound in the [Serverless Framework API Gateway Documentation](https://www.serverless.com/framework/docs/providers/aws/events/apigateway#logs).\n\nThe `executionLogging` setting causes logging of the actual execution of the API Gateway\nendpoints and backing Lambda, with `fullExecutionData` causing the entire request and\nresponse to be logged to CloudWatch, which can be expensive.\n\nThe `accessLogging` setting logs the values specified in `format` to CloudWatch, which\ncan be useful for computing metrics on usage for the API.\n\n## Queryables\n\nThe Filter Extension defines a\n[Queryables](https://docs.ogc.org/is/19-079r2/19-079r2.html#queryables) resource for\ndiscovering properties that may be used to construct filter expressions. Queryables for\neach Collection are served from the `/collections/{collectionId}/queryables` endpoint.\nRoot-level (global to all Collections) queryables are served from the `/queryables`\nendpoint.\n\nCollection queryables are defined in stac-server by adding a `queryables` field to the\nCollection JSON object with the value being the JSON Schema definition of the queryables\nfor that Collection. The content of this `queryables` field is extracted from the\nCollection object and served from the Collection's queryables endpoint, but is removed\nfrom the Collection object when the Collection itself is served from the\n`/collections/{collectionId}` endpoint. Stac-server's root-level queryables resource is\nnot configurable and currently does not advertise any queryable properties. Likewise, if a\nCollection does not define a `queryables` field, no queryable properties are advertised\nfor that Collection. For reference, here is a queryables JSON Schema definition that does\nnot advertise any queryables properties (note the empty `properties` field):\n\n```json\n{\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://stac-api.example.com/queryables\",\n \"type\": \"object\",\n \"title\": \"Queryables for Example STAC API\",\n \"description\": \"Queryable names for the example STAC API Item Search filter.\",\n \"properties\": {},\n \"additionalProperties\": true\n}\n```\n\n### Filter Extension\n\nStac-server currently implements the Filter Extension such that the `id`, `collection`,\n`bbox`, and `geometry` Item fields and all fields in the `properties` object of an Item\nare always available as filter terms for a Collection, regardless of whether a Collection\ndefines a `queryables` field or not. This behavior aligns with a value of `true` for the\n`additionalProperties` field in the queryables definition. Stac-server currently\n**requires** `additionalProperties` to be `true`; a value of `false`, which would restrict\nfiltering to only those `properties` defined in a Collection's queryables schema, is not\nsupported and will raise an error. Thus, adding a `queryables` field to a Collection is\ninformative only - it does not change the behavior of a filter.\n\nNote that when creating a filter expression that uses fields from the `properties` object\nin an Item, the fields **must not** be prefixed, e.g., use `eo:cloud_cover` instead of\n`properties.eo:cloud_cover` in the filter expression. Care must be taken that terms used\nin a filter expression exactly match the field names in the Item `properties` object;\nthere is no validation that filter expression terms are correct, so if you search for a\nfield that doesn't exist in an Item's `properties` object, that Item will never be\nmatched.\n\nThe`id`, `collection`, `bbox`, and `geometry` fields also **must not** be prefixed since\nthey are top-level Item fields. Note that until the [Basic Spatial Functions with\nadditional Spatial\nLiterals](https://docs.ogc.org/is/21-065r2/21-065r2.html#rc_basic-spatial-functions-plus)\nand [Array\nFunctions](https://docs.ogc.org/is/21-065r2/21-065r2.html#_conformance_class_array_functions)\nconformance classes are implemented, searching over the `bbox` and `geometry` fields is\nnot supported.\n\n### Query Extension\n\nUnlike the Filter Extension, the Query Extension does not (yet) define a mechanism to\nadvertise which terms may be used in expressions. However, an optional definition may be\nadded to it soon that defines queryables endpoints the same as used with Filter Extension.\n\n## Aggregation\n\nStac-server supports the [Aggregation Extension](https://github.com/stac-api-extensions/aggregation). This allows the definition of per-collection aggregations that can be\ncalculated, dependent on the relevant fields being available in the STAC Items in that\nCollection. A field named `aggregations` should be added to the Collection object for\nthe collection for which the aggregations are available, e.g.:\n\n```text\n \"aggregations\": [\n {\n \"name\": \"total_count\",\n \"data_type\": \"integer\"\n },\n {\n \"name\": \"datetime_max\",\n \"data_type\": \"datetime\"\n },\n {\n \"name\": \"datetime_min\",\n \"data_type\": \"datetime\"\n },\n {\n \"name\": \"datetime_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"datetime\"\n },\n {\n \"name\": \"grid_code_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"string\"\n },\n {\n \"name\": \"centroid_geohash_grid_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"string\"\n },\n {\n \"name\": \"centroid_geohex_grid_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"string\"\n },\n {\n \"name\": \"centroid_geotile_grid_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"string\"\n },\n {\n \"name\": \"geometry_geohash_grid_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"string\"\n },\n {\n \"name\": \"geometry_geotile_grid_frequency\",\n \"data_type\": \"frequency_distribution\",\n \"frequency_distribution_data_type\": \"string\"\n }\n ]\n```\n\nAvailable aggregations are:\n\n- total_count (count of total items)\n- collection_frequency (Item `collection` field)\n- platform_frequency (Item.Properties.platform)\n- cloud_cover_frequency (Item.Properties.eo:cloud_cover)\n- datetime_frequency (Item.Properties.datetime, monthly interval)\n- datetime_min (earliest Item.Properties.datetime)\n- datetime_max (latest Item.Properties.datetime)\n- sun_elevation_frequency (Item.Properties.view:sun_elevation)\n- sun_azimuth_frequency (Item.Properties.view:sun_azimuth)\n- off_nadir_frequency (Item.Properties.view:off_nadir)\n- grid_code_frequency (Item.Properties.grid:code)\n- grid_geohash_frequency ([geohash grid](https://opensearch.org/docs/latest/aggregations/bucket/geohash-grid/) on Item.Properties.proj:centroid) (Deprecated)\n- grid_geohex_frequency ([geohex grid](https://opensearch.org/docs/latest/aggregations/bucket/geohex-grid/) on Item.Properties.proj:centroid) (Deprecated)\n- grid_geotile_frequency ([geotile grid](https://opensearch.org/docs/latest/aggregations/bucket/geotile-grid/) on Item.Properties.proj:centroid) (Deprecated)\n- centroid_geohash_grid_frequency ([geohash grid](https://opensearch.org/docs/latest/aggregations/bucket/geohash-grid/) on Item.Properties.proj:centroid)\n- centroid_geohex_grid_frequency ([geohex grid](https://opensearch.org/docs/latest/aggregations/bucket/geohex-grid/) on Item.Properties.proj:centroid)\n- centroid_geotile_grid_frequency (geotile on Item.Properties.proj:centroid)\n- geometry_geohash_grid_frequency ([geohash grid](https://opensearch.org/docs/latest/aggregations/bucket/geohash-grid/) on Item.geometry)\n- geometry_geotile_grid_frequency ([geotile grid](https://opensearch.org/docs/latest/aggregations/bucket/geotile-grid/) on Item.geometry)\n\n## Hidden collections filter for authorization\n\nAll endpoints that involve the use of Collections support the use of a \"hidden\" query\nparameter named (for GET requests) or body JSON field (for POST requests) named\n`_collections` that can be used by an authorization proxy (e.g., a pre-hook Lambda)\nto filter the collections a user has access to. This parameter/field will be excluded\nfrom pagination links, so it does not need to be removed on egress.\n\nThis feature must be enabled with the `ENABLE_COLLECTIONS_AUTHX` configuration.\n\nThe endpoints this applies to are:\n\n- /collections\n- /collections/:collectionId\n- /collections/:collectionId/queryables\n- /collections/:collectionId/aggregations\n- /collections/:collectionId/aggregate\n- /collections/:collectionId/items\n- /collections/:collectionId/items/:itemId\n- /collections/:collectionId/items/:itemId/thumbnail\n- /search\n- /aggregate\n\nThe five endpoints of the Transaction Extension do not use this parameter, as there are\nother authorization considerations for these, that are left as future work.\n\nIf this behavior is enabled and a `_collections` parameter is not passed or is passed\nwith an empty string or empty list, the caller will not have access to any collections.\nWhen `*` is included in the list of collections (ideally as the only value), the caller\nwill have access to all collections.\n\n## Ingesting Data\n\nSTAC Collections and Items are ingested by the `ingest` Lambda function, however this Lambda is not invoked directly by a user, it consumes records from the `stac-server-\u003cstage\u003e-queue` SQS. To add STAC Items or Collections to the queue, publish them to the SNS Topic `stac-server-\u003cstage\u003e-ingest`.\n\n**STAC Collections must be ingested before Items that belong to that Collection.** Items should have the `collection` field populated with the ID of an existing Collection. If an Item is ingested before ingestion of the Collection it contains,\ningestion will either fail (in the case of a single Item ingest) or if auto-creation of indexes is enabled (default) and multiple Items are ingested in bulk, the auto-created index will have incorrect mappings.\n\nIf a collection or item is ingested, and an item with that id already exists in STAC, the new item will completely replace the old item, except the `created` property will be retained and the `updated` property updated\nto match the time of the new update.\n\nAfter a collection or item is ingested, the status of the ingest (success or failure) along with details of the collection or item are sent to a post-ingest SNS topic. To take action on items after they are ingested subscribe an endpoint to this topic.\n\nMessages published to the post-ingest SNS topic include the following atributes that can be used for filtering:\n\n| attribute | type | values |\n| ------------ | ------ | ------------------------ |\n| recordType | String | `Collection` or `Item` |\n| ingestStatus | String | `successful` or `failed` |\n| collection | String | |\n\n### Ingest actions\n\nIn addition to ingesting Item and Collection JSON, the ingestion pipeline can also execute\nactions. This is useful when the Transaction Extension is turned off, but data modification\nstill needs to occur.\n\nThere is currently only one action implemented, `truncate`. This action will delete all\nof the Items in the specified Collection, but keep the Collection.\n\nThis action is enabled by setting `ENABLE_INGEST_ACTION_TRUNCATE` to `true`. For example,\na deployment may want to enable this in dev and staging, but disable in prod so you don't\naccidentally delete all the items from a production collection.\n\nThe form of this message is:\n\n```json\n{\n \"type\": \"action\",\n \"command\": \"truncate\",\n \"collection\": \"my_collection_1\"\n}\n```\n\n### Ingesting large items\n\nThere is a 256 KB limit on the size of SQS messages. Larger items can by publishing a message to the `stac-server-\u003cstage\u003e-ingest` SNS topic in with the format:\n\n```json\n{\n \"href\": \"s3://source-bucket/source-key\"\n}\n```\n\nThe `s3://`, `http://`, and `https://` protocols are supported for remote ingest.\n\n### Subscribing to SNS Topics\n\nStac-server can also be subscribed to SNS Topics that publish complete STAC Items as their message. This provides a way to keep stac-server up to date with new data. Use the AWS Lambda console for the function `stac-server-\u003cstage\u003e-subscibe-to-sns` to subscribe to an SNS Topic for which you have the full ARN and permission to subscribe to. This could be an SNS Topic you created yourself to publish STAC records to, or a publicly available one, such as for [Sentinel](https://github.com/sat-utils/sat-stac-sentinel).\n\n_Note_, that adding the subscription via the topic page does not seem to work. Instead, add a trigger on Lambda edit page.\n\n### Ingest Errors\n\nErrors that occur while consuming items from the ingest queue will end up in the dead letter processing queue.\n\n## Supporting Cross-cluster Search and Replication\n\nOpenSearch support cross-cluster connections that can be configured to either allow search\nacross the clusters, treating a remote cluster as if it were another group of nodes in the\ncluster, or configure indicies to be replicated (continuously copied) from from one\ncluster to another.\n\nConfiguring either cross-cluster behavior requires fine-grained access control.\n\n### Cross-cluster Search\n\nThe AWS documentation for cross-cluster search can be found\n[here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/cross-cluster-search.html).\n\n1. Ensure fine-grained access control is enabled.\n2. Create a connection between the source and destination OpenSearch domains.\n3. Ensure there is a `es:ESCrossClusterGet` action in the destination's access policy.\n4. In the source stac-server, create a Collection for each collection to be mapped. This\n must have the same id as the destination collection.\n5. For the source stac-server, configure a `COLLECTION_TO_INDEX_MAPPINGS`\n environment variable with a stringified JSON object mapping the collection name to the\n name of the index. For example, `{\"collection1\": \"cluster2:collection1\", \"collection2\": \"cluster2:collection2\"}` is a value mapping two collections through a\n connection named `cluster2`. Deploy this change.\n\n### Cross-cluster Replication\n\nThe AWS documentation for cross-cluster replication can be found\n[here](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/replication.html).\n\n1. Ensure fine-grained access control is enabled (default as of v2.0.0)\n2. Create the replication connection in the source to the destination\n3. Create the collection in the source's stac-server instance\n\n## Pre- and Post-Hooks\n\nStac-server supports two hooks into the request process: a pre-hook and a post-hook. These are each Lambda functions which, if configured, will be invoked by stac-server. It is assumed that the stac-server Lambda has been granted permission to invoke these Lambda functions, if configured.\n\n### Pre-Hook\n\nIf the stac-server is deployed with the `PRE_HOOK` environment variable set to the name of a Lambda function, then that function will be called as the pre-hook.\n\nThe event passed into the pre-hook Lambda will be an instance of an [API Gateway Proxy Event](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format).\n\nIf the return value from the pre-hook Lambda is an instance of an [API Gateway Proxy Result](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format), then that response will immediately be returned to the client.\n\nIf the return value of the pre-hook Lambda is an instance of an [API Gateway Proxy Event](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format), then that event will be passed along to stac-server.\n\nIf the pre-hook Lambda throws an exception, an internal server error will be returned to the client.\n\nThe pre-hook Lambda configuration may reference any Lambda, not only one deployed as part\nof this stack. There is an example pre-hook Lambda that can be included with this stack,\nwhich provides an example rudimentary authorization mechanism via a hard-coded token.\n\nTo enable this example pre-hook:\n\n- Either (1) in package.json, pass the env var `BUILD_PRE_HOOK=true` in the `build`\n command, or (2) modify bin/build.sh to always build the \"pre-hook\" package.\n- In the serverless.yml file, uncomment the `preHook` function, the `preHook` IAM\n permissions, and the environment variables `PRE_HOOK` and `API_KEYS_SECRET_ID`\n- Create a Secrets Manager secret with the name used in `API_KEYS_SECRET_ID` with\n the keys as the strings allowed for API Keys and the values as an array `[\"write\"]`.\n- Build and deploy.\n\n### Post-Hook\n\nIf the stac-server is deployed with the `POST_HOOK` environment variable set to the name of a Lambda function, then that function will be called as the post-hook.\n\nThe event passed into the post-hook labmda will be the response from the stac-server, and will be an instance of an [API Gateway Proxy Result](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format).\n\nThe return value of the post-hook Lambda must be an instance of an [API Gateway Proxy Result](https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format).\n\nIf the post-hook Lambda throws an exception, an internal server error will be returned to the client.\n\nThe post-hook Lambda configuration may reference any Lambda, not only one deployed as part\nof this stack. There is an example post-hook Lambda that can be included with this stack,\nwhich does nothing, but shows how the API Lambda response can be modified.\n\nThe post-hook Lambda configuration may reference any Lambda, not only one deployed as part\nof this stack. There is an example post-hook Lambda that can be included with this stack,\nwhich provides an example of how to interact with the response, but does not modify it.\n\nTo enable this example post-hook:\n\n- Modify bin/build.sh to not exclude the \"post-hook\" package from being built.\n- In the serverless.yml file, uncomment the `postHook` function and the `postHook`\n IAM permissions.\n- Build and deploy.\n\n### Request Flow\n\n```mermaid\nflowchart\n client -- APIGatewayProxyEvent --\u003e pre-hook\n pre-hook[pre-hook Lambda]\n pre-hook -- APIGatewayProxyResult --\u003e client\n pre-hook -- APIGatewayProxyEvent --\u003e stac-server\n post-hook[post-hook Lambda]\n stac-server -- APIGatewayProxyResult --\u003e post-hook\n post-hook -- APIGatewayProxyResult --\u003e client\n```\n\n### Notes\n\nLambda payloads and responses [must be less than 6 MB](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution). A larger payload will result in an internal server error being returned to the client.\n\nThe outputs of the pre- and post-hooks are validated and, if they don't comply with the defined schemas, an internal server error will be returned to the client. Information about the invalid event, as well as details about the parsing errors, will be logged to CloudWatch.\n\n## Development\n\nInstall [NVM](https://github.com/nvm-sh/nvm) to manage your Node.js environment.\n\n```shell\n# uses version in .nvmrc\nnvm install\nnvm use\n```\n\nThe package-lock.json was built with npm 8.5.0, so use at least this version.\n\nThere are several useful npm commands available to use locally:\n\n```shell\n# Install dependencies in package.json\nnpm install\n\n# Run the build command in each of the packages (runs webpack)\nnpm run build\n\n# Run ESLint\nnpm run lint\n\n# To run both unit and system tests (requires running docker compose containers)\nnpm run test\n\n# To build API docs from the api spec\nnpm run build-api-docs # TODO: this fails\n```\n\n[npm-check-updates](https://www.npmjs.com/package/npm-check-updates) can be used for\nupdating version dependencies to newer ones. Run:\n\n```commandline\nncu -i\n```\n\n### Running Locally\n\nBefore the API can be run, OpenSearch and Localstack need to be running. There is a `compose.yml` file to simplify running OpenSearch locally:\n\n```shell\ndocker compose up -d\n```\n\nThe API can then be run with:\n\n```shell\nnpm run serve\n```\n\nConnect to the server on \u003chttp://localhost:3000/\u003e\n\nOther configurations can be passed as shell environment variables, e.g.,\n\n```shell\nexport ENABLE_TRANSACTIONS_EXTENSION=true\nexport OPENSEARCH_HOST='https://search-stac-server-dev-7awl6h344qlpvly.us-west-2.es.amazonaws.com'\nnpm run serve\n```\n\n### Running Unit Tests\n\nstac-server uses [ava](https://github.com/avajs/ava) to execute tests.\n\n```shell\n# alias to run unit and system tests\nnpm test\n\n# run unit tests in tests directory\nnpm run test:unit\n\n# run unit tests with coverage\nnpm run test:unit:coverage\n\n# run tests from a single test file whose titles match 'foobar*'\nnpx ava tests/test-es.js --match='foobar*'\n```\n\n### Running System and Integration Tests\n\nThe System and Integration tests use an OpenSearch server running in Docker and a local instance of the API.\n\nWhen the system tests run, they:\n\n1. Wait for OpenSearch to be available\n1. Delete all indices from OpenSearch\n1. Start an instance of the API. That API will be available at \u003chttp://localhost:3000/dev/\u003e\n1. Wait for the API to be available\n1. Run the system tests in `./tests/system/test-*.js`\n1. Stop the API\n\nBefore running the system tests, make sure to start OpenSearch using:\n\n```shell\ndocker compose up -d\n```\n\nRunning these tests requires the timeout utility is installed. On Linux,\nthis is probably already installed, and on macOS it can be installed with `brew install coreutils`.\n\nOnce OpenSearch has been started, run the system tests:\n\n```shell\nnpm run test:system\n```\n\nWith coverage:\n\n```shell\nnpm run test:system:coverage\n```\n\nA subset of system tests may be run by providing a glob matching the test files to run:\n\n```shell\nnpm run test:system test-api-item-*\n```\n\n### Updating the OpenAPI specification\n\nThe OpenAPI specification is served by the endpoint `/api`.\n\nThis file is location in [src/lambdas/api/openapi.yaml](src/lambdas/api/openapi.yaml).\n\nWhen the API is updated to a new STAC API release, this file must be updated. To update it,\nfirst install [yq](https://github.com/mikefarah/yq), then run:\n\n```shell\nbin/build-openapi.sh\n```\n\nThis script combines all of the STAC API OpenAPI definitions for each conformance class into one file.\n\nNext, edit that file to make it specific to this server. For example:\n\n- edit to change the title from `STAC API - Item Search` to just `STAC API`\n- remove all of the Filter Extension references\n- Fix each endpoint, especially the Landing Page defintion, which gets duplicated\n- Add definitions for each tag\n\nTo validate the resulting OpenAPI file, run\n\n```shell\nnpm run check-openapi\n```\n\nand fix any errors or warnings.\n\n## About\n\n[stac-server](https://github.com/stac-utils/stac-server) was forked from [sat-api](https://github.com/sat-utils/sat-api). Stac-server is for STAC versions 0.9.0+, while sat-api exists for versions of STAC prior to 0.9.0.\n\n## License\n\nstac-server is licensed under [The MIT License](https://opensource.org/license/mit/).\nCopyright for portions of stac-server is held by Development Seed (2016) as\npart of project [sat-api](https://github.com/sat-utils/sat-api)\n[original license](https://github.com/sat-utils/sat-api/blob/master/LICENSE). Copyright for all changes to stac-server since the fork date is held by Element 84, Inc (2020).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstac-utils%2Fstac-server","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fstac-utils%2Fstac-server","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fstac-utils%2Fstac-server/lists"}