{"id":28395166,"url":"https://github.com/databiosphere/terra-aws-resource-discovery","last_synced_at":"2025-07-22T23:03:17.322Z","repository":{"id":144597750,"uuid":"608845530","full_name":"DataBiosphere/terra-aws-resource-discovery","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2024-10-31T18:51:54.000Z","size":349,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":19,"default_branch":"main","last_synced_at":"2025-06-27T01:38:01.952Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2023-03-02T21:21:06.000Z","updated_at":"2024-10-31T18:51:45.000Z","dependencies_parsed_at":null,"dependency_job_id":"0db1628c-0381-4d08-9fa6-4832acbaae19","html_url":"https://github.com/DataBiosphere/terra-aws-resource-discovery","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/terra-aws-resource-discovery","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-aws-resource-discovery","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-aws-resource-discovery/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-aws-resource-discovery/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-aws-resource-discovery/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/terra-aws-resource-discovery/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-aws-resource-discovery/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266586905,"owners_count":23952205,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-22T02:00:09.085Z","response_time":66,"last_error":null,"robots_txt_status":null,"robots_txt_updated_at":null,"robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:22.445Z","updated_at":"2025-07-22T23:03:17.303Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# terra-aws-resource-discovery\n\nTerra AWS Support Resource Discovery Client Library\n\nTable of Contents\n=================\n* [Introduction](#introduction)\n* [Support Resource Organization](#support-resource-organization)\n  * [Environments](#environments)\n  * [Landing Zones](#landing-zones)\n* [Discovery Implementation](#discovery-implementation)\n  * [Support Resource Deployment](#support-resource-deployment)\n  * [Configuration Schemas](#configuration-schemas)\n  * [Schema Evolution and Versioning](#schema-evolution-and-versioning)\n  * [Configuration Storage Layout](#configuration-storage-layout)\n* [Library Development Notes](#library-development-notes)\n  * [Static Test Data](#static-test-data)\n    * [Updating an Existing Test Case Config File](#updating-an-existing-test-case-config-file)\n    * [Writing a New Test Case Config File](#writing-a-new-test-case-config-file)\n\n# Introduction\nIn order for Terra services to manage and provide access to [Controlled Resources](https://github.com/DataBiosphere/terra-workspace-manager#overview)\nin Amazon Web Services (AWS), there must exist several AWS cloud resources used for this purpose.\nThese resources will be referred to as **Support Resources**.\n\nSupport resources may be [**Global** or **Regional**](https://docs.aws.amazon.com/ram/latest/userguide/working-with-regional-vs-global.html):\n* **Global Support Resources** are resources that do not exist in an AWS Region.  These are\nmost commonly (but not limited to) IAM resources ([IAM Roles](https://aws.amazon.com/iam/features/manage-roles/),\n[IAM Policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_id-based)).\n* **Regional Support Resources** are resources that exist in a specific AWS Region, such as [S3\nstorage buckets](https://aws.amazon.com/s3/) and [KMS Keys](https://aws.amazon.com/kms/).\n\nThis library provides the ability to discover all of the Support Resources needed by Terra services and\npresent in a given AWS environment, so that Terra services can make use of these Support Resources to\ncreate and provide access to Terra Controlled Resources in AWS.\n\n# Support Resource Organization\n## Environments\nIn Terra, an **Environment** corresponds to a single [AWS Account](https://docs.aws.amazon.com/accounts/latest/reference/accounts-welcome.html).\nAll Support Resources and Controlled Resources exist in a single Environment.\n\nThe [`Environment` class](src/main/java/bio/terra/aws/resource/discovery/Environment.java) provides\ngetters for all Global Support Resources usable by Terra Services.\n\n## Landing Zones\nIn Terra, a Landing Zone corresponds to the nexus of a [Terra Environment](#Environments) and an AWS Region.\nEach Regional Support Resource exists within a Landing Zone, as do all Terra Controlled Resources.\n\nThe [`LandingZone` class](src/main/java/bio/terra/aws/resource/discovery/LandingZone.java) provides\ngetters for all Regional Support Resources usable by Terra Services in a given AWS Region in a\nTerra AWS Environment.\n\nInstances of class `LandingZone` are obtained by calling method `getLandingZone()` on an `Environment`\ninstance, passing the corresponding AWS Region for the Landing Zone within the Environment.\n\nThe following diagram illustrates the relationship between Environments, Regions, and Terra\nControlled Resources in AWS:\n![](images/AWS%20Workspace%20Landing%20Zone.png)\n\nThe area in the dark purple dashed box represents the resources (Support and Controlled) that make\nup a Terra AWS Landing Zone.\n\n# Discovery Implementation\n## Support Resource Deployment\nDeployment of Support Resources in an AWS Environment is out of scope for this document.  However,\nproducers of these resources are required to provide discoverability of their Support Resources by\nusing the conventions described below.\n## Configuration Schemas\nWe have chosen to use [Apache Avro](https://avro.apache.org/) for specifying the schema for Support\nResource discovery, for several reasons:\n* Strong schema evolution support.\n* Support for many programming languages, supporting both Java service development and\ninfrastructure-deployment-side testing/validation in other languages.\n* Human readable JSON-based Schema IDL and JSON data files.\n\n*This repository ([`terra-aws-resource-discovery`](https://github.com/DataBiosphere/terra-aws-resource-discovery))\nwill be the single source of truth for Resource Discovery Schemas.*\n\nTwo configuration schemas are specified in this repository:\n* [`Environment.avsc`](src/main/avro/Environment.avsc) - the schema used to describe all Global\nSupport Resources available in a Terra AWS Environment.\n* [`LandingZone.avsc`](src/main/avro/LandingZone.avsc) - the schema used to describe all Regional\n  Support Resources available in a Landing Zone within a Terra AWS Environment.\n\n## Schema Evolution and Versioning\nAvro always requires two versions of schema when deserializing:\n\n* The Writer's Schema (the current schema version used by the writer at the time the data was written)\n* The Reader's Schema (the current schema version used by the reader of the data)\n\nThus, when a configuration  is written to storage, it will include both the version of the schema\nused to write the data (as retrieved from the artifacts of this library, published to GitHub with\neach release) as well as the data itself.\n\nWe will maintain Forward Transitive Compatibility as described in\n[this document](https://docs.confluent.io/platform/current/schema-registry/avro.html#forward-compatibility).\n\nSpecifically, these are the rules we will follow:\n* Any change to either Avro Schema file *must* incur at least a minor version bump.\n* Fields may be added to the schema without breaking forward transitive compatibility.\n* Optional fields may be deleted without breaking forward transitive compatibility.\n* Producers of configuration artifacts (backend infrastructure deployment) *must* be updated before\nconsumers (Terra services).\n* Breaking changes *must* be a major version bump, and should be avoided.  *We will not support\nforward compatibility between major versions.*\n\n## Configuration Storage Layout\nThe `terra-aws-resource-discovery` provides discovery of all Support Resources in a single\nEnvironment through interface [`EnvironmentDiscovery`](src/main/java/bio/terra/aws/resource/discovery/EnvironmentDiscovery.java).\nThree implementations of this interface are provided:\n* Class [`S3EnvironmentDiscovery`](src/main/java/bio/terra/aws/resource/discovery/S3EnvironmentDiscovery.java)\ndiscovers Support Resources by reading them from an S3 bucket that the caller has access to.\n* Class [`FilesystemEnvironmentDiscovery`](src/main/java/bio/terra/aws/resource/discovery/FilesystemEnvironmentDiscovery.java)\ndiscovers Support Resources by reading them from directories within an accessible file system path.\n* Class [`CachedEnvironmentDiscovery`](src/main/java/bio/terra/aws/resource/discovery/CachedEnvironmentDiscovery.java)\nis used in conjunction with one of the two above classes to cache discovery results between calls to\n`discoverEnvironment()`, in order to reduce the number of calls to storage API's.\n\nWhether stored in an S3 Bucket or a local file system directory, the following layout is expected\nby the discovery library (in this example, this is major version 1 of the library, and we are\ndiscovering an Environment with two Landing Zones in AWS regions `eu-central-1` and `us-east-1`:\n```\nv1\n├── v1/environment\n│   ├── v1/environment/config.json\n└── v1/landingzones\n    ├── v1/landingzones/eu-central-1\n    │   └── v1/landingzones/eu-central-1/config.json\n    └── v1/landingzones/us-east-1\n        └── v1/landingzones/us-east-1/config.json\n```\nEach `config.json` file contains the following JSON content:\n```json\n{\n  \"schema\": \"\u003cbase64-encoded avro schema\u003e\",\n  \"payload\": \"\u003cbase64-encoded support resource data\u003e\"\n}\n```\n\nFile `v1/environment/config.json` uses schema [`Environment.avsc`](src/main/avro/Environment.avsc)\nto describe the Global Support Resources in the Environment.\n\nFiles `v1/landingzones/eu-central-1/config.json` and `v1/landingzones/eu-central-1/config.json` use\nschema [`LandingZone.avsc`](src/main/avro/LandingZone.avsc) to describe the Regional Support\nResources in the Environment's Landing Zones in regions `eu-central-1` and `us-east-1`\nrespectively.\n\n# Library Development Notes\n\n## Dependency Locking\nWe use [Gradle's dependency locking](https://docs.gradle.org/current/userguide/dependency_locking.html) to ensure that builds use the same transitive dependencies, so they're reproducible. This means that adding or updating a dependency requires telling Gradle to save the change. Execute the below command when any dependency versions are updated.\n```\n./gradlew dependencies --write-locks\n```\n\n## Static Test Data\nClass [`EnvironmentDiscoveryTestBase`](src/test/java/bio/terra/aws/resource/discovery/EnvironmentDiscoveryTestBase.java)\nserves as a test fixture consuming static test data written in folder\n[`src/test/resources/test_discovery_data`](src/test/resources/test_discovery_data)\nto allow for testing of Avro parsing and schema validation.\n\nIn order to update the static test schema files as schemas evolve (as well as create new test data)\ntest authors can make use of the following scripts in the [`tools`](tools) directory:\n* [`decode-test-data.sh`](tools/decode-test-data.sh) creates an out-of-tree directory (mirroring the\n  structure of the `src/test/resources/test_discovery_data` directory tree), but with the payload\n  parsed into un-encoded JSON.  Changes can be made in this out-of-tree location to the plaintext\n  JSON and written back to the source tree using `encode-test-data.sh` script.\n* [`encode-test-data.sh`](tools/encode-test-data.sh) can be used to write the payload changes made\n  to an out-of-tree directory created with `decode-test-data.sh` back to the in-tree test\n  configuration files in encoded form (along with the current in-tree schema versions).\n* [`parse-schema.sh`](tools/parse-schema.sh) parses the Base64-encoded Avro schema from a\n`config.json` Configuration file/object and prints it as plain JSON.\n* [`parse-payload.sh`](tools/parse-payload.sh) parses the Base64-encoded payload from a\n`config.json` Configuration file/object and prints it as plain JSON.\n* [`print-config.sh`](tools/print-config.sh) takes an Avro schema and payload data file (both in\nplain JSON) and Base64 encodes them into a configuration file format, printing the output to STDOUT\n\n### Updating Multiple Test Case Config Files\n``` shell\n# Decode all test payloads into empty directory ~/DiscoveryTestData\n$ ./tools/decode-test-data.sh src/test/resources/test_discovery_data /DiscoveryTestData\n\n# The following unencoded JSON files mirror those in the src/test/resources/test_discovery_data\n# directory.  These payload files can be edited in-place to update the test data payloads.\n$ find /DiscoveryTestData/ -type file\n/Users/jczerk/DiscoveryTestData//add_field_before_schema_update/v0/environment/payload.json\n/DiscoveryTestData//notebook_lifecycle_mismatch/v0/environment/payload.json\n/DiscoveryTestData//notebook_lifecycle_mismatch/v0/landingzones/us-east-1/payload.json\n/DiscoveryTestData//no_landing_zones/v0/environment/payload.json\n/DiscoveryTestData//validation/v0/environment/payload.json\n/DiscoveryTestData//validation/v0/landingzones/us-west-1/payload.json\n/DiscoveryTestData//validation/v0/landingzones/us-east-1/payload.json\n/DiscoveryTestData//validation/v0/landingzones/fake-region/payload.json\n\n# Now use the encode-test-data.sh script to encode the updated payloads (along with any\n# in-tree schema updates from src/main/avro) into the in-tree test configuration files.\n$ ./tools/encode-test-data.sh src/main/avro/ ~/DiscoveryTestData/ src/test/resources/test_discovery_data/\n```\n### Updating a Single Existing Test Case Config File\n```shell\n# Make any changes to the Avro schema, in this case src/main/avro/Environment.avsc\n\n# Choose the file that you wish to update\nTEST_FILE=\"src/test/resources/test_discovery_data/validation/v0/environment/config.json\"\n\n# Parse the payload from the existing test data file and write it to a scratch file for editing\n./tools/parse-payload.sh ${TEST_FILE} \u003e /tmp/scratch.json\n\n# Make any changes to the test payload to the scratch file directly\n\n# Now write the updated schema and test data back to the original file\n./tools/print-config.sh src/main/avro/Environment.avsc /tmp/scratch.json \u003e ${TEST_FILE}\n```\n\n### Writing a New Test Case Config File\n```shell\n# Write your test case payload to a new file somewhere outside of the terra-aws-resource-discovery\n# filesystem tree (optionally making any required schema changes in src/main/avro)\nNEW_TEST_DATA=/tmp/new_test.json\n\n# Identify the new test data case location\nNEW_TEST_CONFIG=src/test/resources/test_discovery_data/new_test_data/v0/environment/config.json\n\n# Now write the schema and new test data to the new config file\n./tools/print-config.sh src/main/avro/Environment.avsc ${NEW_TEST_DATA} \u003e ${NEW_TEST_CONFIG}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-aws-resource-discovery","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fterra-aws-resource-discovery","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-aws-resource-discovery/lists"}