{"id":21674170,"url":"https://github.com/aehrc/terraform-aws-serverless-beacon","last_synced_at":"2025-04-12T04:22:13.461Z","repository":{"id":34150330,"uuid":"170600205","full_name":"aehrc/terraform-aws-serverless-beacon","owner":"aehrc","description":"Serverless implementation of Beacon V2 protocol, to enable cheaper and faster exchange of genomic and phenotypic information ","archived":false,"fork":false,"pushed_at":"2025-02-27T23:41:17.000Z","size":28514,"stargazers_count":9,"open_issues_count":0,"forks_count":5,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-25T23:51:22.321Z","etag":null,"topics":["beacon","serverless","terraform"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aehrc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-14T00:24:00.000Z","updated_at":"2025-02-27T23:41:27.000Z","dependencies_parsed_at":"2023-02-14T03:02:53.585Z","dependency_job_id":"57534c86-72d6-4828-b982-c1aaf5443fa4","html_url":"https://github.com/aehrc/terraform-aws-serverless-beacon","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fterraform-aws-serverless-beacon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fterraform-aws-serverless-beacon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fterraform-aws-serverless-beacon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aehrc%2Fterraform-aws-serverless-beacon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aehrc","download_url":"https://codeload.github.com/aehrc/terraform-aws-serverless-beacon/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248514511,"owners_count":21116975,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beacon","serverless","terraform"],"created_at":"2024-11-25T13:43:19.073Z","updated_at":"2025-04-12T04:22:13.431Z","avatar_url":"https://github.com/aehrc.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!-- TOC --\u003e\u003ca name=\"serverless-beacon\"\u003e\u003c/a\u003e\n## Serverless Beacon\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat\u0026labelColor=ef8336)](https://pycqa.github.io/isort/)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"assets/logo-black.png\" alt=\"sbacon\" width=\"600\"\u003e\n\u003c/p\u003e\n\n\u003c!-- TOC start (generated with https://github.com/derlin/bitdowntoc) --\u003e\n\n- [Serverless Beacon](#serverless-beacon)\n- [Why serverless?](#why-serverless)\n- [Introduction to sBeacon](#introduction-to-sbeacon)\n- [Installation](#installation)\n   * [Option 1: Setting up the development environment on Amazon Linux](#option-1-setting-up-the-development-environment-on-amazon-linux)\n   * [Option 2: Using the docker image](#option-2-using-the-docker-image)\n   * [Option 3: Using the VSCODE dev containers](#option-3-using-the-vscode-dev-containers)\n- [Deployment](#deployment)\n   * [Direct deployment](#direct-deployment)\n   * [Use as a module](#use-as-a-module)\n- [Development](#development)\n- [API Usage](#api-usage)\n   * [Example data](#example-data)\n   * [Data ingestion API](#data-ingestion-api)\n   * [Query API](#query-api)\n- [Securing the API](#securing-the-api)\n- [Troubleshooting](#troubleshooting)\n   * [Docker error (invalid reference format)](#docker-error-invalid-reference-format)\n   * [Illegal instruction (core dumped)](#illegal-instruction-core-dumped)\n   * [Provider produced inconsistent final plan](#provider-produced-inconsistent-final-plan)\n   * [Updating partition_keys schema in glue.catalog_table according to AWS documentation](#updating-partition_keys-schema-in-gluecatalog_table-according-to-aws-documentation)\n\n\u003c!-- TOC end --\u003e\n\n\u003c!-- TOC --\u003e\u003ca name=\"why-serverless\"\u003e\u003c/a\u003e\n## Why serverless?\nServerless means the service does not require any servers to be provisioned. The\nidea is to minimise running costs, as well as support arbitrary scalablility. It\nalso means setup is very fast.\n\n\u003c!-- TOC --\u003e\u003ca name=\"introduction-to-sbeacon\"\u003e\u003c/a\u003e\n## Introduction to sBeacon\nsBeacon implements Beacon v2 protocol according to the\n[ga4gh specification](https://github.com/ga4gh-beacon/specification). sBeacon can be used as a beacon network participant. Please refer to [https://docs.genomebeacons.org/networks/](https://docs.genomebeacons.org/networks/).\n\n\u003c!-- TOC --\u003e\u003ca name=\"installation\"\u003e\u003c/a\u003e\n## Installation\n\nYou can use either local development or a docker environment for development and deployment. First download the repository using the following command. If you're missing the `git` command please have a look at the **Option 1** commands.\n\n```bash\n# use following or the bitbucket if you have access to it\ngit clone https://github.com/aehrc/terraform-aws-serverless-beacon.git\ncd terraform-aws-serverless-beacon\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"option-1-setting-up-the-development-environment-on-amazon-linux\"\u003e\u003c/a\u003e\n### Option 1: Setting up the development environment on Amazon Linux\n\nNote: the following instructions are strictly for `Amazon Linux 2023 AMI 2023.0.20230419.0 x86_64 HVM kernel-6.1` AMI with name `al2023-ami-2023.0.20230419.0-kernel-6.1-x86_64`.\n\nSkip to next section if you're only interested in deployment or using a different architecture compared to AWS lambda environment. The following setup must be performed on a latest Amazon Linux instance to match the lambda runtimes. If this is not a viable option, please resort to using Docker.\n\nRun the following shell commands to setup necessary build tools. Valid for Amazon Linux development instances.\n\nRequired dependencies\n* Compressionlibraries `xz`, `bzip2` and `zlib`\n* Exact python version - `Python3.12`\n\nInstall system-wide dependencies\n\n```bash\n# Install development essentials\nsudo yum update\nsudo yum upgrade\nsudo yum install -y git openssl-devel libcurl-devel wget bzip2-devel xz-devel libffi-devel zlib-devel autoconf intltool \n```\n\nInstall `Python 3.12` to a virtual environment\n```bash\n# Download and install python\ncd ~\nwget https://www.python.org/ftp/python/3.12.5/Python-3.12.5.tgz\n\ntar xzf Python-3.12.5.tgz\ncd Python-3.12.5\n./configure --enable-optimizations\nsudo make altinstall\n\ncd ~\npython3.12 -m venv py312\n\n# activate py312 environment\nsource ~/py312/bin/activate \n```\n\nMake sure you have the terraform version `Terraform v1.9.4` or newer if you're not using the docker image. Run the following command to get the terraform binary.\n\n```bash\n# only for linux - find other OS version here https://releases.hashicorp.com/terraform/1.9.4/\ncd ~\nwget https://releases.hashicorp.com/terraform/1.9.4/terraform_1.9.4_linux_amd64.zip\nsudo unzip terraform_1.9.4_linux_amd64.zip -d /usr/bin/\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"option-2-using-the-docker-image\"\u003e\u003c/a\u003e\n### Option 2: Using the docker image\n\nInitialise the docker container using the following command.\n\n```bash\n# on x86_64 machines\ndocker build -t csiro/sbeacon ./docker\n# on aarch64\ndocker build --platform linux/amd64  -t csiro/sbeacon ./docker\n```\n\nThis will initialise the docker container that contains everything you need including terraform. In order to start the docker container from within the repository directory run the following command.\n\n```bash\ndocker run --rm -it -v `pwd`:`pwd` -v /var/run/docker.sock:/var/run/docker.sock  -w `pwd` --platform linux/x86_64 csiro/sbeacon:latest /bin/bash\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"option-3-using-the-vscode-dev-containers\"\u003e\u003c/a\u003e\n### Option 3: Using the VSCODE dev containers\n\nYour system must have docker installed with the active user having essential permissions to use containers.\n\nWe have placed a devcontainer configuration in the `.devcontainer` directory. Install `dev containers` extension in your VSCODE ([extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)).\n\nOpen the cloned repository directory folder using VSCODE.\n\nClick on the `Remote Indicator (\u003e\u003c)` icon on bottom left and press `Reopen in container` to get started. You will have VSCODE open inside the appropriate development environment with essential plugins including `aws cli` and `terraform cli`.\n\n\u003c!-- TOC --\u003e\u003ca name=\"deployment\"\u003e\u003c/a\u003e\n## Deployment\n\nYou can simply deploy the cloned repository following the establishment of AWS keys in the development terminal. Alternatively, sBeacon can be used as a module in an existing terraform project.\n\nDo this only once or as you change core libraries or the python lambda layer.\n\n```bash\n$ ./init.sh\n```\n\nNow set the AWS access keys and token as needed. Since docker uses the same user permissions this may not be needed if you're using an authorised EC2 instance.\n\n```bash\nexport AWS_ACCESS_KEY_ID=\"AWS_ACCESS_KEY_ID\"\nexport AWS_SECRET_ACCESS_KEY=\"AWS_SECRET_ACCESS_KEY\"\nexport AWS_SESSION_TOKEN=\"AWS_SESSION_TOKEN\"\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"direct-deployment\"\u003e\u003c/a\u003e\n### Direct deployment\n\nInstall using `terraform init` to pull the module, followed by running `terraform apply` will create the infrastucture. For adding data to the beacon, see the API. To shut down the entire service run `terraform destroy`. Any created datasets will be lost (but not the VCFs on which they are based).\n\n```bash\nterraform init\nterraform plan # should finish without errors\nterraform apply\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"use-as-a-module\"\u003e\u003c/a\u003e\n### Use as a module\n\nYour beacon deployment could be a part of a larger program with a front-end and other services. In that case, on the parent folder that the repo folder resides, create a `main.tf` file.\n\n```bash\n# main.tf\nmodule \"serverless-beacon\" {\n    # repo folder\n    source                      = \"./terraform-aws-serverless-beacon\"\n    beacon-id                   = \"au.csiro-serverless.beacon\"\n    # bucket prefixes\n    variants-bucket-prefix      = \"sbeacon-\"\n    metadata-bucket-prefix      = \"sbeacon-metadata-\"\n    lambda-layers-bucket-prefix = \"sbeacon-lambda-layers-\"\n    # beacon variables\n    beacon-name                 = \"\"\n    organisation-id             = \"\"\n    organisation-name           = \"\"\n    # aws region\n    region                      = \"REGION\"\n}\n``` \nPlease refer to [./examples/minimum/](./examples/minimum/) or  [./examples/full](./examples/full) to find a minimal and a complete setup. Consider adding `outputs.tf` file as well.\n\nFinally deploy using,\n\n```bash\nterraform init\nterraform plan # should finish without errors\nterraform apply\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"development\"\u003e\u003c/a\u003e\n## Development\n\nAll the layers needed for the program to run are in layers folder. To add a new layer for immediate use with additional configs, run the following commands. Once the decision to use the library is finalised update the `init.sh` script to automate the process.\n\n* Python layer\n```bash\ncd terraform-aws-serverless-beacon\npip install --target layers/\u003cLibrary Name\u003e/python \u003cLibrary Name\u003e\n```\n\n* Binary layer\n```bash\n# clone the repo somewhere else\ngit clone \u003cREPO\u003e \ncd \u003cREPO\u003e\nmkdir build \u0026\u0026 cd build \u0026\u0026 cmake .. \u0026\u0026 make \u0026\u0026 make install\n\n# copy the bin and lib folders to a folder inside layers\ncp bin terraform-aws-serverless-beacon/layers/\u003cLibrary Name\u003e/\ncp lib terraform-aws-serverless-beacon/layers/\u003cLibrary Name\u003e/\n\n# troubleshoot with \"ldd ./binary-name\" to see what libaries needed\n# you can use the following command to copy the libraries to binaries/lib/\nldd \u003cbinary file\u003e | awk 'NF == 4 { system(\"cp \" $3 \" ./layers/binaries/lib\") }'\n```\n\n* Collaborative development\n\nPlease make a copy of `backend.tf.template` with suited parameters and rename as `backend.tf`. Refer to documentation for more information [https://www.terraform.io/language/settings/backends/configuration](https://www.terraform.io/language/settings/backends/configuration). If this is not done, make sure the terraform lock and state files are stored securely to avoid infrastructure-vs-code inconsistencies. Please refer to [./examples/full](./examples/full) to find a an example backend.\n\n\u003c!-- TOC --\u003e\u003ca name=\"api-usage\"\u003e\u003c/a\u003e\n## API Usage\n\n\u003c!-- TOC --\u003e\u003ca name=\"example-data\"\u003e\u003c/a\u003e\n### Example data\n\nPlease find the data in [./examples/test-data/](./examples/test-data/) and use the [./docs/USAGE-GUIDE.md](./docs/USAGE-GUIDE.md) to try the provided test data.\n\n\u003c!-- TOC --\u003e\u003ca name=\"data-ingestion-api\"\u003e\u003c/a\u003e\n### Data ingestion API\n\nPlease refer to the documentation outlined at [./docs/INGESTION-GUIDE.md](./docs/INGESTION-GUIDE.md).\n\n\u003c!-- TOC --\u003e\u003ca name=\"query-api\"\u003e\u003c/a\u003e\n### Query API\n\nQuerying is available as per API defined by BeaconV2 [https://beacon-project.io/#the-beacon-v2-model](https://beacon-project.io/#the-beacon-v2-model). \n* All the available endpoints can be retrieved using the deployment url's `/map`. \n* Schema for beacon V2 configuration can be obtained from `/configuration`.\n* Entry types are defined at `/entry_types`.\n\n\u003c!-- TOC --\u003e\u003ca name=\"securing-the-api\"\u003e\u003c/a\u003e\n## Securing the API\n\nPlease refer to the documentation outlined at [./docs/AUTH-GUIDE.md](./docs/AUTH-GUIDE.md).\n\n\u003c!-- TOC --\u003e\u003ca name=\"troubleshooting\"\u003e\u003c/a\u003e\n## Troubleshooting\n\n\u003c!-- TOC --\u003e\u003ca name=\"docker-error-invalid-reference-format\"\u003e\u003c/a\u003e\n### Docker error (invalid reference format)\n\nThis is likely caused by white spaces in your current working directory absolute path. Please use the following command to start images.\n\n```bash\ndocker run --rm -it -v \"`pwd`\":\"`pwd`\" -v /tmp:/tmp  -u `id -u`:`id -g` -w \"`pwd`\" csiro/sbeacon:latest /bin/bash\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"illegal-instruction-core-dumped\"\u003e\u003c/a\u003e\n### Illegal instruction (core dumped)\nYou'll also need to do this if lambda functions start to display \"Error: Runtime exited with error: signal: illegal instruction (core dumped)\". In this case it's likely AWS Lambda has moved onto a different architecture from haswell (Family 6, Model 63). You can use cat /proc/cpuinfo in a lambda environment to find the new CPU family and model numbers, or just change -march=haswell to -msse4.2 or -mpopcnt for less optimisation.\n\n```bash\n./init.sh -msse4.2 -O3\n```\n\n\u003c!-- TOC --\u003e\u003ca name=\"provider-produced-inconsistent-final-plan\"\u003e\u003c/a\u003e\n### Provider produced inconsistent final plan\n\nIf `terraform apply --auto-approve` complaints about a provider error. Please retry. If the issue persists, please raise an issue with the complete terraform log.\n\n\u003c!-- TOC --\u003e\u003ca name=\"updating-partition_keys-schema-in-gluecatalog_table-according-to-aws-documentation\"\u003e\u003c/a\u003e\n### Updating partition_keys schema in glue.catalog_table according to AWS documentation\n\n```bash\nError: error setting partition_keys: Invalid address to set: []string{\"partition_keys\", \"0\", \"parameters\"}\n```\n\nThis is a known issue as outline in the following PR in terraform AWS.\n\n* https://github.com/hashicorp/terraform-provider-aws/pull/26702\n\nThere is not workaround for this yet and we must delete `sbeacon-terms-index` table and `sbeacon-terms` tables before performing a terraform apply. After that, we can do the terraform apply and then run the indexer again.\n\nIssue exists to date and has been active for the last few years - https://github.com/hashicorp/terraform-provider-aws/issues/26686\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fterraform-aws-serverless-beacon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faehrc%2Fterraform-aws-serverless-beacon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faehrc%2Fterraform-aws-serverless-beacon/lists"}