{"id":28395131,"url":"https://github.com/databiosphere/terra-gcs-bq-streaming-functions","last_synced_at":"2025-06-27T01:31:17.259Z","repository":{"id":46029611,"uuid":"368622396","full_name":"DataBiosphere/terra-gcs-bq-streaming-functions","owner":"DataBiosphere","description":null,"archived":true,"fork":false,"pushed_at":"2021-11-19T00:30:35.000Z","size":663,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-01T06:52:19.970Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-05-18T17:57:50.000Z","updated_at":"2025-05-28T14:44:14.000Z","dependencies_parsed_at":"2022-08-24T17:41:44.179Z","dependency_job_id":null,"html_url":"https://github.com/DataBiosphere/terra-gcs-bq-streaming-functions","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/terra-gcs-bq-streaming-functions","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-gcs-bq-streaming-functions","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-gcs-bq-streaming-functions/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-gcs-bq-streaming-functions/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-gcs-bq-streaming-functions/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/terra-gcs-bq-streaming-functions/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fterra-gcs-bq-streaming-functions/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172364,"owners_count":23269993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:15.311Z","updated_at":"2025-06-27T01:31:17.250Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Terra Cloud Functions Repository\n\n## Overview\nThis repository contains lightweight, portable functions that target the `Java 11` runtime for Google Cloud Functions.\n\nThe open source Java Functions Framework provides an API that can be used to author your functions, as well as an invoker which can be called to run the functions locally on your machine, or anywhere with a `Java 11` environment.\n\nCloud Build resources that enable remote builds to take place in the cloud can be customized in `cloudbuild.yaml`.\n\nThere are a number of Java Deployment options for deploying cloud functions.\n\nFor more details, please refer to the following articles\n\nhttps://cloud.google.com/blog/topics/developers-practitioners/deploying-serverless-platforms-github-actions\n\nhttps://cloud.google.com/functions/docs/first-java#gradle_1\n\nhttps://cloud.google.com/functions/docs/concepts/java-deploy#gradle\n\nThis repository contains the following Java sources for various Cloud Function applications\n\n| Java Class                                  | Cloud Function              | Application |\n|---------------------------------------------|-----------------------------|-------------|\n| TestRunnerStreamingFunction.java            | testrunner-results-streamer | TestRunner results streamer function |\n| DeltaLayerRawFunction.java                  |                             | Delta Layer |\n\n## Deploying Cloud Functions to Java 11 Runtime with GitHub Actions\n\nThe `deploy-cloud-function` action deploys a Cloud Function to the appropriate environment, it accepts the following input parameters\n\n| Input                        | Description                | Required    | Default |\n|------------------------------|----------------------------|-------------|---------|\n| deployer-sa-email            | The email address of the IAM service account used for deploying the cloud function                                    | Yes | n/a |\n| deployer-sa-key              | The IAM service account key used for deploying the cloud function                                                     | Yes | n/a |\n| service-account              | The email address of the IAM service account associated with the function at runtime                                  | Yes | n/a |\n| func                         | The fully specified name of the cloud function                                                                        | Yes | n/a |\n| entry-point                  | Name of a Google Cloud Function (as defined in source code) that will be executed                                     | Yes | n/a |\n| memory                       | Limit on the amount of memory the function can use (128MB, 256MB, 512MB, 1024MB, 2048MB, 4096MB, and 8192MB)          | No  | 256MB |\n| trigger-bucket               | Google Cloud Storage bucket name. Every change in files in this bucket will trigger function execution                | Yes | n/a |\n| project                      | GCP Project                                                                                                           | Yes | n/a |\n| region                       | The Cloud region for the function. Overrides the default functions/region property value for this command invocation  | No  | us-central1 |\n| source                       | Local directory of cloud function source                                                                              | No  | . |\n| env-vars-file                | Path to a local YAML file with definitions for all environment variables. All existing environment variables will be removed before the new environment variables are added | No | n/a |                                                                                                 | Yes |\n\nPlease refer to `deploy-testrunner-dev-streamer.yml` for a working example.\n\nThe `deploy-cloud-function` action requires a deployer service account (identified as `deployer-sa-key`, `deployer-sa-email`) for cloud function deployment and an identity (`service-account`) that the cloud function assumes at runtime.\nThese service accounts are provisioned by Terraform and their secrets are stored in the Vault.\n\nFor more information, please refer to\n* [deltalayer module](https://github.com/broadinstitute/terraform-ap-modules/tree/master/deltalayer)\n* [deltalayer deployments](https://github.com/broadinstitute/terraform-ap-deployments/tree/master/deltalayer)\n* [testrunner module](https://github.com/broadinstitute/terraform-ap-modules/tree/master/testrunner)\n* [testrunner deployments](https://github.com/broadinstitute/terraform-ap-deployments/tree/master/testrunner)\n\n## Syncing secrets from Vault to GitHub via Atlantis\n\nFor the purpose of Cloud Function deployment using the `deploy-cloud-function` action. \nThe required Vault secrets need to be synced to this GitHub repo via Atlantis, the process is documented [here](https://docs.google.com/document/d/1JbjV4xjAlSOuZY-2bInatl4av3M-y_LmHQkLYyISYns/edit#heading=h.kor6m5ppv2u).\n\nFor a working example of moving the secrets tto this repo, please see\n\n[github/tfvars/databiosphere-terra-gcs-bq-streaming-functions.tfvars](https://github.com/broadinstitute/terraform-ap-deployments/blob/master/github/tfvars/databiosphere-terra-gcs-bq-streaming-functions.tfvars)\n\n## Tutorial\n\n#### A Sample Cloud Function\n\n```ProtoApp.java```\n```java\npackage bio.terra.cloudfiletodatastore.proto;\n\nimport bio.terra.cloudfiletodatastore.FileUploadedMessage;\nimport bio.terra.cloudfiletodatastore.MessageProcessor;;\nimport java.util.logging.Logger;\n\n/** This class is a subclass of App with a custom implementation of the process() logic. */\npublic class ProtoApp extends MessageProcessor {\n  private static final Logger logger = Logger.getLogger(ProtoApp.class.getName());\n\n  public ProtoApp(FileUploadedMessage fileUploadedMessage) {\n    super(fileUploadedMessage);\n  }\n\n  // Business logic\n  @Override\n  public void processMessage() {\n    String sourceBucket = message.getSourceBucket();\n    String resourceName = message.getResourceName();\n    String projectId = System.getenv(\"GCLOUD_PROJECT\");\n    String dataSet = System.getenv(\"BQ_DATASET\");\n    String table = System.getenv(\"BQ_TABLE\");\n    logger.info(\n        String.format(\n            \"Received GCS event from GCP %s, source bucket %s, resource name %s for BigQuery %s.%s.\",\n            projectId, sourceBucket, resourceName, dataSet, table));\n  }\n}\n```\n\n```ProtoFunc.java```\n```java\npackage bio.terra.cloudfiletodatastore.proto;\n\nimport bio.terra.cloudfiletodatastore.FileUploadedMessage;\nimport bio.terra.cloudfunctions.common.GoogleCloudEventHarness;\nimport bio.terra.cloudfunctions.common.MediaTypeWrapper;\nimport com.google.events.cloud.storage.v1.StorageObjectData;\nimport java.util.logging.Level;\nimport java.util.logging.Logger;\n\n/**\n * This class is a subclass of CloudEventsHarness (i.e. a Cloud Function) with a custom App.\n *\n * \u003cp\u003eThe no-arg and setter method is for supporting Cloud Function initialization and for this\n * function to be potentially used in a managed DI framework as a service.\n */\npublic class ProtoFunc extends GoogleCloudEventHarness {\n    private static final Logger logger = Logger.getLogger(ProtoFunc.class.getName());\n\n    @Override\n    public void doAccept() {\n        try {\n            String expectedBucket = System.getenv(\"GOOGLE_BUCKET\");\n            StorageObjectData event = getEvent(StorageObjectData.class);\n            MediaTypeWrapper mediaType = new MediaTypeWrapper(event.getContentType());\n            if (isGoogleStorageObjectFinalize()\n                    \u0026\u0026 expectedBucket.equals(event.getBucket())\n                    \u0026\u0026 mediaType.isApplicationGzip()) {\n                // App can be injected through DI framework (Spring or Java CDI).\n                FileUploadedMessage fileUploadedMessage =\n                        new FileUploadedMessage(\n                                event.getName(), event.getBucket(), event.getSize(), event.getTimeCreated());\n                ProtoApp app = new ProtoApp(fileUploadedMessage);\n                app.process();\n            } else {\n                logger.log(\n                        Level.SEVERE,\n                        String.format(\n                                \"Malformed event data: Expected %s event from bucket %s of content type %s but received %s event from bucket %s of content type %s\",\n                                GoogleCloudEventHarness.GOOGLE_STORAGE_OBJECT_FINALIZE,\n                                expectedBucket,\n                                \"application/gzip\",\n                                getContext().eventType(),\n                                event.getBucket(),\n                                event.getContentType()));\n                return;\n            }\n        } catch (Exception e) {\n            logger.log(Level.SEVERE, \"An unexpected error occurred.\", e);\n            throw new RuntimeException(e);\n        }\n    }\n}\n```\n\n#### A Sample GitHub Action workflow for deploying Cloud Function\n\nThe following workflow assumes that the GitHub secrets (e.g. DEV_PROTO_CF_DEPLOYER_SA, DEV_PROTO_FUNC_SA) already exists in the repo.\n\n```build-proto-cf-handler.yml```\n```shell\n# This workflow will build a backend cloud function with Gradle\n# For more information see: https://github.com/google-github-actions/deploy-cloud-functions\n\nname: Build and deploy ProtoFunc cloud function\n\non:\n  push:\n    branches:\n      - main\n\njobs:\n  deploy-protofunc:\n\n    runs-on: self-hosted\n    env:\n      DEPLOY_ENV: DEV\n    steps:\n      - uses: actions/checkout@v2\n\n      - name: Configure secrets, SA, Project ID\n        id: config-step\n        run: |\n          DEPLOYER_SA_KEY=$(echo -n ${{ secrets[format('{0}_PROTO_CF_DEPLOYER_SA', env.DEPLOY_ENV)] }} | base64 -d)\n          DEPLOYER_SA_EMAIL=$(echo -n $DEPLOYER_SA_KEY | jq -r .client_email)\n          PROJECT_ID=$(echo -n $DEPLOYER_SA_KEY | jq -r .project_id)\n          FUNC_SA_KEY=$(echo -n ${{ secrets[format('{0}_PROTO_FUNC_SA', env.DEPLOY_ENV)] }} | base64 -d)\n          FUNC_SA_EMAIL=$(echo -n $FUNC_SA_KEY | jq -r .client_email)\n          echo ::add-mask::$DEPLOYER_SA_KEY\n          echo ::add-mask::$DEPLOYER_SA_EMAIL\n          echo ::add-mask::$FUNC_SA_KEY\n          echo ::add-mask::$FUNC_SA_EMAIL\n          echo ::set-output name=deployer-sa-email::$DEPLOYER_SA_EMAIL\n          echo ::set-output name=deployer-sa-key::$DEPLOYER_SA_KEY\n          echo ::set-output name=project-id::$PROJECT_ID\n          echo ::set-output name=func-sa-email::$FUNC_SA_EMAIL\n          echo ::set-output name=project-id::$PROJECT_ID\n\n      # Deploy cloud function from the source\n      - name: Deploy the ProtoFunc function\n        id: deploy-proto-cloudfunc\n        uses: ./.github/actions/deploy-cloud-function\n        with:\n          func: proto-cloudfunc\n          trigger-bucket: ${{ steps.config-step.outputs.project-id }}-proto-bucket\n          entry-point: bio.terra.cloudfiletodatastore.proto.ProtoFunc\n          memory: 512MB\n          project: ${{ steps.config-step.outputs.project-id }}\n          service-account: ${{ steps.config-step.outputs.func-sa-email }}\n          deployer-sa-email: ${{ steps.config-step.outputs.deployer-sa-email }}\n          deployer-sa-key: ${{ steps.config-step.outputs.deployer-sa-key }}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-gcs-bq-streaming-functions","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fterra-gcs-bq-streaming-functions","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fterra-gcs-bq-streaming-functions/lists"}