{"id":28395121,"url":"https://github.com/databiosphere/azul","last_synced_at":"2025-06-27T01:31:13.784Z","repository":{"id":37502460,"uuid":"139095537","full_name":"DataBiosphere/azul","owner":"DataBiosphere","description":"Metadata indexer and query service used for AnVIL, HCA, LungMAP,  and CGP","archived":false,"fork":false,"pushed_at":"2025-06-25T07:40:07.000Z","size":608269,"stargazers_count":8,"open_issues_count":745,"forks_count":2,"subscribers_count":9,"default_branch":"develop","last_synced_at":"2025-06-25T08:38:35.889Z","etag":null,"topics":["boardwalk"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.yaml","contributing":"CONTRIBUTING.rst","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":"security.txt","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2018-06-29T03:18:14.000Z","updated_at":"2025-06-25T00:41:49.000Z","dependencies_parsed_at":"2023-10-12T10:53:04.913Z","dependency_job_id":"d525b85e-f7b5-468a-b2d3-ea94a846dede","html_url":"https://github.com/DataBiosphere/azul","commit_stats":null,"previous_names":[],"tags_count":197,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/azul","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fazul","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fazul/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fazul/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fazul/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/azul/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fazul/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172361,"owners_count":23269991,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boardwalk"],"created_at":"2025-05-31T19:39:11.183Z","updated_at":"2025-06-27T01:31:13.765Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"The Azul project contains the components that together serve as the backend to\nBoardwalk, a web application for browsing genomic data sets.\n\n\n# 1. Architecture Overview\n\n## 1.1 Components\n\n[Data Store]: https://github.com/HumanCellAtlas/data-store\n\nAzul consists of two components: an indexer and a web service. The Azul indexer\nis an AWS Lambda function that responds to web-hook notifications about bundle\naddition and deletion events occurring in a [Data Store] instance. The indexer\nresponds to those notifications by retrieving the bundle's metadata from said\ndata store, transforming it and writing the transformed metadata into an\nElasticsearch index. The transformation extracts selected entities and\ndenormalizes the relations between them into a document shape that facilitates\nefficient queries on a number of customizable metadata facets.\n\nThe Azul web service, another AWS Lambda function fronted by API Gateway, serves\nas a thin translation layer between Elasticsearch and the Boardwalk UI,\nproviding features like pluggable authentication, field name translation and\nintrospective capabilities such as facet and entity type discovery.\n\nBoth the indexer and the web service allow for project-specific customizations\nvia a plug-in mechanism, allowing the Boardwalk UI codebase to be functionally\ngeneric with minimal need for project-specific behavior.\n\n\n## 1.2 Architecture Diagram\n\n![Azul architecture diagram](docs/azul-arch.svg)\n\n\n# 2. Getting Started\n\n\n## 2.1 Development Prerequisites\n\n- Python, the specific verson is defined in an environment variable called\n  `azul_python_version` defined in [environment.py](environment.py)\n\n- The `bash` shell\n\n- GNU make 3.81 or newer\n\n- git 2.36.0 or newer\n\n- [Docker], for running the tests (the community edition is sufficient). The\n  required version is specified in a variable called `azul_docker_version` in\n  [environment.py](environment.py).\n\n- Terraform, to manage deployments. Azul requires a specific version of\n  Terraform, which is defined in a variable called `azul_terraform_version` in\n  [environment.py](environment.py). Refer to the official documentation on how\n  to [install terraform]. Terraform comes as a single, statically linked binary,\n  so the easiest method of installation is to download the binary and put it in\n  a directory mentioned in the `PATH` environment variable.\n\n- AWS credentials configured in `~/.aws/credentials` and/or `~/.aws/config`\n\n- [git-secrets](#211-git-secrets)\n\n- [jq](https://stedolan.github.io/jq/)\n\n- The build process relies on numerous utilities that are pretty much standard \n  on any modern Unix. Things like `perl`, `sort`, `comm`, `uniq`, `sed`, `cp`, \n  `mv` and `rm`.\n\n- For VPN support: OpenSSL (version 1.1.10 and 3.0.5 are known to work but other \n  versions should work, too). LibreSSL, which became the default on macOS at \n  some point, is an acceptible replacement. Version 2.8.3 is known to work.  \n\n- Users of macOS 12 (Monterey) should follow additional steps outlined in \n  [Troubleshooting](#setting-up-the-azul-build-prerequisites-on-macos-12-monterey)\n\n- Users of macOS 11 (Big Sur) should follow additional steps outlined in \n  [Troubleshooting](#installing-python-3812-on-macos-11-big-sur)\n\n[install terraform]: https://developer.hashicorp.com/terraform/downloads\n[Docker]: https://docs.docker.com/install/overview/\n\n\n### 2.1.1 git-secrets\n\n[git-secrets] helps prevent secrets (passwords, credentials, etc.) from being\ncommitted to a Git repository. See the *Installing git-secrets* section of the\nproject's README for instructions how to install [git-secrets] on your OS.\n\nOnce installed, [git-secrets] will need to be configured individually in each \none of your existing clones, be they clones of this repository or any of the \nteam's other repositories. Run\n\n```\ncd /path/to/clone\ngit secrets --install  # install the hooks\n```\n    \nTo register the provider that adds AWS-specific secret patterns, run\n\n```\ngit secrets --global --register-aws\n```\n\nOptionally, to configure [git-secrets] in all repository clones created \nsubsequently, run:\n\n```\ngit secrets --install ~/.git-templates/git-secrets\ngit config --global init.templateDir ~/.git-templates/git-secrets\n```\n\nYou must now verify the proper function of [git-secrets] in each one of your \nexisting clones, be they clones of this repository or any of the team's other \nrepositories:\n\n1) Run `cd /path/to/clone`\n\n2) Make sure there is no `foo.txt` in the current directory\n\n3) Run `(echo -e 'AWS_ACCOUNT_ID=00000000000\\x30' \u003e foo.txt \u0026\u0026 git add foo.txt \u0026\u0026 git hook run pre-commit); git rm -fq foo.txt`\n\n**This must produce output containing `[ERROR] Matched one or more prohibited \npatterns`. If it doesn't, proper function of [git-secrets] has not been \nverified!**\n\nIf you get `git: 'hook' is not a git command. See 'git --help'.`, you are using \nan outdated version of `git`.\n\nIf you get `error: cannot find a hook named pre-commit`, [git-secrets] has not \nbeen configured for the clone.\n\nIf you get no output, the AWS provider has not been registered.\n\n[git-secrets]: https://github.com/awslabs/git-secrets\n\n\n## 2.2 Runtime Prerequisites (Infrastructure)\n\nAn instance of the HCA [Data Store] aka DSS. The URL of that instance can be\nconfigured in `environment.py` or `deployments/*/environment.py`.\n\nThe remaining infrastructure is managed internally using TerraForm.\n\n\n## 2.3 Project configuration\n\nGetting started without attempting to make contributions does not require AWS\ncredentials. A subset of the test suite passes without configured AWS\ncredentials. To validate your setup, we'll be running one of those tests at the\nend.\n\n1. Load the environment defaults\n\n   ```\n   source environment\n   ```\n\n2. Activate the `dev` deployment:\n\n   ```\n   _select dev\n   ```\n\n3. Load the environment:\n\n   ```\n   source environment\n   ```\n\n   The output should indicate that the environment is being loaded from the\n   selected deployment (in this case, `dev`).\n\n4. Create a Python virtual environment and activate it:\n\n   ```\n   make virtualenv\n   source .venv/bin/activate\n   ```\n\n5. Install the development prerequisites:\n\n   ```\n   make requirements\n   ```\n\n   Linux users whose distribution does not offer the required Python version\n   should consider installing [pyenv] first, then Python using `pyenv install\n   x.y.z` and setting `PYENV_VERSION` to `x.y.z`, where `x.y.z` is the value of\n   `azul_python_version` in [environment.py](environment.py). You may need to\n   update [pyenv] itself before it recognizes the given Python version. Even if\n   a distribution provides the required minor version of Python natively, using\n   [pyenv] is generally preferred because it offers every patch-level release of\n   Python, supports an arbitrary number of different Python versions to be\n   installed concurrently and allows for easily switching between them.\n\n   Ubuntu users using their system's default Python installation must\n   install `python3-dev` before any wheel requirements can be built.\n\n   ```\n   sudo apt install python3-dev\n   ```\n\n   [pyenv]: https://github.com/pyenv/pyenv\n\n6. Run `make`. It should say `Looking good!` If one of the check target fails,\n   address the failure and repeat. Most check targets are defined in `common.mk`.\n\n7. Make sure Docker works without explicit root access. Run the following\n   command *without `sudo`*:\n\n   ```\n   docker ps\n   ```\n\n   If that fails, you're on your own.\n\n8. Finally, confirm that everything is configured properly on your machine by\n   running the unit tests:\n\n   ```\n   make test\n   ```\n\n### 2.3.1 GitHub credentials\n\nIntegration tests require a GitHub personal access token to be configured.\n\n1. Log into your account on https://github.com/. Click your user icon and \nnavigate to *Settings* -\u003e *Developer settings* -\u003e *Personal access tokens*\n\n2. Click *Generate new token*\n\n3. Enter an appropriate description such as \"Integration tests for Azul\"\n\n4. Select *No expiration*\n\n5. Do not select any scopes\n\n6. Click *Generate token* and copy the resulting token\n\n7. Edit the `deployments/.active/environment.local.py` file and modify the\n   `GITHUB_TOKEN` variable: \n\n   ```\n   'GITHUB_TOKEN': '\u003cthe token you just copied\u003e'\n   ```\n   \n   Do not add the token to any `environment.py` files.\n\n8. Repeat the previous step for any deployments you intend to use for running \n   the integration tests.\n\n### 2.3.2 AWS credentials\n\nYou should have been issued AWS credentials. Typically, those credentials\nrequire assuming a role in an account other than the one defining your IAM\nuser.  Just set that up normally in `~/.aws/config` and `~/.aws/credentials`.\nIf the  assumed role additionally requires an MFA token, you should run\n`_login`  immediately after running `source environment` or switching\ndeployments with  `_select`.\n\n\n### 2.3.3 Google Cloud credentials\n\nWhen it comes to Azul and Google Cloud, we distinguish between two types of\naccounts: an Azul deployment uses a *service account* to authenticate against\nGoogle Cloud and Azul developers use their *individual Google account* in a web\nbrowser. For the remainder of this section we'll refer to the individual Google\naccount simply as \"your account\". For developers at UCSC this is their\n`…@ucsc.edu` account.\n\nOn Slack, ask for your account to be added as an owner of the Google Cloud\nproject that hosts—or will host—the Azul deployment you intend to work with.\nFor the lower HCA DCP/2 deployments (`dev`, `sandbox` and personal deployments),\nthis is `platform-hca-dev`. The project name is configured via the\n`GOOGLE_PROJECT` variable in `environment.py` for each deployment.\n\n\n### 2.3.4 Google Cloud, TDR and SAM\n\n\nThe Terra ecosystem is tightly integrated with Google's authentication\ninfrastructure, and the same two types of accounts mentioned in the previous\nsection are used to authenticate against SAM and [Terra Data Repository]\n(TDR). Meaning that there are now at least two Google accounts at play:\n\n1) your individual Google account (\"your account\"),\n\n2) a service account for each shared or personal Azul deployment.\n\nYou use your account to interact with Google Cloud in general, along with both\nproduction and non-production instances of Terra, SAM, and TDR, provided you\nhave access. You also use your account for programmatic interactions with the\nabove systems and the Google Cloud resources they host, like the BiqQuery\ndatasets and GCS buckets that TDR manages. For programmatic access to the\nlatter, you can either `gcloud auth login` with your account or use the\n`service_account_credentials` context manager from `aws.deployment`.\n\n[Terra Data Repository]: https://jade.datarepo-dev.broadinstitute.org/\n\nIn order for an Azul deployment to index metadata stored in a TDR instance,\nthe Google service account for that deployment must be registered with SAM and\nauthorized for repository read access to datasets and snapshots. Additionally,\nin order for the deployment to accept unauthenticated servce requests, a second\nGoogle service account called the *public* account must likewise be registered\nand authorzied.\n\nThe SAM registration of the service accounts is handled automatically during\n`make deploy`. To register without deploying, run `make sam`. Mere\nregistration with SAM only provides authentication. Authorization to access\nTDR datasets and snapshots is granted by adding the registered service accounts\nto dedicated SAM groups (an extension of a Google group). This must be\nperformed manually by someone with administrator access to that SAM group. For\nnon-production instances of TDR, the indexer service account needs to be added\nto the group `azul-dev`.\n\nA member of the `azul-dev` group has read access to TDR. An *administrator* of\nthis group can add other accounts to it, and optionally make them\nadministrators, too. Before any account can be added to a group, it needs to be\nregistered with SAM. While `make deploy` does this automatically for the\ndeployment's service account, for your account, you must follow the steps below:\n\n\n1. Log into Google Cloud by running\n\n    ```\n    gcloud auth login\n    ```\n\n    A browser window opens to complete the authentication flow interactively.\n    When being prompted, select your account.\n\n    For more information refer to the Google authorization\n    [documentation](https://cloud.google.com/sdk/docs/authorizing).\n\n2. Register your account with SAM. Run\n\n    ```\n    (account=\"$(gcloud config get-value account)\"\n    token=\"$(gcloud auth --account $account print-access-token)\"\n    curl $AZUL_SAM_SERVICE_URL/register/user/v1  -d \"\" -H \"Authorization: Bearer $token\")\n    ```\n\n3. Ask an administrator of the `azul-dev` group to add your account to the\n   group. The best way to reach an administrator is via the `#team-boardwalk`\n   channel on Slack. Also, ask for a link to the group and note it in your\n   records.\n\n4. If you've already attempted to create your deployment via `make deploy`,\n   visit the link, sign in as your account and add your deployment's service\n   account to the group. Run `make deploy` again.\n\nFor production, use the same procedure, but substitute `azul-dev` with\n`azul-prod`.\n\n\n### 2.3.5 Creating a personal deployment\n\nCreating a personal deployment of Azul allows you test changes on a live system\nin complete isolation from other users. If you intend to make contributions,\nthis is preferred. You will need IAM user credentials to the AWS account you are\ndeploying to.\n\n1. Choose a name for your personal deployment. The name should be a short handle\n   that is unique within the AWS account you are deploying to. It should also be\n   informative enough to let others know whose deployment this is. We'll be\n   using `foo` as an example here. The handle must only consist of digits or\n   lowercase alphabetic characters, must not start with a digit and must be\n   between 2 and 16 characters long.\n\n2. Create a new directory for the configuration of your personal deployment:\n\n   ```\n   cd deployments\n   cp -r sandbox yourname.local\n   ln -snf yourname.local .active\n   mv .active/.example.environment.local.py .active/environment.local.py \n   cd ..\n   ```\n\n3. Read all comments in `deployments/.active/environment.py` and\n   `deployments/.active/environment.local.py` and make the appropriate edits.\n\n\n## 2.4 PyCharm\n\nRunning tests from PyCharm requires `environment` to be sourced. The easiest way\nto do this automatically is by installing `envhook.py`, a helper script that\ninjects the environment variables from `environment` into the Python interpreter\nprocess started from the project's virtual environment in `.venv`.\n\nTo install `envhook.py` run\n\n```\nmake envhook\n```\n\nThe script works by adding a `sitecustomize.py` file to your virtual\nenvironment. If a different `sitecustomize` module is already present in your\nPython path, its `sitecustomize.py` file must be renamed or removed before the\ninstallation can proceed. The current install location can be found by importing\n`sitecustomize` and inspecting the module's `__file__` attribute.\n\nWhether you installed `envook.py` or not, a couple more steps are necessary to\nconfigure PyCharm for Azul:\n\n1. Under *Settings* -\u003e *Project—Interpreter* select the virtual environment\n   created above.\n\n2. Set the `src` and `test` folders as source roots by right-clicking each\n   folder name and selecting *Mark Directory as* → *Sources Root*.\n\n3. Exclude the `.venv`, `lambdas/indexer/vendor`, and  `lambdas/service/vendor`\n   folders by right-clicking each folder name and selecting *Mark Directory as*\n   → *Excluded*.\n\nNewer versions of PyCharm install another `sitecustomize` module which attempts\nto wrap the user-provided one, in our case `envhook.py`. This usually works\nunless `envhook.py` tries to report an error. PyCharm's `sitecustomize` swallows\nthe exception and, due to a bug, raises different one. The original exception\nis lost, making diagnosing the problem harder. Luckily, the `sitecustomize`\nmodule is part of a rarely used feature that can be disabled by unchecking\n*Show plots in tool window* under *Settings* — *Tools* — *Python Scientific*.\n\n\n# 3. Deployment\n\n\n## 3.1 One-time provisioning of shared cloud resources\n\nMost of the cloud resources used by a particular deployment (personal or main\nones alike) are provisioned automatically by `make deploy`. A handful of\nresources must be created manually before invoking this Makefile target for\nthe first time in a particular AWS account. This only needs to be done once\nper AWS account, before the first Azul deployment is created in that account.\nAdditional deployments do not require this step.\n\n### 3.1.1 Versioned bucket for shared state\n\nCreate an S3 bucket for shared Terraform and Chalice state. The bucket must\nnot be publicly accessible since Terraform state may include secrets. If your\ndevelopers assume a role via Amazon STS, the bucket should reside in the same\nregion as the Azul deployment. This is because temporary STS AssumeRole\ncredentials are specific to a region and won't be recognized by an S3 region\nthat's different from the one the temporary credentials were issued in. The \nname of the bucket is not configurable but instead dictated by Azul's internal \nconvention for bucket names. Use the commands below to create that bucket.\n\n```\n_select dev.shared  # or prod.shared, anvildev.shared, anvilprod.shared …\nbucket=\"$(python -c 'from azul.deployment import aws; print(aws.shared_bucket)')\"\naws s3api create-bucket --bucket \"$bucket\"\naws s3api put-bucket-tagging \\\n          --bucket \"$bucket\" \\\n          --tagging TagSet=\"[{Key=owner,Value=$AZUL_OWNER}]\"\n```\n\n### 3.1.2 Route 53 hosted zones\n\nAzul uses Route 53 to provide user-friendly domain names for its services. The \nDNS setup for Azul deployments has historically been varied and rather \nprotracted. Azul's infrastrcture code will typically manage Route 53 records \nbut the zones have to be created manually.  \n\nCreate a Route 53 hosted zone for the Azul service and indexer. Multiple\ndeployments can share a hosted zone, but they don't have to. The name of the\nhosted zone is configured with `AZUL_DOMAIN_NAME`. `make deploy` will\nautomatically provision record sets in the configured zone, but it will not\ncreate the zone itself or register the  domain name it is associated with.\n\nOptionally, create a hosted zone for the DRS domain alias of the Azul service. \nThe corresponding environment variable is `AZUL_DRS_DOMAIN_NAME`. This feature \nhas not been used since 2020 when Azul stopped offering DRS for HCA.\n\nThe hosted zone(s) should be configured with tags for cost tracking. A list of\ntags that should be provisioned is noted in\n[src/azul/deployment.py:tags](src/azul/deployment.py).\n\n### 3.1.3 AWS Chatbot integration with Slack\n\nAzul deployments can make use of an AWS Chatbot instance to forward messages\nfrom the SNS monitoring topic to a channel in a Slack workspace. Both the topic\nand the Chatbot instance are shared by all deployments that are collocated in\none AWS account and that have monitoring enabled via the\n`AZUL_ENABLE_MONITORING` environment variable. Most of the AWS Chatbot\nintegration is [managed by Terraform](#314-shared-resources-managed-by-terraform)\nbut the following manual steps must be performed once per AWS account containing\nsuch deployments, before Terraform can take care of the rest. The AWS Chatbot\nintegration can be enabled or disabled separately for each AWS account by\nsetting the `azul_slack_integration` environment variable in the configuration\nfor the main deployment in that account. If it is disabled in an account, these\nsteps can be skipped in that account.\n\n1. In the AWS Chatbot console, under *Configure a chat client*, select the\n   *Slack* chat client option, then click the *Configure client* button.\n   \n2. Once redirected to Slack's authorization page, you may be prompted to sign\n   in using your UCSC account, in order to provide permission for Chatbot to\n   access the Slack workspace. When this step is completed, you should see the\n   workspace name and ID listed in the console.\n\n3. Use the ID displayed in the console to set the `workspace_id` attribute of\n   the `azul_slack_integration` variable in the main deployment's environment\n   file for that account.\n\n4. Set the `channel_id` attribute to the ID of the appropriate channel. Get the\n   channel ID by right-clicking the channel in Slack and selecting *View channel\n   details*. The ID is listed at the bottom of the *About* tab.\n\n### 3.1.4 Shared resources managed by Terraform\n\nThe remaining resources for each of the AWS accounts hosting Azul deployments\nare provisioned through Terraform. The corresponding resource definitions reside\nin a separate *Terraform component*.\n\nA Terraform component is a set of related resources. It is our own bastardized\nform of Terraform's *module* concept, aimed at facilitating encapsulation and\nreuse. Each deployment has at least a main component and zero or more child\ncomponents. The main component is identified by the empty string for a name;\nchild components have a non-empty name. The `dev` component has a child\ncomponent `dev.shared`. To deploy the main component of the `dev` deployment, \none selects the `dev` deployment and runs `make apply` from \n`${project_root}/terraform` (or `make deploy` from the project root). To deploy \nthe `shared` child component of the `dev` deployment, one selects `dev.shared` \nand runs `make apply` from `${project_root}/terraform/shared`. In other words, \nthere is one generic set of resource definitions for a child component, but \nmultiple concrete deployment directories.\n\nThere are currently two Terraform components: `shared` and `gitlab`. \nInterestingly, not every deployment uses these components. Typically, only the \n`dev` and `prod` deployments use them. The other deployment share them with \n`dev` or `prod`, depending on which of those deployments they are colocated \nwith. Two deployments are colocated if they use the same AWS account. The \n`shared` component contains the resources shared by all deployments in an AWS \naccount.\n\nTo deploy the remaining shared resources, run: \n\n```\n_select dev.shared  # or prod.shared, anvildev.shared, anvilprod.shared …\ncd terraform/shared\nmake validate\nbucket=\"$(python -c 'from azul.deployment import aws; print(aws.shared_bucket)')\"\nterraform import aws_s3_bucket.shared \"$bucket\"\nmake\n```\n\nThe invocation of `terraform import` puts the bucket we created \n[earlier](#311-versioned-bucket-for-shared-state) under management by Terraform.\n\n### 3.1.5 GitLab\n\nA self-hosted GitLab instance is provided by the `gitlab` TerraForm component. \nIt provides the necessary CI/CD infrastructure for one or more Azul deployments \nand protects access to that infrastructure through a VPN. That same VPN is also\nused to access to Azul deployments with private APIs (see AZUL_PRIVATE_API in \n[environment.py]). Like the `shared` component, the `gitlab` component belongs \nto one main deployment in an AWS account (typically `dev` or `prod`) and is \nshared by the other deployments colocated with that deployment.\n\n[environment.py]: /environment.py\n\nThe following resources must be created manually before deploying the `gitlab` \ncomponent:\n\n- An EBS volume needs to be created. See [gitlab.tf.json.template.py] and the\n  [section on CI/CD](#85-storage) for details.\n\n- A certificate authority must be set up for VPN access. For details refer to\n  [section on GitLab CA](#812-setting-up-the-certificate-authority).\n\n\n## 3.2 One-time manual configuration of deployments\n\nIn order for users to authenticate using OAuth 2.0, an OAuth 2.0 consent screen\nmust be configured once per Google project, and an OAuth 2.0 client ID must\nbe created for each deployment.\n\n### 3.2.1 Google OAuth 2.0 consent screen\n\nThese steps are performed once per Google project.\n\n1. Log into the Google Cloud console and select the desired project, e.g. `dev` \n   or `prod`\n\n2. Navigate to *APIs \u0026 Services* -\u003e *OAuth Consent Screen*\n\n3. Click *CONFIGURE CONSENT SCREEN*\n\n4. For *User Type*, select *External*\n\n5. Click *CREATE*\n\n6. For *App name*, enter `Azul {stage}`, where `{stage}` is the last component\n   of the Google project name, e.g. `dev` or `prod`\n\n7. Provide appropriate email addresses for *App information* -\u003e \n   *User support email* and *Developer contact information* -\u003e \n   *Email addresses*, e.g. `azul-group@ucsc.edu`\n\n8. Click *SAVE AND CONTINUE*\n\n9. For scopes, select:\n   ```\n   https://www.googleapis.com/auth/userinfo.email\n   https://www.googleapis.com/auth/userinfo.profile\n   openid\n   ```\n\n10. Click *SAVE AND CONTINUE* twice\n\n11. Click *PUBLISH APP* and *CONFIRM*\n\n### 3.2.2 Google Oauth 2.0 Client ID\n\nThese steps are performed once per deployment (multiple times per project).\n\n1. Log into the Google Cloud console and select the desired project, e.g. `dev`\n   or `prod`\n   \n2. Navigate to *APIs \u0026 Services* -\u003e *Credentials*; click *+ CREATE CREDENTIALS*\n   -\u003e *OAuth Client ID*\n\n3. For *Application Type*, select *Web application*\n\n4. For *Name*, enter `azul-{stage}` where stage is the name of the deployment\n\n5. Add an entry to *Authorized JavaScript origins* and enter the output from\n   `python3 -m azul config.service_endpoint`\n\n6. Add an entry to *Authorized redirect URIs*. Append `/oauth2_redirect` to the\n    value of the previous field and enter the resulting value.\n   \n7. Click *Create*\n\n8. Copy the OAuth Client ID (_not_ the client secret) and insert it into the\n    deployment's `environment.py` file:\n\n    ```\n    'AZUL_GOOGLE_OAUTH2_CLIENT_ID': 'the-client-id'\n    ```\n\n9. `_refresh`\n\n## 3.3 Provisioning cloud infrastructure\n\nOnce you've configured the project and your personal deployment or a shared\ndeployment you intend to create, and once you manually provisioned\nthe shared cloud resources, it is time to provision the cloud infrastructure\nfor your deployment. Run\n\n```\nmake deploy\n```\n\nto prepare the Lambda functions defined in the `lambdas` directory for\ndeployment via Terraform. It will display a plan and ask you to confirm it.\nPlease consult the Terraform documentation for details.\n\nAny time you wish to change the code running in the lambdas you will need to\nrun `make deploy`.\n\nSome Terraform configuration is generated by `make -C lambdas`, but the rest is\ndefined in `….tf.json` files which in turn are generated from\n`….tf.json.template.py` templates which are simple Python scripts containing the\ndesired JSON as Python dictionary and list literals and comprehensions.\nRunning `make deploy` will run `make -C lambda` and also expand the\ntemplate files. Changes to either the templates or anything in the `lambdas`\ndirectory requires running `make deploy` again in order to update cloud\ninfrastructure for the selected deployment.\n\n\n## 3.4 Creating the Elasticsearch indices\n\nWhile `make deploy` takes care of creating the Elasticsearch domain, the actual\nElasticsearch indices for the selected deployment must be created by running\n\n```\nmake create\n```\n\nIn a newly created deployment, the indices will be empty and requests to the\ndeployment's service REST API may return errors. To fill the indices,\ninitiate a [reindexing](#37-reindexing). In an existing deployment\n`make create` only creates indices that maybe missing. To force the recreation\nof indices run `make delete create`.\n\n## 3.5 Locating REST API endpoints via DNS\n\nThe HTTP endpoint offered by API Gateway have somewhat cryptic and hard to\nremember domain names:\n\n```\nhttps://klm8yi31z7.execute-api.us-east-1.amazonaws.com/hannes/\n```\n\nFurthermore, the API ID at the beginning of the above URL is likely to change\nany time the REST API is re-provisioned.  To provide stable and user-friendly\nURLs for the API lambdas, we provision a *custom domain name* object in API\nGateway along with an ACM certificate and a CNAME record in Route 53. the\nuser-friendly domain names depend on project configuration. The default for HCA\nis currently\n\n```\nhttp://indexer.${AZUL_DEPLOYMENT_STAGE}.singlecell.gi.ucsc.edu/\nhttp://service.${AZUL_DEPLOYMENT_STAGE}.singlecell.gi.ucsc.edu/\n```\n\nPersonal deployments are subdomains of the domain for the `dev` deployment:\n\n```\nhttp://indexer.${AZUL_DEPLOYMENT_STAGE}.dev.singlecell.gi.ucsc.edu/\nhttp://service.${AZUL_DEPLOYMENT_STAGE}.dev.singlecell.gi.ucsc.edu/\n```\n\n## 3.6 Private API\n\nFollow these steps to put a deployment's API Gateway in the GitLab VPC so that a\nVPN connection is required to access the deployment. See [8.1 VPN access to\nGitLab](#81-vpn-access-to-gitlab) for details. Read this entire section before\nfollowing these steps.\n\n1. Destroy the current deployment (`make -C terraform destroy`).\n\n2. Increment `AZUL_DEPLOYMENT_INCARNATION`.\n \n3. Set `AZUL_PRIVATE_API` to `1`.\n \n4. Redeploy (`make deploy`).\n\nGoing in the opposite direction i.e., attempting to change `AZUL_PRIVATE_API`\nfrom `1` to `0` will result in `Cannot update endpoint from PRIVATE to EDGE`\nduring `make deploy`. The error message will be shown for every REST API\nseparately. It should be sufficient to simply `terraform taint` the REST API\nresources mentioned in the error messages and then to run `make deploy` again.\nIt is possible that this also works when changing `AZUL_PRIVATE_API` from `0` to\n`1`. Try that first, before destroying the entire deployment.\n\n### Troubleshooting\n\nTransient errors might be encountered during the deploy such as `SQS Error Code:\nAWS.SimpleQueueService.NonExistentQueue. SQS Error Message: The specified queue\ndoes not exist for this wsdl version` In such cases rerunning `make deploy`\nshould resolve the issue.\n\n[aws_cloudwatch_log_group]: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group\n\nIf the error `ResourceAlreadyExistsException: The specified log group already\nexists` is encountered, follow the steps below to import the\n[aws_cloudwatch_log_group] resources into terraform and retry the deploy.\n\n1. `cd terraform`\n\n2. `terraform import aws_cloudwatch_log_group.indexer /aws/apigateway/azul-indexer-foo`\n \n3. `terraform import aws_cloudwatch_log_group.service /aws/apigateway/azul-service-foo`\n \n4. `cd ..`\n \n5. `make deploy`\n\nIf the error `azul.RequirementError: The service account (SA) '...' is not\nauthorized to access ... or that resource does not exist. Make sure that it \nexists, that the SA is registered with SAM and has been granted read access to \nthe resource` is encountered, ask an administrator of the Terra group `azul-dev` \nto add the service account as specified in the error messaged to that group. See\n[2.3.4 Google Cloud, TDR, and SAM](#234-google-cloud-tdr-and-sam) for details.\n\n[KMSAccessDeniedException]: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-kmsaccessdeniedexception-errors/\n\nAfter a successful invocation of `make deploy`, if the deployment is unresponsive\nand CloudWatch shows logs entries in the `/aws/apigateway/…` log group but not in\n`/aws/lambda/…`, first confirm whether the issue is the known\n[KMSAccessDeniedException] error. In the AWS Console, go to the Lambda function\ndetails page, click on the `Test` tab, and click on the `Test` buttton. \n\nNote that it is normal for some Lambda functions to fail the test due to the\nparameters of the test event. Examine the error message to determine if the\nfailure is due to a `KMSAccessDeniedException` which would be explicitly\nspecified.\n\nTo resolve a `KMSAccessDeniedException` run the `reset_lambda_role.py` script to\nreset all the Lambda functions in the selected deployment.\n\n## 3.7 Reindexing\n\nThe DSS instance used by a deployment is likely to contain existing bundles. To\nindex them run:\n\n```\nmake reindex\n```\n\nWhen reindexing, artificial notifications are generated by Azul.\n\nThe `reindex` make target will purchase a BigQuery slot commitment if:\n\n1. No slot commitment is currently active, and\n2. At least one catalog being indexed uses the TDR repository plugin.\n\nTo avoid cost-ineffective slot purchases, the `reindex_no_slots` target should be\nused instead of `reindex` if the reindexing is expected to complete in 15\nminutes or less.\n\n## 3.8 Cancelling an ongoing (re)indexing operation\n\n```\npython scripts/manage_queues.py purge_all\n```\n\nAfter that it is advisable to delete the indices and reindex at some later time.\n\n## 3.9 Deleting all indices\n\nTo delete all Elasticsearch indices run\n\n```\nmake delete\n```\n\nThe indices can be created again using\n\n```\nmake create\n```\n\nbut they will be empty.\n\n## 3.10 Deleting a deployment\n\n\n1. `cd` to the project root, then\n\n   ```\n   source environment\n   ```\n\n2. Select the deployment to deleted\n\n   ```\n   _select foo.local\n   ```\n\n3. Delete all Elasticsearch indices in the selected deployment\n\n   ```\n   make delete\n   ```\n\n4. Delete the API Gateway base path mappings\n\n   ```\n   cd terraform\n   make init\n   terraform destroy $(terraform state list | grep aws_api_gateway_base_path_mapping | sed 's/^/-target /')\n   cd ..\n   ```\n\n5. Destroy cloud infrastructure\n\n   ```\n   make -C terraform destroy\n   ```\n\n   The destruction of `aws_acm_certificate` resources may time out. Simply\n   repeat this step until it succeeds.\n\n6. From the shared bucket (run `python -c 'from azul.deployment import aws; \n   print(aws.shared_bucket)'` to reveal its name), delete all keys relating to \n   your deployment.\n\n7. Delete the local Terraform state file at\n   `deployments/.active/.terraform.{$AWS_PROFILE}/terraform.tfstate`.\n\n\n# 4. Running indexer or service locally\n\nWhile this method *does* run the service or indexer locally on your machine, it\nstill requires that the cloud resources used by them are already deployed.\nSee sections [2](#2-getting-started) and [3](#3-deployment) on how to do that.\n\n1. As usual, activate the virtual environment and `source environment` if you\n   haven't done so already\n\n2. `cd lambdas/service`\n\n3. Run\n\n   ```\n   make local\n   ```\n\n4. You can now hit the app under `http://127.0.0.1:8000/`\n\nPyCharm recently added a feature that allows you to attach a debugger: From the\nmain menu choose *Run*, *Attach to local process* and select the `chalice`\nprocess.\n\n\n# 5. Troubleshooting\n\n\n## `Error: Invalid index` during `make deploy`\n\n```\naws_route53_record.service_0: Refreshing state... [id=XXXXXXXXXXXXX_service.dev.singlecell.gi.ucsc.edu_A]\n Error: Invalid index\n   on modules.tf.json line 8, in module.chalice_indexer.es_endpoint:\n    8:                 \"${aws_elasticsearch_domain.index.endpoint}\",\n     |----------------\n     | aws_elasticsearch_domain.index is empty tuple\n The given key does not identify an element in this collection value.\n```\n\nThis may be an [issue](https://github.com/hashicorp/terraform/issues/25784) with\nTerraform. To work around this, run …\n\n```\nterraform state rm aws_elasticsearch_domain.index\n```\n\n… to update the Terraform state so that it reflects the deletion of the\nElasticsearch domain. Now running `make deploy` should succeed.\n\n\n## `NoCredentialProviders` while running `make deploy`\n\nIf you get …\n\n```\nFailed to save state: failed to upload state: NoCredentialProviders: no valid providers in chain.\n…\nThe error shown above has prevented Terraform from writing the updated state\nto the configured backend. To allow for recovery, the state has been written\nto the file \"errored.tfstate\" in the current working directory.\n\nRunning \"terraform apply\" again at this point will create a forked state,\nmaking it harder to recover.\n```\n\n… during `make deploy`, your temporary STS credentials might have expired while\n`terraform apply` was running. To fix, run …\n\n```\n_login\n(cd terraform \u0026\u0026 terraform state push errored.tfstate)\n```\n\n… to refresh the credentials and upload the most recent Terraform state to the\nconfiguration bucket.\n\n\n## `AccessDeniedException` in indexer lambda\n\nIf you get the following exception:\n```\nAn error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::{account_id}:assumed-role/azul-indexer-{deployment_stage}/azul-indexer-{deployment_stage}-index is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:{aws_region}:{account_id}:parameter/dcp/dss/{deployment_stage}/environment: ClientError\nTraceback (most recent call last):\n    ...\nbotocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::{account_id}:assumed-role/azul-indexer-{deployment_stage}/azul-indexer-{deployment_stage}-index is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:{aws_region}:{account_id}:parameter/dcp/dss/integration/environment\n```\n\nCheck whether the DSS switched buckets. If so, the lambda policy may need to be\nupdated to reflect that change. To fix this, redeploy the lambdas (`make\npackage`) in the affected deployment.\n\n\n## `make requirements_update` does not update transitive requirements\n\nIn some cases, `make requirements_update` might not produce any updates to\ntransitive requirements, even if you expect them. For example, a sandbox build\non Gitlab might identify updated transitive requirements even though doing `make\nrequirements_update` locally doesn't.\n\nThis is a side effect of the Docker build cache on two different machines\ndiverging to reflect different states on PyPI. This can be fixed by incrementing\n`azul_image_version` in the Dockerfile.\n\n\n##  Unable to re-register service account with SAM\n\nIf you have destroyed your deployment and are rebuilding it, it's possible that\nSAM will not allow the Google service account to be registered again because\nthe service account's email is the same in the current and previous incarnation\nof the deployment, while the service account's `uniqueID` is different. SAM\ndoes not support this.\n\nA warning message stating that `SAM does not allow re-registration of service\naccount emails` will be visible during the `make sam` step of the deployment\nprocess. To get around this, increment the current value of\n`AZUL_DEPLOYMENT_INCARNATION` in the deployment's `environment.py` file, then\nredeploy.\n\n\n## Unexpected warnings cause tests to fail in `tearDownClass`\n\nUnexpected warnings that occur during testing will cause failures in \n`AzulTestCase.tearDownClass`. There is a context manager in `AzulTestCase` that\nkeeps record of emitted warnings during test execution. Due to the unit test \ndiscovery process loading modules as it traverses directories, it’s possible \nthat a warning is emitted outside the scope of the context manager.\n\nIn the two commands below, the unit test discovery process occurs within a \ndifferent directory.\n\n```\n$ (cd test \u0026\u0026 python -m unittest service.test_app_logging.TestServiceAppLogging)\n```\n\nIn the first case, it's possible that an unpermitted warning is emitted outside \nthe `AzulTestCase` context manager, due to modules being loaded recursively from\nthe directory `test/`. If a warning is emitted outside the context manager no \ntest failure will occur.\n\n```\n$ (cd test/service \u0026\u0026 python -m unittest test_app_logging.TestServiceAppLogging)\n```\n\nIn the second case, the test discovery process loads fewer modules due to the  \nnarrowed working directory. This may emit a warning during test execution, \nenabling the context manager to catch the unpermitted warning, and fail \nappropriately.\n\nSimilarly, when running tests in PyCharm, its own proprietary test discovery \nprocess may also increase the chance of the `AzulTestCase` context manager\ncausing a failure.\n\nIf these failures occur, add the warning to the list of permitted warnings\nfound in [`AzulTestCase`](test/azul_test_case.py) and commit the modifications. \n\n\n## Setting up the Azul build prerequisites on macOS 12 (Monterey)\n\nThe steps below are examplary for Python 3.12.7. Replace `3.12.7` with the value\nof `azul_python_version` in [environment.py](environment.py).\n\nMake `bash` the default shell. Google it.\n\nInstall Homebrew. Google it. \n\nInstall pyenv:\n\n```\nbrew install zlib pyenv\n```\n\nInstall python\n\n```\npyenv install 3.12.7\n```\n\nSet `PYENV_VERSION` to `3.12.7` in `environment.local.py` at the project root.\nDo not set `SYSTEM_VERSION_COMPAT`. For a more maintainable configuration use \n`os.environ['azul_python_version']` as the value and `import os` at the top.\n\nInstall Docker Desktop. Google it.\n\nInstall Terraform by downloading and unziping the binary to a directory on the \n`PATH`. Be sure to download the file for the architecture of your Mac. For Apple \nSilicon the file name contains `arm64`, for older Intel Macs it's `amd64`.\n\n\n## Installing Python 3.8.12 on macOS 11 (Big Sur)\n\n[pyenv macOS 11 GitHub issue](https://github.com/pyenv/pyenv/issues/1740)\n\nUsers of macOS 11 or later may encounter a `build failed` error when installing\nPython through pyenv. A patch was made available to remedy this:\n\nFirst, ensure that bzip2 and any other requirements for the Python build\nenvironment are met. See [pyenv wiki] for details:\n\n[pyenv wiki]:https://github.com/pyenv/pyenv/wiki#suggested-build-environment\n\n```\nbrew install openssl readline sqlite3 xz zlib bzip2\n```\n\nFollow any additional steps that `brew` prompts for at the end of the\ninstallation. These should include modifying path variables `LDFLAGS` and\n`CPPFLAGS`. The commands from the `brew` output to modify the aforementioned\npath variables can be placed in `~/.bash_profile` to make the change persistent.\n\nThen install Python 3.8.12 using `pyenv` by running:\n\n```\npyenv install 3.8.12\n```\n\nUsers of macOS 11 or later may encounter `pip` installation errors due to `pip`\nnot being able to locate the appropriate wheels. The information below will\nhelp remedy this:\n\n[Resolution source](https://stackoverflow.com/a/63972598)\n\n[macOS 11 Release Notes](https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-release-notes#Third-Party-Apps)\n\n`pip` will not be able to locate the appropriate wheels due to the major release\nversion of macOS being incremented from `10.x` to `11.x`, instead pip will\nattempt to compile wheels manually for wheels that it cannot locate.\n\nIn order to be able to run `make requirements` successfully, a backwards\ncompatibility flag needs to be added to the `environment.local.py` file in the\nproject root. The flag is `SYSTEM_VERSION_COMPAT=1` and it needs to be inserted\ninto the file (starting from line 25) as a key/value pair:\n`'SYSTEM_VERSION_COMPAT': 1`.\n\n\n# 6. Branch flow \u0026 development process\n\n**This section should be considered a draft. It describes a future extension to the current branching flow.**\n\nThe section below describes the flow we want to get to eventually, not the one\nwe are currently using while this repository recovers from the aftermath of its\ninception.\n\nThe declared goal here is a process that prevents diverging forks yet allows\neach project to operate independently as far as release schedule, deployment\ncadence, project management and issue tracking is concerned. The main challenges\nare 1) preventing contention on a single `develop` or `master` branch, 2)\nisolating project-specific changes from generic ones, 3) maintaining a\nreasonably linear and clean history and 4) ensuring code reuse.\n\nThe [original repository](https://github.com/DataBiosphere/azul), also known as\n*upstream*, should only contain generic functionality and infrastructure code.\nProject-specific functionality should be maintained in separate project-specific\nforks of that repository. The upstream repository will only contain a `master`\nbranch and the occasional PR branch.\n\nAzul dynamically imports project-specific plugin modules from a special location\nin the Python package hierarchy: `azul.projects`. The package structure in\nupstream is\n\n```\nroot\n├── ...\n├── src\n│   └── azul\n│       ├── index\n│       │   └── ...\n│       ├── projects (empty)\n│       ├── service\n│       │   └── ...\n│       └── util\n│       │   └── ...\n└── ...\n```\n\nNote that the `projects` directory is empty.\n\nThe directory structure in forked repositories is generally the same with one\nimportant difference. While a fork's `master` branch is an approximate mirror of\nupstream's `master` and therefore also lacks content in `projects`, that\ndirectory *does* contain modules in the fork's `develop` branch. In\n`HumanCellAtlas/azul-hca`, the fork of Azul for the HumanCellAtlas project, the\n`develop` branch would look like this:\n\n\n```\nroot\n├── ...\n├── src\n│   └── azul\n│       ├── index\n│       │   └── ...\n│       ├── projects\n│       │   └── hca\n│       │       └── ...\n│       ├── service\n│       │   └── ...\n│       └── util\n│       │   └── ...\n└── ...\n```\n\nThe `develop` branch would only contain changes to the `azul.projects.hca`\npackage. All other changes would have to be considered generic—they would occur\non the fork's `master` branch and eventually be merged into upstream's `master`\nbranch. The `master` branches in each fork should not be divergent for sustained\nperiods of time while the project-specific branches can and will be.\n\nThe reason why each fork maintains a copy of the `master` branch is that forks\ngenerally need to have a place to test and evaluate generic features before they\nare promoted upstream. If there wasn't a `master` branch in a fork, the\nproject-specific `develop` branch in that fork would inevitably conflate\nproject-specific changes with generic ones. It would be very hard to selectively\npromote generic changes upstream, even if the generic changes were separate\ncommits.\n\nThe flow presented here establishes an easy-to-follow rule: If you're modifying\n`azul.projects.hca`, you need to do so in a PR against `develop`. If you're\nmodifying anything else, you need to do so in a PR against `master`. The figure\nbelow illustrates that.\n\n```\n                                                      ●────● feature/generic-foo\n                                                     ╱\n                                              4     ╱\n    ─────●────────────────────────────────────●────●──────────────        master\n          ╲                                  ╱\n azul      ╲                                ╱\n ─ ─ ─ ─ ─ ─╲─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ╱ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─\n azul-hca    ╲                            ╱\n              ╲                          ╱\n    ──────●────●────●────●────●────●────●──────────────────────────       master\n           ╲   1     ╲    ╲   A'   B'\n            ╲         ╲    ╲\n             ╲         ╲    ●────● feature/master/generic-stuff\n              ╲         ╲   A    B\n               ╲         ╲\n                ●─────────●─────────────●────●────●─────────────────     develop\n                2         3              ╲   C'   D'\n                                          ╲\n                                           ●────● feature/develop/specific-stuff\n                                                C    D\n```\n\nMerge commit 1 from the upstream `master` branch integrates upstream changes\ninto the fork. These may be generic changes merged upstream from other forks or\nchanges that were directly PR-ed against `master` in upstream. Commit 2 marks\nthe beginning of the `develop` branch, adding the `azul.projects.hca` package.\nMerge commit 3 brings the changes from commit 1 into the `develop` branch.\n\nAnother important rule is that collaborative branches like `develop` and\n`master` are never rebased. Changes are exchanged between them using merge\ncommits instead. Individual branches however, like feature branches, are always\nrebased onto the base branch. In the above example,\n`feature/master/generic-stuff` is first rebased onto `master`, creating commits\nA' and B'. Later those changes are merged upstream via commit 4. Both the rebase\nand the merge happen via a pull request, but the landing action will be \"Rebase\nand merge\" for the first PR and \"Create a merge commit\" for the second.\n\nThe reason for this distinction is that rebasing usually triggers more rebasing\nof branches that were based on the rebased branch. It also rewrites the commit\ntimestamps, thereby obfuscating the history to some extent. For these two\nreasons, rebasing is not a sustainable practice for collaborative branches. For\nindividual branches however, rebasing is possible because feature branches are\ntypically not used as a base for other branches. Rebasing is also desirable\nbecause it produces a cleaner, linear history and we should use it whenever\npossible. The back and forth merging between collaborative branches produces a\nhistory that's somewhat convoluted so it is important to keep the history as\nclean as possible in between merges.\n\nGeneric changes don't have to be conceived in a fork. We can also PR them\ndirectly against the upstream repository as illustrated by branch\n`feature/generic-foo`.\n\nThe most common type of pull request in a fork is one against that fork's\n`develop` branch, `feature/develop/specific-stuff` for example. Note that\nchanges occurring on `develop` are never merged upstream.\n\nAs mentioned before, merge commit 4 is done via a pull request against the\nupstream repository. It is possible and perfectly acceptable that such upstream\nPRs combine multiple unrelated changes. They should be requested by the team\nlead for the forking project and reviewed by an upstream lead. Shortly after the\nPR lands, the requesting lead should perform a fast-forward merge of the\nupstream `master` branch into the fork's `master` branch. This will propagate\nthe merge commit downstream before any subsequent commits occurring on fork's\n`master` have a chance to complicate the history by introducing the infamous\nmerge of merge commits.\n\n```\n$ git branch\n* master\n  develop\n$ git merge --ff-only upstream/master\nUpdating 450b0c0..212003c\nFast-forward\n```\n\nThis procedure requires that the lead's local clone of the fork be set up with\ntwo remotes: `origin` (the forked repository) and `upstream` (the upstream\nrepository). Other team members can usually get by with just one remote,\n`origin`.\n\n\n## 6.1 Deployment branches\n\nThe code in the upstream repository should never be deployed anywhere because it\ndoes not contain any concrete modules to be loaded at runtime. The code in a\nfork, however, is typically active in a number of deployments. The specifics\nshould be left to each project but the rule of thumb should be that each\ndeployment corresponds to a separate branch in the fork. The `azul-hca` fork has\nfour deployments: development, integration, staging and production. The\ndevelopment deployment, or `dev`, is done from the `develop` branch. Whenever a\ncommit is pushed to that branch, a continuous deployment script deploys the code\nto AWS. The other deployment branches are named accordingly. Changes are\npromoted between deployments via a merge. The merge is likely going to be a\nfast-forward. A push to any of the deployment branches will trigger a CI/CD\nbuild that performs the deployment. The promotion could be automatic and/or\ngated on a condition, like tests passing.\n\n\n# 7. Operational Procedures\n\n\n## 7.1 Main deployments and promotions\n\nWe will refer to the branch of the stage to which you are deploying as the\n**`TARGET`** branch. The branch of the stage just below will be referred to as\nthe **`SOURCE`** branch.\n\nThis cheat sheet may differ from branch to branch. Be sure to follow the cheat\nsheet in the README on the branch currently checked out.\n\n\n### 7.1.1 Initial setup\n\n[Gitlab instance]: https://gitlab.dev.singlecell.gi.ucsc.edu/\n\n_Note: You can skip this step if you've deployed or promoted with Gitlab at\nleast once already._\n\n[SSH keys]: https://gitlab.dev.singlecell.gi.ucsc.edu/profile/keys\n\n1. For promotion, we recommend keeping a separate clone of Azul that is never in\n   a dirty state. To create this if it doesn't yet exist run\n\n   ```\n   git clone git@github.com:DataBiosphere/azul.git azul.stable\n   ```\n\n   Then follow the setup instructions in [2.3 Project configuration](#23-project-configuration).\n\n2. Next you will need to login to our [Gitlab instance] in order to be able to\n   push to Gitlab which automatically takes care of most of the deployment\n   process. If you haven't signed on yet, sign on with Github. You will need at\n   least `developer` permissions in order to be able to `push` to Gitlab.\n   Contact the team lead if you have problems signing on or have insufficient\n   permissions.\n\n3. Deposit you public SSH key into the [SSH keys] section of your profile so\n   that you can push to Git repositories hosted on that Gitlab instance.\n\n4. Now that your SSH key is set up, you will need to add Gitlab as a remote. Run\n\n   ```\n   git remote add gitlab.dev git@ssh.gitlab.dev.singlecell.gi.ucsc.edu:ucsc/azul.git\n   ```\n\n   Run\n\n   ```\n   git fetch gitlab.dev\n   ```\n\n   to ensure that your connection is working.\n\nIf you have been given write access to our production Gitlab instance, you need\nto repeat these steps for that instance as well. For the name of the `git`\nremote use `gitlab.prod` instead of `gitlab.dev` in step 4 above. The hostname\nof that instance is the same as that of the Gitlab instance for the lesser\ndeployments, without `.dev`.\n\nNote that access to the production instance of Gitlab does not necessarily\nimply access to production AWS account which that Gitlab instance deploys to.\nSo while you may be able to run certain `make` targets like `make reindex` or\n`make deploy` against the development AWS account (with `dev`, `integration`\nor `staging` selected), you may not be able to do the same for the production\nAWS account (with `prod` selected).\n\n\n### 7.1.2 Prepare for promotion\n\n_NOTE: Skip these steps if you are deploying without promoting changes._\n\n[DCP release SOP]: https://allspark.dev.data.humancellatlas.org/dcp-ops/docs/wikis/SOP:%20Releasing%20new%20Versions%20of%20DCP%20Software\n\n_NOTE: If promoting to `staging` or `prod` you will need to do these steps **at least\n24 hours in advance** so that the release notes are ready in time._\n\n1. From the `azul.stable` clone make sure all of the relevant branches are up to\n   date\n\n   ```\n   cd azul.stable\n   git checkout SOURCE\n   git pull\n   git checkout TARGET\n   git pull\n   ```\n\n2. You should be on the `TARGET` branch. Run\n\n   ```\n   git merge --no-ff SOURCE\n   ```\n\n   and resolve conflicts in necessary. Conflict resolution should only be\n   necessary if cherry-picks occurred on the target branch.\n\n3. The merge may have affected `README.md`, the file you are looking at right\n   now. Reopen the file now to ensure you are following the updated version.\n\n4. Now you need to create the release notes. (Skip this step if no link to the release\n   notes document can be found either in the #dcp-ops channel on HCA Slack or in\n   the Google Drive folder mentioned in the [DCP release SOP].\n\n   To produce the list of changes for the DCP release notes, first find the\n   previous release tag for the TARGET branch. Then run:\n\n   ```\n   git log LAST_RELEASE_TAG..HEAD --format=\"%C(auto) %h %s\" --graph\n   ```\n   Edit this output so that the commits within merged branches are removed, along with\n   merge commits between deployments. For example\n   ```\n   *  C  \u003c-- merge commit\n   |\\\n   | *  B\n   |/\n   *  A\n   *  Merge branch 'develop' into integration\n   ```\n   should be changed to look like\n   ```\n   *  C  \u003c-- merge commit\n   *  A\n   ```\n\n   For the version, use the full hash of the latest commit:\n\n   ```\n   git log -1 --format=\"%H\"\n   ```\n\n5. At this point you should determine whether or not you will need to reindex.\n   The `CHANGELOG.yml` _should_ contain this information but is notoriously\n   unreliable. Try running\n\n   ```\n   git diff LAST_RELEASE_TAG..HEAD src/azul/project/ src/azul/indexer.py src/azul/plugin.py src/azul/transformer.py\n   ```\n\n   where `LAST_RELEASE_TAG` is the previous release of the target branch. If the diff\n   contains non-trivial changes reindexing is probably necessary. When in doubt\n   assume yes.\n\n\n### 7.1.3 Finishing up deployment / promotion\n\nIf promoting to staging or production this part of the process must be\ncoordinated on the\n[#dcp-ops](https://humancellatlas.slack.com/messages/G9XD6L0AD) Slack channel.\nWhile any component can technically promote to integration at any time, you\nshould consider that promoting to integration while the DCP-wide test is red\nfor that deployment could interfere with other teams' efforts to fix the test.\nIf in doubt ask on #dcp-ops.\n\nNone of these steps can be performed ahead of time. Only perform them once you\nare ready to actually deploy.\n\n1. Activate your virtual environment and run\n   ```\n   source environment\n   ```\n   and then select the target deployment stage with\n   ```\n   _select STAGE\n   ```\n   where stage is one of `dev`, `integration`, `staging`, or `prod`\n\n2. Now you need to push the current branch to Github. This is needed because\n   the Gitlab build performs a status check update on Github. This would fail\n   if Github didn't know the commit.\n\n   ```\n   git push origin\n   ```\n\n3. Finally, push to Gitlab.\n\n   ```\n   git push gitlab.dev   # for a dev, integration or staging deployment\n   git push gitlab.prod  # for a prod deployment\n   ```\n\n   The build should start immediately. You can monitor its progress from the\n   [Gitlab Pipelines page](https://gitlab.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/pipelines).\n\n   If reindexing and promoting to staging or production, send a second\n   warning about reindexing to the #data-wrangling channel at this point.\n\n   Wait until the pipeline on Gitlab succeeds or fails. If the build fails before\n   the `deploy` stage, no permanent changes were made to the deployment but you\n   need to investigate the failure. If the pipeline fails at or after the `deploy`\n   stage, you need triage the failure. If it can't be resolved manually, you need\n   to reset the branch back to the LAST_RELEASE_TAG and repeat step 2 in this section.\n\n4. Invoke the health and version endpoints.\n\n   * For the `develop` branch and the corresponding `dev` deployment use\n\n     ```\n     http https://indexer.dev.singlecell.gi.ucsc.edu/version\n     http https://service.dev.singlecell.gi.ucsc.edu/version\n     http https://indexer.dev.singlecell.gi.ucsc.edu/health\n     http https://service.dev.singlecell.gi.ucsc.edu/health\n     ```\n\n   * For the `integration` branch/deployment use\n\n     ```\n     http https://indexer.integration.singlecell.gi.ucsc.edu/version\n     http https://service.integration.singlecell.gi.ucsc.edu/version\n     http https://indexer.integration.singlecell.gi.ucsc.edu/health\n     http https://service.integration.singlecell.gi.ucsc.edu/health\n     ```\n\n   * For the `staging` branch/deployment use\n\n     ```\n     http https://indexer.staging.singlecell.gi.ucsc.edu/version\n     http https://service.staging.singlecell.gi.ucsc.edu/version\n     http https://indexer.staging.singlecell.gi.ucsc.edu/health\n     http https://service.staging.singlecell.gi.ucsc.edu/health\n     ```\n\n   * For the `prod` branch/deployment use\n\n     ```\n     http https://indexer.singlecell.gi.ucsc.edu/version\n     http https://service.singlecell.gi.ucsc.edu/version\n     http https://indexer.singlecell.gi.ucsc.edu/health\n     http https://service.singlecell.gi.ucsc.edu/health\n     ```\n\n5. Assuming everything is successful, run\n\n   ```\n   make tag\n   ```\n\n   and the\n\n   ```\n   git push ...\n   ```\n\n   invocation that it echoes.\n\n6. In Zenhub, move all tickets from the pipeline representing the source\n   deployment of the promotion to the pipeline representing the target\n   deployment.\n\n7. In the case that you need to reindex run the manual `reindex` job on the\n   Gitlab pipeline representing the most recent build on the current branch.\n\n\n## 7.2 Big red button\n\nIn the event of an emergency, Azul can be shut down immediately using the\n`enable_lambdas.py` script. Before using this script, make sure that the desired\ndeployment is selected and your Python virtual environment is activated.\n\nShut down Azul by running\n\n```\npython scripts/enable_lambdas.py --disable\n```\n\nOnce your issue has been resolved, you can resume Azul's services by running\n\n```\npython scripts/enable_lambdas.py --enable\n```\n\n\n## 7.3 Copying bundles\n\nIn order to copy bundles from one DSS instance to another, you can use\n`scripts/copy_bundles.py`. The script copies specific bundles or all bundles\nlisted in a given manifest. It iterates over all source bundles, and all files\nin each source bundle. It copies the files by determining the native URL\n(`s3://…` ) of the DSS blob object for each file and passing that native URL to\nthe destination DSS' `PUT /files` endpoint as the source URL parameter for that\nrequest. This means that it is actually the destination DSS that physically\ncopies the files. Once all files in a bundle were copied, the script requests\nthe `PUT /bundles` endpoint to create a copy of the source bundle.\n\nThe script is idempotent, meaning you can run it repeatedly without harm,\nmostly thanks to the fact that the DSS' `PUT /files` and `PUT /bundles`\nendpoints are idempotent. If a script invocation resulted in a transient error,\nrunning the script again will retry all DSS requests, both successful requests\nand requests that failed in the previous invocation.\n\nIn order to determine the native URL of the source blob, the script needs\ndirect read access to the source DSS bucket. This is because blobs are an\nimplementation detail of the DSS and obtaining their native URL is not\nsupported by the DSS.\n\nFurthermore, The destination DSS requires the source object to carry tags\ncontaining the four checksums of the blob. Some blobs in some DSS instances\nhave those tags, some don't. It is unclear the tags are supposed to be present\non all blob objects or if their presence is incidental. To work around this,\nthe script can optionally create those tags when the destination DSS complains\nthat they are missing. To enable the creation of checksum tags on source blob\nobjects, use the `---fix-tags` option. Please be aware that `--fix-tags`\nentails modifying object tags in the source (!) bucket.\n\nThe destination DSS instance requires read access to the blobs in the source\nDSS bucket. The `integration` and `staging` instances can read each other's\nbuckets so copies can be made between those two instances. To copy bundles from\na DSS instance that is in a different AWS account compared to the destination\ninstance, from prod to integration, for example, you will likely need to modify\nthe source DSS bucket's bucket policy.\n\nYou should never copy **to** the HCA `prod` instance of the DSS.\n\nHere is a complete example for copying bundles from `prod` to `integration`.\n\n1) Ask someone with admin access to the DSS `prod` bucket (`org-hca-dss-prod`)\n   to add the following statements to the bucket policy of said bucket. The\n   first statement gives the destination DSS read access to the source DSS\n   instance. The second statement gives you read access to that bucket (needed\n   for direct access) and permission to set tags on objects (needed for\n   `--fix-tags`).\n\n   ```json\n   [\n       {\n           \"Sid\": \"copy-bundles\",\n           \"Effect\": \"Allow\",\n           \"Principal\": {\n               \"AWS\": [\n                   \"arn:aws:iam::861229788715:role/dss-integration\",\n                   \"arn:aws:iam::861229788715:role/dss-s3-copy-sfn-integration\",\n                   \"arn:aws:iam::861229788715:role/dss-s3-copy-write-metadata-sfn-integration\"\n               ]\n           },\n           \"Action\": [\n               \"s3:GetObject\",\n               \"s3:GetObjectTagging\"\n           ],\n           \"Resource\": \"arn:aws:s3:::org-hca-dss-prod/*\"\n       },\n       {\n           \"Sid\": \"direct-read-access-and-retag-blobs\",\n           \"Effect\": \"Allow\",\n           \"Principal\": {\n               \"AWS\": [\n                   \"arn:aws:iam::861229788715:role/dcp-admin\",\n                   \"arn:aws:iam::861229788715:role/dcp-developer\"\n               ]\n           },\n           \"Action\": [\n               \"s3:GetObject\",\n               \"s3:GetObjectTagging\",\n               \"s3:PutObjectTagging\"\n           ],\n           \"Resource\": [\n               \"arn:aws:s3:::org-hca-dss-prod/*\"\n           ]\n       }\n   ]\n   ```\n\n2) Select the `integration` deployment:\n\n   ```\n   _select integration\n   ```\n\n3) Run\n\n   ```\n   python scripts/copy_bundles.py --map-version 1.374856 \\\n                                  --fix-tags \\\n                                  --source https://dss.data.humancellatlas.org/v1 \\\n                                  --destination https://dss.integration.data.humancellatlas.org/v1 \\\n                                  --manifest /path/to/manifest.tsv\n   ```\n\n   The `--map-version` option adds a specific duration to the version of each\n   copied file and bundle. Run `python scripts/copy_bundles --help` for details.\n\n\n# 8. Continuous deployment and integration\n\nFor the purposes of continually testing and deploying the Azul application, we \nrun the community edition of GitLab on a project-specific EC2 instance. There is \ncurrently one such instance for the `sandbox` and `dev` deployments and another \none for `prod`.\n\nThe GitLab instances are provisioned through the `gitlab` *Terraform component*.\nFor more information about *Terraform components*, refer the [section on shared \nresources managed by Terraform](#314-shared-resources-managed-by-terraform). \nWithin the `gitlab` component, the `dev.gitlab` child component provides a \nsingle Gitlab EC2 instance that serves our CI/CD needs not only for `dev` but \nfor `integration` and `staging` as well. The `prod.gitlab` child component \nprovides the Gitlab EC2 instance for `prod`.\n\nTo access the web UI of the Gitlab instance for `dev`, visit\n`https://gitlab.dev.explore.…/`, authenticating yourself with your GitHub\naccount. After attempting to log in for the first time, one of the\nadministrators will need to approve your access. For `prod` use\n`https://gitlab.explore.…/`.\n\n[gitlab.tf.json.template.py]: /terraform/gitlab/gitlab.tf.json.template.py\n\nTo have the Gitlab instance build a branch, one pushes that branch to the Azul\nfork hosted on the Gitlab instance. The URL of the fork can be viewed by\nvisiting the GitLab web UI. One can only push via SSH, using a public SSH key \nthat must be deposited in each user's profile on the GitLab web UI. \n\nAn Azul build on Gitlab runs the `test`, `package`, `deploy`,  and\n`integration_test` Makefile targets, in that order. The target deployment for\nfeature branches is `sandbox`, the protected branches (`develop` and `prod` use \ntheir respective deployments.\n\n\n## 8.1 VPN access to GitLab\n\nThe GitLab EC2 instance resides in a VPC that can only be accessed through a\nVPN. The VPN uses AWS Client VPN. It is Amazon's flavor of OpenVPN. The AWS\nClient VPN endpoint is set up by Terraform as part of the `dev.gitlab` and\n`prod.gitlab` components. VPN clients authenticate via certificates signed by\na certificate authority (CA) that is self-signed. A system administrator\n(currently the technical lead) manages the CA on their local disk. That is\nthe only place where the private key for signing the CA certificate is kept.\nIf the CA private key is lost, the CA must be reinitialized, the VPN must be\nredeployed and new client certificates must be issued. Each deployment of\nGitLab uses a separate CA and therefore a separate set of client\ncertificates.\n\nEach client certificate is backed by a private key as well. That private key\nresides solely on the developer's local disk. If the developer's private key\nis lost, a new one must be issued. \n\n\u003c!--\nFIXME: Automate the revocation of VPN client certificates\n       https://github.com/DataBiosphere/azul/issues/3929\n--\u003e\n\nWhen a developer with VPN access departs the team, either the entire CA must be\nreinitialized and all remaining client certificates reissued or the departing\ndeveloper's certificates must be revoked by adding it to the list of revoked\nclient certificates on the AWS Client VPN instance. The VPN's server's\ncertificate and private key is stored in ACM so that AWS Client VPN can\nauthenticate itself to clients and check validity of the certificates that\nclients present to the server. Both client and server keys must be signed by\nthe same CA.\n\n### 8.1.1 Setting up a VPN client\n\nInstall an OpenVPN client. On Ubuntu, the respective package is called\n`network-manager-openvpn-gnome`. Popular clients for macOS are [Tunnelblick]\n(free) and [Viscosity] (for pay, with 30 day trial). For Windows, only \n[Viscosity] was tested but the [official Windows client] may also work there.\n\n[Tunnelblick]: https://tunnelblick.net/index.html\n[Viscosity]: https://www.sparklabs.com/viscosity/\n[official Windows client]: https://openvpn.net/client-connect-vpn-for-windows/\n\n\u003c!--\nFIXME: Figure out why Tunnelblick doesn't work\n       https://github.com/DataBiosphere/azul/issues/3930\n--\u003e\n\nGenerate a certificate request, import the certificate and generate the `.ovpn` \nfile containing the configuration for the VPN connection:\n\n```\n_select dev.gitlab  # or prod.gitlab, anvildev.gitlab\ncd terraform/gitlab/vpn\ngit submodule update --init easy-rsa\nmake init  # (do this only once per GitLab deployment)\nmake request  # then send request to administrator\nmake import  # paste the certificate\nmake config \u003e ~/azul-gitlab-dev.ovpn  # or azul-gitlab-prod.ovpn\n```\n\nThe `make init` step creates a PKI directory in `~/.local/share` outside of the \nAzul source tree. It should only be done once per GitLab deployment. On a second \nattempt it will ask for confirmation to overwrite the existing directory. If \nconfirmed, existing OpenVPN client connections will remain functional (as they \nkeep a copy of the private key) but you will lose the ability to regenerate the \n`.ovpn` file.\n\nNow import the generated `.ovpn` file into your client. `make config` prints\ninstructions on how to do so on Ubuntu. For other VPN clients the process is\npretty much self-explanatory. Delete the file after importing it. It contains\nthe private key and can always be regenerated again later using `make config`. \n\n### 8.1.2 Ensuring split tunnel on client\n\nExcept on stable deployments, you should configure the client to only route VPC\ntraffic through the VPN. The VPN server will not forward any other traffic, in\nwhat's commonly referred to as a *split tunnel*. The key indicator of a split\ntunnel is that it doesn't set up a default route on the client system. There\nwill only be a route to the private 172.… subnet of the GitLab VPC but the\ndefault route remains in place.\n\nOn stable deployments, split tunnels are prohibited.\n\nThe `make config` step prints instruction on how to configure a split tunnel\non Ubuntu. \n\nFor Viscosity, the steps are as follows:\n\n1) Click the Viscosity menu bar icon (or the task bar icon on windows)\n\n2) Click *Preferences*\n\n3) Right-click `azul-gitlab-dev` or `azul-gitlab-prod` -\u003e click *Edit*\n\n4) Click the *Networking* tab\n\n5) Under *All traffic*, select *Automatic (Set by server)*\n\n6) Click *Save*\n\nFor Tunnelblick, the steps are as follows:\n\n1) Right-click the Tunnelblick menu bar icon\n\n2) Click *VPN Details …*\n\n3) Click on the left-hand side bar entry for the connection you just imported\n\n4) On the *Settings* tab of the right-hand side of the window, make sure that\n   the *Route all IPv4 traffic through the VPN* option is unchecked\n\n### 8.1.3 Setting up the certificate authority\n\nThis must be done by a system administrator before a GitLab instance is first \ndeployed:\n\n```\n_select dev.gitlab  # or prod.gitlab\ncd terraform/gitlab/vpn\ngit submodule update --init easy-rsa\nmake ca  # initialize the CA (do this only once)\nmake server  # build the server certificate\nmake publish  # upload the server certificate to ACM\ncd ..\nmake apply  # (re)deploy GitLab\n```\n\n### 8.1.4 Issuing a certificate\n\nTo issue a client certificate for a developer so that they can access the VPN,\nask the developer to send you a certificate request as described in the previous \nsection . The request must be made under the developer's email address as the \ncommon name (CN). Sign the request:\n\n```\n_select dev.gitlab  # or prod.gitlab\ncd terraform/gitlab/vpn\ngit submodule update --init easy-rsa\nmake import/joe@foo.org\nmake sign/joe@foo.org\n```\n\nSend the resulting certificate back to the requesting developer.\n\nThe communication channel through which requests and certificates are messaged\ndoes not need to be private but it needs to ensure the integrity of the\nmessages.\n\n### 8.1.5 Revoking a certificate\n\n```\n_select dev.gitlab  # or prod.gitlab\ncd terraform/gitlab/vpn\ngit submodule update --init easy-rsa\nmake revoke/joe@foo.org\nmake publish_revocations\n```\n\nTo list all previously issued certificates, use `make list`. \n\nThere are now precautions in place to prevent this situation but I'll mention it \nanyways. If this list contains more than one active certificate for the same CN, \nall but the most recent one needs to be revoked by serial. Since `easyrsa` does\nnot support this out of the box, we need to jump through some extra hoops:  \n\n```\neval \"`make _admin _env`\"\nmv $EASYRSA_PKI/issued/joe@foo.org.crt $EASYRSA_PKI/issued/joe@foo.org.crt.orig\ncp $EASYRSA_PKI/certs_by_serial/\u003cSERIAL_OF_CERT_TO_BE_REVOKED\u003e.pem $EASYRSA_PKI/issued/joe@foo.org.crt\nmake revoke/joe@foo.org\nmake publish_revocations\nmv $EASYRSA_PKI/issued/joe@foo.org.crt.orig $EASYRSA_PKI/issued/joe@foo.org.crt\n```\n\n### 8.1.6 Issuing a certificate on a person's behalf\n\nA private key and OpenVPN configuration can be generated by a system\nadministrator on behalf of any person that doesn't have a configured working\ncopy of this repository. Doing so has the disadvantage of making that\nperson's private key known to the system administrator and anyone that\neavesdrops on the channel through which the OpenVPN configuration\n(which includes the private key) is communicated to the person.\n\nTo generate the key and OpenVPN configuration file on another person's behalf, \ninvoke the `make` steps as outlined in [8.1.1](#811-setting-up-a-vpn-client) and \n[8.1.3](#813-issuing-a-certificate) but use `make client_cn=joe@foo.org` instead \nof `make`.\n\n\n## 8.2 The Sandbox Deployment\n\nThere is only one such deployment and it should be used to validate feature\nbranches (one at a time) or to run experiments. This implies that access to the\nsandbox must be coordinated externally e.g., via Slack. The project lead owns\nthe sandbox deployment and coordinates access to it.\n\n\n## 8.3 Security\n\nGitlab has AWS write permissions for the AWS services used by Azul and the\nprinciple of least privilege is applied as much as IAM allows it. Some AWS\nservices support restricting the creation and deletion of resource by matching\non the name. For these services, Gitlab can only create, modify or write\nresources whose name begins with `azul-*`. Other services, such as API Gateway\nonly support matching on resource IDs. This is unfortunate because API Gateway\nallocates the ID. Since it is therefore impossible to know the ID of an API before\ncreating it, Gitlab must be given write access to **all** API IDs. For details\nrefer to the `azul-gitlab` role and the policy of the same name, both defined in\n[gitlab.tf.json.template.py].\n\n[permissions boundary]: https://aws.amazon.com/blogs/security/delegate-permission-management-to-developers-using-iam-permissions-boundaries/\n\nGitlab does not have general write permissions to IAM, its write access is\nlimited to creating roles and attaching policies to them as long as the roles\nand policies specify the `azul-gitlab` policy as a [permissions boundary]. This\nmeans that code running on the Gitlab instance can never escalate privileges\nbeyond the boundary. This mechanism is defined in the `azul-gitlab-iam` policy.\n\nCode running on the Gitlab instance has access to credentials of a Google Cloud\nservice account that has write privileges to Google Cloud. This service account \nfor Gitlab is created automatically by TF but its private key is not. They need \nto created manually and copied to `/mnt/gitlab/runner/config/etc` on the \ninstance. See [section 8.10](#810-the-gitlab-build-environment) for details.\n\n\n## 8.4 Networking\n\nThe networking details are documented in [gitlab.tf.json.template.py]. The\nGitlab EC2 instance uses a VPC and is fronted by an Application Load Balancer\n(ALB) and a Network Load Balancer (NLB). The ALB proxies HTTPS access to the\nGitlab web UI, the NLB provides SSH shell access and `git+ssh` access for\npushing to the project forks on the instance.\n\n\n## 8.5 Storage\n\nThe Gitlab EC2 instance is attached to an EBS volume that contains all of\nGitlab's data and configuration. That volume is not controlled by Terraform and\nmust be created manually before terraforming the `gitlab` component for the\nfirst time. Details about creating and formatting the volume can be found in\n[gitlab.tf.json.template.py]. The volume is mounted at `/mnt/gitlab`. The\nconfiguration changes are tracked in a local Git repository on the system \nadministrator's computer. The system administrator keeps the configuration files \nconsistent between GitLab instances.\n\nWhen an instance boots and finds the EBS volume empty, Gitlab will initialize it\nwith default configuration. That configuration is very vulnerable because the\nfirst user to visit the instance will be given the opportunity to chose the root\npassword. It is therefore important that you visit the Gitlab UI immediately\nafter the instance boots for the first time on an empty EBS volume.\n\nOther than that, the default configuration is functional but lacks features like\nsign-in with Github and a Docker image repository. To enable those you could\nfollow the respective Gitlab documentation but a faster approach is to compare\n`/mnt/gitlab/config/gitlab.rb` between an existing Gitlab instance and the new\none. Just keep in mind that the new instance might have a newer version of\nGitlab which may have added new settings. You may see commented-out default\nsettings in the new gitlab.rb file that may be missing in the old one.\n\n## 8.5.1 Freeing up storage space\n\nThere are three docker daemons running on the instance: the RancherOS system \ndaemon, the RancherOS user daemon and the Docker-in-Docker (DIND) daemon. For \nreasons unknown at this time, the DIND keeps caching images, continually \nconsuming disk space until the `/mnt/gitlab` volume fills up. In the past, this \noccurred once every six months or so. One of the symptoms might be a failing unit\ntest job with message like \n\n\u003e `2021-03-11 19:38:05,133 WARNING MainThread: There was a general error with document ContributionCoordinates(entity=EntityReference(entity_type='files', entity_id='5ceb5dc3-9194-494a-b1df-42bb75ab1a04'), aggregate=False, bundle=BundleFQID(uuid='94f2ba52-30c8-4de0-a78e-f95a3f8deb9c', version='2019-04-03T103426.471000Z'), deleted=False): {'_index': 'azul_v2_dev_test_files', '_type': 'doc', '_id': '5ceb5dc3-9194-494a-b1df-42bb75ab1a04_94f2ba52-30c8-4de0-a78e-f95a3f8deb9c_2019-04-03T103426.471000Z_exists', 'status': 403, 'error': {'type': 'cluster_block_exception', 'reason': 'blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];'}}. Total # of errors: 1, giving up.`\n\nA cron job running on the instance should prevent this by periodically pruning\nunused images. If the above error occurs despite that, there might be a problem\nwith that cron job. To manually clean up unused images run:\n\n```\nsudo docker exec -it gitlab-dind docker image prune -a --filter \"until=720h\"\n```\n\non the instance.\n\n## 8.6 The Gitlab web application\n\nThe instance runs Gitlab CE running inside a rather elaborate concoction of\nDocker containers. See [gitlab.tf.json.template.py] for details. Administrative\ntasks within a container should be performed with `docker exec`. To reconfigure\nGitlab, for example, one would run `docker exec -it gitlab gitlab-ctl\nreconfigure`.\n\n\n## 8.7 Registering the Gitlab runner\n\nThe runner is the container that performs the builds. The instance is configured\nto automatically start that container. The primary configuration for the runner\nis in `/mnt/gitlab/runner/config/config.toml`. There is one catch, on a fresh\nEBS volume that just been initialized, this file is missing, so the container\nstarts but doesn't advertise any runners to Gitlab.\n\nThe easiest way to create the file is to kill the `gitlab-runner` container and\nthe run it manually using the `docker run` command from the `systemd` unit file, \nbut adding `-it` after `run` and `register` at the end of the command. \nYou will be prompted to supply a URL and a registration token as\n[documented here](https://docs.gitlab.com/runner/register/).\n\nNote that since version 15.0.0 of GitLab, there is no way to convert a runner\nfrom shared to project-specific or vice versa. If you want to register a runner\nreserved to a specific group, you must get the registration token from\nthe *CI/CD* — *Runners* page of the respective group. Runners reserved to a\nproject must be registered from the project's *Settings* — *CI/CD* — *Runners*\npage. Shared runners are registered via *Admin* — *Overview* — *Runners*. \n\nSpecify `docker` as the runner type and \n\n`docker.gitlab.anvil.gi.ucsc.edu/ucsc/azul/runner:latest` \n\nas the image for Azul runners. For generic runners you could use the \n`docker:20.10.18-ce` image instead, but you'd need to match the tag (aka \nversion) of the image currently used for the `gitlab-dind` container. \n\nHere's an example terminal transcript:\n\n```\n$ systemctl stop gitlab-runner.service\n\n$ systemctl show gitlab-runner.service | grep ExecStart=\nExecStart={ path=/usr/bin/docker ; argv[]=/usr/bin/docker run --name gitlab-runner …\n\n$ /usr/bin/docker run -it --name gitlab-runner --rm --volume /mnt/gitlab/runner/config:/etc/gitlab-runner --network gitlab-runner-net --env DOCKER_HOST=tcp://gitlab-dind:2375 gitlab/gitlab-runner:v15.9.1 register\nRuntime platform                                    arch=amd64 os=linux pid=7 revision=d540b510 version=15.9.1\nRunning in system-mode.\n\nEnter the GitLab instance URL (for example, https://gitlab.com/):\nhttps://gitlab.anvil.gi.ucsc.edu/\nEnter the registration token:\nREDACTED\nEnter a description for the runner:\n[cd20ca0ec956]:\nEnter tags for the runner (comma-separated):\n\nEnter optional maintenance note for the runner:\n\nWARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://gitlab.com/gitlab-org/gitlab/-/issues/380872\nRegistering runner... succeeded                     runner=GR1348941eDiqsoCC\nEnter an executor: docker, shell, ssh, docker-ssh+machine, instance, custom, docker-ssh, parallels, virtualbox, docker+machine, kubernetes:\ndocker\nEnter the default Docker image (for example, ruby:2.7):\ndocker.gitlab.anvil.gi.ucsc.edu/ucsc/ azul/runner:latest\nRunner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!\n\nConfiguration (with the authentication token) was saved in \"/etc/gitlab-runner/config.toml\"\n```\n\nOnce the container exits, `config.toml` should have been created. Edit it and \nadjust the `volumes` setting to read\n\n```\nvolumes = [\"/var/run/docker.sock:/var/run/docker.sock\", \"/cache\", \"/etc/gitlab-runner/etc:/etc/gitlab\"]\n```\n\nIf you already have a GitLab instance to copy `config.toml` from, do that and\nregister the runners as described above. Copy the runner tokens from the newly\nadded runners at the end of config.toml to the preexisting runners. Then\ndiscard the newly added runners from the file. For another instance's\n`config.toml` to work on a new instance, the only piece of information that\nneeds to be updated is the runner token. That's because the runner token is\nderived from the registration token which is different between the two\ninstances.\n\nFinally, start the runner unit using `systemctl start gitlab-runner.service` or\nsimply reboot the instance. Either way, the Gitlab UI should now show the newly \nregistered runners.\n\n\n## 8.8 The Gitlab runner image for Azul\n\nBecause the first stage of the Azul pipeline on Gitlab creates a dedicated image\ncontaining the dependencies of the subsequent stages, that first stage only\nrequires the `docker` client binary, `make` and `bash` to be in the runner.\nThese are provided by yet another custom Docker image for the Gitlab runner that\nexecutes Azul builds. This image must be created when the EBS volume attached to\nthe Gitlab instance is first provisioned, or when the corresponding Dockerfile\nis modified. See `terraform/gitlab/runner/Dockerfile` for details on how to\nbuild the image and register it with the runner.\n\n\n## 8.9 Updating Gitlab\n\nModify the Docker image tags in [gitlab.tf.json.template.py] and run `make\napply` in `terraform/gitlab`. The instance will be terminated (the EBS volume\nwill survive) and a new instance will be launched, with fresh containers from\nupdated images. This should be done regularly.\n\n\n## 8.10 The Gitlab Build Environment\n\nThe `/mnt/gitlab/runner/config/etc` directory on the Gitlab EC2 instance is\nmounted into the build container as `/etc/gitlab`. The Gitlab build for Azul\ncopies the files from the `azul` subdirectory of that directory into the Azul\nproject root. Secrets and other Gitlab-specific settings should be specified in\n`/mnt/gitlab/runner/config/etc/azul/environment.local` which will end up in\n`${project_root}/environment.local` where `source environment` will find and load\nthem. For secrets, we prefer this mechanism over specifying them as environment\nvariables under project settings on the Gitlab web UI. Only people with push\naccess can push code to intentionally or accidentally expose those variables,\npush access is tied to shell access which is what one would normally need to\nmodify those files.\n\n\n## 8.11. Cleaning up hung test containers\n\nWhen cancelling the `make test` job on Gitlab, test containers will be left\nrunning. To clean those up, ssh into the instance as described in\n[gitlab.tf.json.template.py] and run `docker exec gitlab-dind docker ps -qa |\nxargs docker exec gitlab-dind docker kill` and again but with `rm` instead\nof `kill`.\n\n\n# 9. Kibana and Cerebro\n\nKibana is a web UI for interactively querying and managing an Elasticsearch\ninstance. To use Kibana with Azul's AWS Elasticsearch instance, you have two\noptions:\n\n* For one, you can add your local IP to the policy of Azul's AWS Elasticsearch\n  instance and access its Kibana directly. This can take 10 minutes and you\n  might have to do it repeatedly because the policy is reset periodically,\n  potentially multiple times a day.\n\n* Alternatively, you can use `scripts/kibana_proxy.py` to run Kibana locally\n  and have it point at Azul's AWS Elasticsearch instance. The script also\n  starts a signing proxy which eliminates the need to add your local IP to the\n  Elasticsearch policy, using your local AWS credentials instead for\n  authentication.\n\n  For the script to work, you need to\n\n  * have Docker installed,\n\n  * a deployment selected, and\n\n  * `environment` sourced.\n\n[Cerebro] is a cluster management web UI for Elasticsearch. It is very useful\nfor determining the status of individual nodes and shards. In addition to the\nKibana container, `scripts/kibana_proxy.py` also starts one for Cerebro.\n\nLook for this line in the script output:\n\n```\nNow open Kibana at http://127.0.0.1:5601/ and open Cerebro at\nhttp://127.0.0.1:5602/#/overview?host=http://localhost:5603 (or paste in\nhttp://localhost:5603)\n```\n\nand open the specified URLs in your browser.\n\n[Cerebro]: https://github.com/lmenezes/cerebro\n\n## 9.1 Connecting Kibana to a local Elasticsearch instance\n\nCertain unit tests use a locally running Elasticsearch container. It's possible \nto connect a Kibana instance to such a container, in order to aid debugging.\n\nWhile the unit test is running (paused at a breakpoint), open a terminal window.\n\nDownload the Kibana container:\n\n```\nkibana_image=$azul_docker_registry$(python -m azul 'docker.resolve_docker_image_for_launch(\"pycharm\")')\ndocker pull $kibana_image\n```\n\nCopy the container name for the Elasticsearch instance you want to examine. This\nis likely the most recent entry in\n\n```\ndocker ps\n```\n\nRun\n\n```\ndocker run --link ES_CONTAINER_NAME:elasticsearch -p 5601:5601 $kibana_image\n```\n\nwhere `ES_CONTAINER_NAME` is what you copied from above.\n\nKibana should now be available at `http://0.0.0.0:5601`.\n\nSome of these steps were taken or modified from the official [Elasticsearch \ndocumentation](https://www.elastic.co/guide/en/kibana/7.10/docker.html#_run_kibana_on_docker_for_development).\n\n# 10. Managing dependencies\n\nWe pin all dependencies, direct and transitive ones alike. That's the only way\nto get a somewhat reproducible build. It's possible that the build still\nfails if a dependency version is deleted from pypi.org or if a dependency\nmaintainer re-releases a version, but aside from caching all dependencies,\npinning them is next best thing for reproducibility of the build.\n\nNow, while pinning direct dependencies should be routine, chasing down\ntransitive dependencies and pinning those is difficult, tedious and prone to\nerrors. That's why we automate that step: When a developer updates, adds or\nremoves a direct dependency, running `make requirements_update` will reevaluate\nall transitive dependencies and update their pins. If the added direct\ndependency has transitive dependencies, those will be picked up. It's likely\nthat the reevaluation picks up updates to transitive dependencies unrelated to\nthe modified direct dependency, but that's unavoidable. It's even possible that\na direct dependency update causes a downgrade of a transitive dependency if the\nupdated direct dependency further restricts the allowed version range of the\ntransitive dependency.\n\nWe distinguish between run-time and build-time — or _development_ —\ndependencies. A run-time dependency is a one that is needed by deployed code.\nA build-time dependency is one that is **not** needed by deployed code, but by\nsome other code, like unit tests, for example. A developer's virtualenv will\nhave both run-time and build-time dependencies installed. Combined with the\ndistinction between direct and transitive dependencies this yields four\ncategories of dependencies. Let's refer to them as DR (direct run-time), TR\n(transitive run-time), DB (direct build-time) and TB (transitive build-time).\nThe intersections DR ∩ TR, DB ∩ TB, DR ∩ DB, TR ∩ TB and DR ∩ TB should all be\nempty but the intersection TR ∩ DB may not be.\n\n![Azul architecture diagram](docs/dependencies.svg)\n\nAmbiguities can arise as to which version of a requirement should be used when\nmultiple requirements have overlapping transitive dependencies. We can't\nresolve these ambiguities automatically because different versions of a package\nmay have different dependencies in and of themselves, so pinning just the\ndependency in question might omit some of its dependencies. By pinning it\nexplicitly the normal dependency resolution kicks in, including all transitive\ndependencies of the pinned version.\n\n`make requirements_update` will raise an exception when ambiguous requirements\nare found.\n\n```\nERROR   MainThread: Ambiguous version of transitive runtime requirement jsonschema==2.6.0,==3.2.0. Consider pinning it to the version used at build time (==3.2.0).\n```\n\nWith this example case the solution would be to add `jsonschema` as a\ndirect run-time requirement in the file `reqirements.txt` along with a comment\n`# resolve ambiguity with build-time dependency`, and then to run `make\nrequirements_update` to remove the package as a transitive run-time requirement.\n\nThere is a separate category for requirements that need to be installed before\nany other dependency is installed, either run-time or build-time, in order to\nensure that the remaining dependencies are resolved and installed correctly.\nWe call that category  _pip requirements_ and don't distinguish between direct\nor transitive requirements in that category.\n\n\n# 11. Making wheels\n\n_Note: Support for custom wheels is currently disabled. We don't currently have \nany dependencies for which a binary wheel is unavailable. We'll leave this \nsection in place until support is needed and enabled again_  \n\nSome of Azul's dependencies contain native code that needs to be compiled into\na binary executable which is then dynamically loaded into the Python\ninterpreter process when the package is imported. These dependencies are\ncommonly distributed in the form of wheels. A wheel is a Python package\ndistribution that contains the pre-compiled binary code for a particular\noperating system and processor architecture combination, aka platform. Many such\npackages lack a wheel for the `linux_x86_64` platform that Lambda functions\nexecute on. Chalice will attempt to build the wheel on the fly during `chalice\npackage` (`make -C lambdas`) but only if invoked on a system with `linux_x86_64`.\nOn macOS, Chalice will fail to build a wheel for the `linux_x86_64` platform but\nonly prints a warning that's easily missed. The deployed Lambda will likely\nfail with an import error.\n\nIf you add a dependency on a package with native code, you need to build the\nwheel manually:\n\n```\n(.venv) ~/workspace/hca/azul$ docker run -it -v ${project_root}/:/root/azul python:3.12.7-slim-bookworm bash\n\nroot@97804cb60d95:/# pip --version\npip 24.2 from /usr/local/lib/python3.12/site-packages/pip (python 3.12)\n\nroot@97804cb60d95:/# cd /root/azul/lambdas/.wheels\n\nroot@97804cb60d95:~/azul/lambdas/.wheels# pip wheel jsonobject==2.0.0\nCollecting jsonobject==2.0.0\n  Downloading jsonobject-2.0.0.tar.gz (402 kB)\n     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 403.0/403.0 KB 9.0 MB/s eta 0:00:00\n  Preparing metadata (setup.py) ... done\nCollecting six\n  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)\nSaved ./six-1.16.0-py2.py3-none-any.whl\nBuilding wheels for collected packages: jsonobject\n  Building wheel for jsonobject (setup.py) ... done\n  Created wheel for jsonobject: filename=jsonobject-2.0.0-cp39-cp39-linux_x86_64.whl size=1606493 sha256=7f69b1ef612e13265ea95817e24b7d33ec63f07c0924f8c8692ee689679e1a18\n  Stored in directory: /root/.cache/pip/wheels/c1/1b/00/8958e64a98b73db2ca8d997a7034c93b545cdcf30054aa7e43\nSuccessfully built jsonobject\n\nroot@97804cb60d95:~/azul/lambdas/.wheels# ls -l\ntotal 1584\n-rw-r--r-- 1 root root 1606493 May 10 00:35 jsonobject-2.0.0-cp39-cp39-linux_x86_64.whl\n-rw-r--r-- 1 root root   11053 May 10 00:35 six-1.16.0-py2.py3-none-any.whl\n\nroot@97804cb60d95:~/azul/lambdas/.wheels# exit\nexit\n\n(.venv) ~/workspace/hca/azul$ ls -l lambdas/.wheels\ntotal 1584\n-rw-r--r-- 1 root root 1606493 May  9 17:35 jsonobject-2.0.0-cp39-cp39-linux_x86_64.whl\n-rw-r--r-- 1 root root   11053 May  9 17:35 six-1.16.0-py2.py3-none-any.whl\n\n(.venv) ~/workspace/hca/azul$ sudo chown -R `id -u`:`id -g` lambdas/.wheels\n\n(.venv) ~/workspace/hca/azul$ ls -l lambdas/.wheels\ntotal 1584\n-rw-r--r-- 1 hannes hannes 1606493 May  9 17:35 jsonobject-2.0.0-cp39-cp39-linux_x86_64.whl\n-rw-r--r-- 1 hannes hannes   11053 May  9 17:35 six-1.16.0-py2.py3-none-any.whl\n(.venv) ~/workspace/hca/azul$ \n```\n\nThen modify the `wheels` target in `lambdas/*/Makefile` to unzip the wheel into\nthe corresponding vendor directory.\n\nAlso see https://chalice.readthedocs.io/en/latest/topics/packaging.html\n\n\n# 12. Development tools\n\n\n## 12.1 OpenAPI development\n\n[Azul Service OpenAPI page]: https://service.dev.singlecell.gi.ucsc.edu/\n\nTo assist with adding documentation to the [Azul Service OpenAPI page] we can\nrun the service app locally:\n\n```\nmake -C lambdas/service local\n```\n\nThe script serves the Swagger editor locally at a URL where your current version\nof the API documentation is visible. Change the docs in `azul/service/app.py`,\nsave, refresh the page, and your changes will appear immediately.\n\n\n## 12.2 Tracking changes to the OpenAPI definition\n\nChanges to the OpenAPI definition are tracked in the source tree. When making \nchanges that affect the definition, run:\n\n```\nmake -C lambdas openapi\n```\n\nand commit any modifications to the `openapi.json` file. Failure to do so will \nbreak continuous integration during `make check_clean`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fazul","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fazul","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fazul/lists"}