{"id":28395123,"url":"https://github.com/databiosphere/data-browser","last_synced_at":"2026-04-02T10:44:17.616Z","repository":{"id":36978562,"uuid":"124946326","full_name":"DataBiosphere/data-browser","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2026-02-27T22:44:54.000Z","size":52472,"stargazers_count":11,"open_issues_count":123,"forks_count":5,"subscribers_count":24,"default_branch":"main","last_synced_at":"2026-02-28T00:06:05.944Z","etag":null,"topics":["boardwalk"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":".gitlab/README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"security.txt","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2018-03-12T20:19:09.000Z","updated_at":"2026-02-27T19:11:25.000Z","dependencies_parsed_at":"2023-12-14T08:26:11.169Z","dependency_job_id":"6c8ca4e0-ae8b-4f3d-b997-a3a8520c7fd5","html_url":"https://github.com/DataBiosphere/data-browser","commit_stats":null,"previous_names":[],"tags_count":74,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/data-browser","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fdata-browser","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fdata-browser/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fdata-browser/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fdata-browser/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/data-browser/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fdata-browser/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30156253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T22:39:40.138Z","status":"ssl_error","status_checked_at":"2026-03-05T22:39:24.771Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["boardwalk"],"created_at":"2025-05-31T19:39:11.551Z","updated_at":"2026-03-06T00:07:22.570Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"The files in this directory define how GitLab builds Data Browser\ndistributions (a tarball of the site content) and uploads them to the GitLab\npackage registry. The Azul pipeline will download these distributions and\ndeploy them to the S3 bucket backing the CloudFront distribution for\nbrowser \u0026 portal.\n\nThere is typically one site of the Data Browser per Azul atlas, an atlas being\na set of catalogs. Often there are multiple atlases hosted in an Azul\ndeployment and therefore multiple Data Browser sites accessing that Azul\ninstance, one per atlas. \n\nIn order to avoid duplicating substantial amounts of GitLab pipeline YAML, we\ndynamically generate the GitLab jobs that produce the distributions. That's\nmade possible in GitLab via the parent-child pipeline mechanism.\n\n`ci.yaml` defines the parent pipeline. The first job in the parent pipeline\nbuilds the Docker image to be used for all subsequent jobs in the pipeline.\nThis image is defined in the `Dockerfile`. The first job uses the Azul runner\nimage, a minimal image containing a `docker` client. The Azul runner image is\npushed manually when the Azul project is first set up in a GitLab instance.\nThe GitLab job token that is issued to pipelines running for this project\nmust be granted access to the pull from the registry for the Azul project, so\nthat the runner can pull the Azul runner image for the first job. \n\nThe second job in the parent pipeline generates the GitLab pipeline YAML for\nthe child pipeline. The third job triggers a child pipeline using the YAML\ngenerated by the second job, which that job exposes as an artifact. The\ngeneration is done by a a Typescript program running on NodeJS, not because\nTypescript is particularly good at generating YAML but simply because\nbuilding the Data Browser already requires NodeJS.\n\nThe generated child pipeline contains two stages: one stage builds the\ndistributions, the other stage uploads them to the package registry. There is\none job per atlas in each of these two stages: a `build` job and a `publish`\njob.\n\nThe `sites` directory contains GitLab YAML job fragments that are merged into\nthe generated YAML for the child pipeline. There is one subdirectory per Azul\ndeployment and site. The Azul deployment is fixed as a job variable in\nthe _Settings_, _CI/CD_, and _Variables_ section of the GitLab project. The\nname of the variable is `AZUL_DEPLOYMENT_STAGE`. The GitLab administrator\npicks the Azul deployment by configuring the job variable, and the pipeline\nbuilds a distribution per site in that deployment. Each site contains a\nnumber of YAML job fragments:\n\n- `sites/\u003cdeployment\u003e/\u003csite\u003e/base.yaml` is merged into the `.base` job\n  definition that the `build` and `publish` jobs extend. It typically only\n  contains a `variables` section that sets the environment variables for both\n  jobs, but specific to a site and a deployment\n\n- `sites/\u003cdeployment\u003e/\u003csite\u003e/build.yaml` is merged into the `build` job\n  definition. It contains at least a `script` section that actually builds\n  the distribution \n\n- `sites/\u003cdeployment\u003e/pipeline.yaml` is merged into the child pipeline, at the\n  document root. It typically contains a `variables` section with environment\n  variables that are not specific to a site.\n\nThe merging uses https://github.com/kleber-swf/deepmerge-json which allows for\nmerging of JSON arrays, something that GitLab's native inheritance/inclusion\nmechanisms doesn't support. This allows for customizing the `rules`\nsection of a job, say, by appending a rule. See\nhttps://github.com/kleber-swf/deepmerge-json#array-merge\n\nTo reuse a YAML fragment file in another site or deployment, use symlinks.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fdata-browser","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fdata-browser","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fdata-browser/lists"}