{"id":28395151,"url":"https://github.com/databiosphere/featured-notebooks","last_synced_at":"2025-08-17T16:43:34.800Z","repository":{"id":42186207,"uuid":"274259675","full_name":"DataBiosphere/featured-notebooks","owner":"DataBiosphere","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-18T17:06:45.000Z","size":229,"stargazers_count":2,"open_issues_count":11,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-01T06:52:22.408Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-06-22T23:00:21.000Z","updated_at":"2024-07-18T17:03:29.000Z","dependencies_parsed_at":"2022-08-12T08:50:48.127Z","dependency_job_id":null,"html_url":"https://github.com/DataBiosphere/featured-notebooks","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/featured-notebooks","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ffeatured-notebooks","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ffeatured-notebooks/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ffeatured-notebooks/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ffeatured-notebooks/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/featured-notebooks/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Ffeatured-notebooks/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172364,"owners_count":23269993,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:20.777Z","updated_at":"2025-06-27T01:31:16.590Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# featured-notebooks\nExample Jupyter notebook source files for the [BioData Catalyst](https://biodatacatalyst.nhlbi.nih.gov) and\n[AnVIL](https://anvilproject.org) consortiums. These notebooks are published to the\n[Terra notebook environment](https://support.terra.bio/hc/en-us/articles/360027237871-Terra-s-Jupyter-Notebooks-environment-Part-I-Key-components).\n\nJupyter notebook `.ipynb` files are generated from source files with the\n[herzog](https://github.com/xbrianh/herzog) library. \n\n## Notebooks Represented\n* BYOD -- Tutorial for bringing your own data to the Terra platform\n* GWAS Blood Pressure -- Two part series teaching how to use Gen3 data to perform a GWAS; [mirror of template workspace](https://app.gitbook.com/@bdcatalyst/s/biodata-catalyst-tutorials/tutorials/terra-tutorials/bdcatalyst-gwas-terra-gen3-dockstore)\n* Prepare IGV Viewer Input -- Prepares data for IGV within Terra\n* VCF Merge/Subsample Tutorial -- Merging and subsampling for jointly called VCFs\n* Workflow Cost Estimator -- Estimate costs of workflows, even on billing projects that obscure this information\n* xvcfmerge Array Input -- Demonstrate Terra's data table structure being converted to inputs for xvcfmerge workflows\n\n### Notebook Source Schema\nA notebook is defined by creating a directory under ${REPO_ROOT}/notebooks containing\n  - `main.py`, the source script for the notebook in [herzog](https://github.com/xbrianh/herzog) format.\n  - `requirements.txt`, pip-installable requirements needed for notebook execution.\n  - `publish.txt`, a list of destination Google Storage URLs where the notebook will be published.\n\n### The Herzog Format\nNotebook source files, `main.py`, are executable Python scripts. Contents of cells are denoted with the herzog\nPython context managers:\n```\nwith herzog.Cell(\"{mode}\"):\n\t...\n```\nwhere `{mode}` is `python` or `markdown`. Python statements outside of herzog contexts will not be rendered into\ncells. Test code or mock environments should not be placed outside herzog contexts.\n\n## Publishing \u0026 Testing\n\n### Publishing Notebooks\nNotebooks are published with make commands, e.g.\n```\nmake publish/byod\n```\nBefore publishing, make sure that the Google Bucket in the publish.txt file matches that of the destination workspace. \n\n### ad-hoc publication\nA convenience script is provided to generate herzog scripts into .ipynb files and copy them into Google Storage\nlocations.\n```\nscrips/generate_to_gs.sh notebooks/byod/main.py gs://my-bucket/my-notebook-location\n```\n\n### Testing Notebooks\nTests are performed in Docker containers of the expected notebook deployment platform, e.g. Terra, with the command\n```\nmake test\n```\n\nTests can also be performed on individual notebooks\n```\nmake test/byod\n```\n\nThese recipes pass the source script through the [flake8](https://flake8.pycqa.org/en/latest/) linter and\n[mypy](https://mypy.readthedocs.io/en/stable/) static analysis tool, and executes with a Docker container that is\ntypical of Terra notebook runtime environments for Python. If there are no errors,\n[herzog](https://github.com/xbrianh/herzog) is used to generate the source script into an `.ipynb`, which is copied\ninto the Terra workspace bucket.\n\n### Authorization for Testing and Publishing\n\nGoogle user credentials are required to publish notebooks to Terra workspaces. Additionally, notebook execution may\nrequire Google application default credentials. Both sets can be obtained by executing the commands\n```\ngcloud auth login\ngcloud auth application-default login\n```\n\nCredentials are injected into the local Docker container when testing and publishing notebooks, and are expected to be\nin the standard location under `~/.config`.\n\n### Local Development Environment\n\n1. Notebook source scripts should be developed using a Python 3.7 virtual environment, which matches the typical Python version\n   in the Terra notebook runtime, and `requirement-dev.txt` should be installed.\n\n   On MacOS or similar systems, this can be done by issuing the commands\n   ```\n   /usr/local/bin/python3.7 -m venv {vpath}\n   source {vpath}/bin/activate\n   pip install -r requirements-dev.txt\n   ```\n   where `{vpath}` should be replaced with the desired location of the virtual environment.\n\n2. The Google Storage utility \"gsutil\" should be installed according to the instructions found\n   [here](https://cloud.google.com/storage/docs/gsutil_install).\n\n## Other\n### Links\nProject home page [GitHub](https://github.com/DataBiosphere/featured-notebooks)  \n\n### Bugs\nPlease report bugs, issues, feature requests, etc. on [GitHub](https://github.com/DataBiosphere/featured-notebooks).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Ffeatured-notebooks","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Ffeatured-notebooks","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Ffeatured-notebooks/lists"}