{"id":28395229,"url":"https://github.com/databiosphere/encode-ingest","last_synced_at":"2025-08-08T19:22:50.437Z","repository":{"id":37808015,"uuid":"232907607","full_name":"DataBiosphere/encode-ingest","owner":"DataBiosphere","description":"Batch ETL pipeline to mirror ENCODE data into the Jade Data Repository.","archived":false,"fork":false,"pushed_at":"2023-08-04T15:59:11.000Z","size":1719,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-06-01T06:52:33.652Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/DataBiosphere.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-01-09T21:15:51.000Z","updated_at":"2022-11-22T16:52:06.000Z","dependencies_parsed_at":"2023-01-20T23:17:33.983Z","dependency_job_id":null,"html_url":"https://github.com/DataBiosphere/encode-ingest","commit_stats":null,"previous_names":[],"tags_count":241,"template":false,"template_full_name":null,"purl":"pkg:github/DataBiosphere/encode-ingest","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fencode-ingest","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fencode-ingest/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fencode-ingest/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fencode-ingest/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/DataBiosphere","download_url":"https://codeload.github.com/DataBiosphere/encode-ingest/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/DataBiosphere%2Fencode-ingest/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":262172427,"owners_count":23270009,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-31T19:39:35.012Z","updated_at":"2025-06-27T01:31:29.465Z","avatar_url":"https://github.com/DataBiosphere.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ENCODE Ingest\n\nBatch ETL pipeline to mirror ENCODE data into the Terra Data Repository (TDR). See the [architecture documentation](https://github.com/DataBiosphere/encode-ingest/blob/master/ARCHITECTURE.md) for \nfurther design details.\n\n## Getting Started\n\nOrchestration of the ETL flows in this project is implemented using [Argo Workflows](https://argoproj.github.io/argo-workflows/).\n\nThe core extraction and transformation data pipelines are implemented in [Scio](https://spotify.github.io/scio/) on top of Apache Beam.\n\nAfter cloning the repository, ensure you can compile the code, auto-generate schema classes\nand run the test suite from the repository root:\n\n`sbt test`\n\n## Development Process\n\nAll development should be done on branches off of the protected `master` branch. After review, merge to `master`\nand then follow the instuctions in the [monster-deploy repo](https://github.com/broadinstitute/monster-deploy/)\n\nWhen modifying the Scio data pipelines, it's possible to run the pipeline locally by invoking the relevant pipeline:\n\n* **Extraction:** \n\n`sbt \"encode-extraction / runMain org.broadinstitute.monster.encode.extraction.ExtractionPipeline  --outputDir=\u003csome local directory\u003e\"`\n* **Transformation** \n\n`sbt \"encode-transformation-pipeline / runMain org.broadinstitute.monster.encode.transformation.TransformationPipeline --inputPrefix=\u003cextraction dir\u003e --outputPrefix=\u003coutput dir\u003e\"`\n\nDevelopment of Argo changes requires deployment to the DEV environment as documented in the [monster-deploy repo](https://github.com/broadinstitute/monster-deploy/)\n\n## Build and deploy code chages from branch\n\n\n#### *Do you have a new Macbook Pro with the M1 chip?*\n\n1. If so, clone this repo `https://github.com/DataBiosphere/ingest-utils.git`\n1. Check out the branch `ah_m1arch\n`\n\n### Steps in encode-ingest repo\n\n1. Before you commit changes, make sure the build occurs without error `sbt compile` and that the tests run without errors `sbt test`. \n\t\n\t*The build may reformat some of your files. Make sure to do a diff and add any changes to your git staging area.*\n1. Commit changes to local branch `git commit -m \"\u003ccomment\u003e\"` \n1. Create a version tag for the branch `git tag v1.0.\u003cnew-number\u003e`. For example `v1.0.120`\n\t\n\t*You can run a `git log\t` to see the previous version number or go to the actions tab for the repo `https://github.com/DataBiosphere/encode-ingest/actions` to see the previous version built. It is very important the the version format is exact. No `.` between the `v` and the `1`.*\n1. Now push the branch and tag to the remote `git push origin : v1.0.\u003cx\u003e`\n\n\t*You can also push your branch to the remote and then separately push the tags to the remote `git push --tags`*\n\t\n\t*This will start 2 build actions. If the build fails, you can look at the details for errors or just run a local `sbt compile`. In some cases you may have forgotten to commit formatting changes done automatically by `sbt compile`. Just add them to your staging area and start with #2 above.*\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fencode-ingest","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdatabiosphere%2Fencode-ingest","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdatabiosphere%2Fencode-ingest/lists"}