{"id":35418580,"url":"https://github.com/elifesciences/data-hub-ejp-xml-pipeline","last_synced_at":"2026-04-02T12:01:32.245Z","repository":{"id":37076095,"uuid":"259486264","full_name":"elifesciences/data-hub-ejp-xml-pipeline","owner":"elifesciences","description":"Data pipeline for loading eJP Xml files from S3 bucket into Bigquery ","archived":false,"fork":false,"pushed_at":"2026-03-24T10:41:48.000Z","size":740,"stargazers_count":2,"open_issues_count":0,"forks_count":1,"subscribers_count":6,"default_branch":"develop","last_synced_at":"2026-03-25T13:21:04.107Z","etag":null,"topics":["data-team"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/elifesciences.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":"maintainers.txt","copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2020-04-28T00:07:19.000Z","updated_at":"2026-03-17T20:43:17.000Z","dependencies_parsed_at":"2023-02-11T17:15:16.117Z","dependency_job_id":"43861b1e-99b5-43c1-b949-4bbf67fcd0f1","html_url":"https://github.com/elifesciences/data-hub-ejp-xml-pipeline","commit_stats":null,"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/elifesciences/data-hub-ejp-xml-pipeline","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elifesciences%2Fdata-hub-ejp-xml-pipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elifesciences%2Fdata-hub-ejp-xml-pipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elifesciences%2Fdata-hub-ejp-xml-pipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elifesciences%2Fdata-hub-ejp-xml-pipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/elifesciences","download_url":"https://codeload.github.com/elifesciences/data-hub-ejp-xml-pipeline/tar.gz/refs/heads/develop","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/elifesciences%2Fdata-hub-ejp-xml-pipeline/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31305971,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-02T09:48:21.550Z","status":"ssl_error","status_checked_at":"2026-04-02T09:48:19.196Z","response_time":89,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-team"],"created_at":"2026-01-02T16:00:46.532Z","updated_at":"2026-04-02T12:01:32.199Z","avatar_url":"https://github.com/elifesciences.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# eJP XML Data Pipeline\n\nThis repository consists of a generic data pipeline that is used to ETL eJP XML dumps in S3 buckets.\nBeing generic, it needs to be configured for its data source, data sink, and transformations.\nThe sample configuration for this data pipeline can be found in `sample_data_config` directory of this project\n\n## Running The Pipeline Locally\n\nThis repo is designed to run in as a containerized application in the development environment.\nTo run this locally, review the `docker-compose.dev.override.yml` and `docker-compose.yaml` files, and ensure that the different credentials files required by different data pipelines are correctly provided.\nFollowing are the credentials that you may need to provide\n\n- GCP's service account json key (mandatory for all data pipelines)\n- AWS credentials\n\n### Using Docker\n\nTo run the whole test on the application:\n\n    make build-dev end2end-test\n\nTo run tests excluding the end to end tests:\n\n    make build-dev test-exclude-e2e\n\nTo run the pipeline (which uses a state file):\n\n    make data-hub-pipelines-run-ejp-xml-pipeline\n\n### Using a Virtual Environment\n\nTo set up the development environment:\n\n    # initial setup\n    make dev-venv\n    # update dependencies\n    make dev-install\n\nTo run the test:\n\n    make dev-test\n\nTo run the pipeline (which uses a state file):\n\n    make dev-run-ejp-xml-pipeline\n\nTo clear the state:\n\n    make dev-clear-state\n\n## Project Folder/Package Organisation\n\n- `ejp_xml_pipeline` package consist of the packages and libraries and functions needed to run the pipeline.\n- `tests` contains the tests run on this implementation. These include these types\n  - unit tests\n  - end to end tests\n- `sample_data_config` folder contains the sample configurations for the data pipeline\n\n## CI/CD\n\nThis runs on Jenkins and follows the standard approaches used by the `eLife Data Team` for CI/CD.\nNote that as part of the CI/CD, another Jenkins pipeline is always triggered whenever there is a commit to the develop branch. The latest commit reference to a `develop` branch is passed on as a parameter to this Jenkins pipeline to be triggered.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felifesciences%2Fdata-hub-ejp-xml-pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Felifesciences%2Fdata-hub-ejp-xml-pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Felifesciences%2Fdata-hub-ejp-xml-pipeline/lists"}