{"id":21579180,"url":"https://github.com/subugoe/leine","last_synced_at":"2025-03-18T07:41:07.725Z","repository":{"id":46135233,"uuid":"353038355","full_name":"subugoe/leine","owner":"subugoe","description":"Data Pipelines for @subugoe/wag","archived":false,"fork":false,"pushed_at":"2021-11-11T12:48:22.000Z","size":100,"stargazers_count":1,"open_issues_count":19,"forks_count":0,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-01-24T14:19:06.297Z","etag":null,"topics":["data-pipeline","data-science","rstats"],"latest_commit_sha":null,"homepage":"https://subugoe.github.io/leine","language":"R","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/subugoe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-03-30T14:52:02.000Z","updated_at":"2021-04-19T07:39:12.000Z","dependencies_parsed_at":"2022-09-03T19:52:24.801Z","dependency_job_id":null,"html_url":"https://github.com/subugoe/leine","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subugoe%2Fleine","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subugoe%2Fleine/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subugoe%2Fleine/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subugoe%2Fleine/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/subugoe","download_url":"https://codeload.github.com/subugoe/leine/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244180857,"owners_count":20411600,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-pipeline","data-science","rstats"],"created_at":"2024-11-24T13:12:38.476Z","updated_at":"2025-03-18T07:41:07.691Z","avatar_url":"https://github.com/subugoe.png","language":"R","funding_links":[],"categories":[],"sub_categories":[],"readme":"# leine\n\n\u003c!-- badges: start --\u003e\n[![Main](https://github.com/subugoe/leine/workflows/.github/workflows/main.yaml/badge.svg)](https://github.com/subugoe/leine/actions)\n[![Codecov test coverage](https://codecov.io/gh/subugoe/leine/branch/master/graph/badge.svg)](https://codecov.io/gh/subugoe/leine?branch=master)\n[![CRAN status](https://www.r-pkg.org/badges/version/leine)](https://CRAN.R-project.org/package=leine)\n[![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://www.tidyverse.org/lifecycle/#experimental)\n\u003c!-- badges: end --\u003e\n\n[wikipedia:](https://en.wikipedia.org/wiki/Leine):\n\n\u003e The Leine (German: [ˈlaɪnə]; Old Saxon Lagina) is a river in Thuringia and Lower Saxony, Germany.\n\n[WAG](https://www.sub.uni-goettingen.de/kontakt/abteilungen-a-z/abteilungs-und-gruppendetails/abteilunggruppe/wissen-als-gemeingut/)\nruns several big data pipelines used in various data products.\n\nThese pipelines, though largely not themselves run in R, are here organised into an R package.\n\n## Design\n\nWAG is a relatively small team of data analysts, serving academic and librarian stakeholders with various data products.\n\nThe data engineering of our pipelines has to correspond to these constraints:\n\n- Our most important, and scarce resource is developer time.\n- Our most important, and hard target is reproducibility.\n\n### Priorities\n\nFrom this follows:\n\n1. Cheap compute is good, but **convenience** is better.\n    Our workloads are comparatively minor, labor costs are a much bigger driver.\n1. Special-purpose tools are good, but **standardising** on fewer tools is better.\n    Given our small, and sometimes churning team, we can only support very few tools.\n1. Working prototypes are good, but **reproducibility** is better.\n    For our academic (as well as librarian) stakeholders, reproduciblity trumps all else.\n1. Interactive, one-off results are good, but **automation**, **testing** and **documentation** are better.\n    Given churn (and vacation, context-switching, etc.), we must avoid low [bus factors](https://en.wikipedia.org/wiki/Bus_factor).\n    Data pipelines, especially, must be designed to be run and be maintainable without the original developer.\n\n### ELT\n\nOur data pipelines follow an extract-load-transform paradigm.\nThey are centered a \"data river\" (or data like) hosted on the Google Cloud Platform (GCP).\n\n1. Data river\n    1. Data is **extracted** from sources in its *rawest form* into to GCP Cloud Storage (for long-term versioned coldline storage).\n    1. Data is then **loaded** into GCP BigQuery.\n        If the source data is schemaless or noncompliant, it is loaded *without schema*, with entire unparsed entries as cells.\n1. Data warehouse\n    1. Data is then **transformed** into a canonical form in GCP BigQuery with a well-defined schema.\n1. Data mart\n    1. Data is then further transformed according to shared needs of WAGs data products on GCP BigQuery.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubugoe%2Fleine","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsubugoe%2Fleine","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubugoe%2Fleine/lists"}