{"id":46519343,"url":"https://github.com/robert-koch-institut/mex-extractors","last_synced_at":"2026-03-06T19:09:31.110Z","repository":{"id":317665689,"uuid":"1002813002","full_name":"robert-koch-institut/mex-extractors","owner":"robert-koch-institut","description":"RKI Metadata Exchange | Collection of ETL-pipelines to get metadata from various sources, transform it into a common format and load it into configurable sinks.","archived":false,"fork":false,"pushed_at":"2026-02-24T15:11:51.000Z","size":1575,"stargazers_count":2,"open_issues_count":6,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-02-24T15:43:59.276Z","etag":null,"topics":["etl","python","research-data"],"latest_commit_sha":null,"homepage":"https://robert-koch-institut.github.io/mex-extractors/","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/robert-koch-institut.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":"AUTHORS","dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-06-16T07:24:09.000Z","updated_at":"2026-02-24T09:48:24.000Z","dependencies_parsed_at":"2025-10-23T13:11:56.102Z","dependency_job_id":"5d531597-754c-4618-b79c-e459351926cd","html_url":"https://github.com/robert-koch-institut/mex-extractors","commit_stats":null,"previous_names":["robert-koch-institut/mex-extractors"],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/robert-koch-institut/mex-extractors","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-koch-institut%2Fmex-extractors","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-koch-institut%2Fmex-extractors/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-koch-institut%2Fmex-extractors/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-koch-institut%2Fmex-extractors/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/robert-koch-institut","download_url":"https://codeload.github.com/robert-koch-institut/mex-extractors/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/robert-koch-institut%2Fmex-extractors/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30192485,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-06T19:07:06.838Z","status":"ssl_error","status_checked_at":"2026-03-06T18:57:34.882Z","response_time":250,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["etl","python","research-data"],"created_at":"2026-03-06T19:09:26.285Z","updated_at":"2026-03-06T19:09:31.096Z","avatar_url":"https://github.com/robert-koch-institut.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MEx extractors\n\nETL pipelines for the RKI Metadata Exchange.\n\n[![cookiecutter](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/cookiecutter.yml/badge.svg)](https://github.com/robert-koch-institut/mex-template)\n[![cve-scan](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/cve-scan.yml/badge.svg)](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/cve-scan.yml)\n[![documentation](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/documentation.yml/badge.svg)](https://robert-koch-institut.github.io/mex-extractors)\n[![linting](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/linting.yml/badge.svg)](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/linting.yml)\n[![open-code](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/open-code.yml/badge.svg)](https://gitlab.opencode.de/robert-koch-institut/mex/mex-extractors)\n[![testing](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/testing.yml/badge.svg)](https://github.com/robert-koch-institut/mex-extractors/actions/workflows/testing.yml)\n\n## Project\n\nThe Metadata Exchange (MEx) project is committed to improve the retrieval of RKI\nresearch data and projects. How? By focusing on metadata: instead of providing the\nactual research data directly, the MEx metadata catalog captures descriptive information\nabout research data and activities. On this basis, we want to make the data FAIR[^1] so\nthat it can be shared with others.\n\nVia MEx, metadata will be made findable, accessible and shareable, as well as available\nfor further research. The goal is to get an overview of what research data is available,\nunderstand its context, and know what needs to be considered for subsequent use.\n\nRKI cooperated with D4L data4life gGmbH for a pilot phase where the vision of a\nFAIR metadata catalog was explored and concepts and prototypes were developed.\nThe partnership has ended with the successful conclusion of the pilot phase.\n\nAfter an internal launch, the metadata will also be made publicly available and thus be\navailable to external researchers as well as the interested (professional) public to\nfind research data from the RKI.\n\nFor further details, please consult our\n[project page](https://www.rki.de/DE/Aktuelles/Publikationen/Forschungsdaten/MEx/metadata-exchange-plattform-mex-node.html).\n\n[^1]: FAIR is referencing the so-called\n[FAIR data principles](https://www.go-fair.org/fair-principles/) – guidelines to make\ndata Findable, Accessible, Interoperable and Reusable.\n\n**Contact** \\\nFor more information, please feel free to email us at [mex@rki.de](mailto:mex@rki.de).\n\n### Publisher\n\n**Robert Koch-Institut** \\\nNordufer 20 \\\n13353 Berlin \\\nGermany\n\n## Package\n\nThe `mex-extractors` package implements a variety of ETL pipelines to **extract**\nmetadata from primary data sources using a range of different technologies and\nprotocols. Then, we **transform** the metadata into a standardized format using models\nprovided by `mex-common`. The last step in this process is to **load** the harmonized\nmetadata into a sink (file output, API upload, etc).\n\n## License\n\nThis package is licensed under the [MIT license](/LICENSE). All other software\ncomponents of the MEx project are open-sourced under the same license as well.\n\n## Development\n\n### Installation\n\n- install python on your system\n- on unix, run `make install`\n- on windows, run `.\\mex.bat install`\n\n### Linting and testing\n\n- run all linters with `make lint` or `.\\mex.bat lint`\n- run unit and integration tests with `make test` or `.\\mex.bat test`\n- run just the unit tests with `make unit` or `.\\mex.bat unit`\n\n### Updating dependencies\n\n- update boilerplate files with `cruft update`\n- update global requirements in `requirements.txt` manually\n- update git hooks with `pre-commit autoupdate`\n- update package dependencies using `uv sync --upgrade`\n- update github actions in `.github/workflows/*.yml` manually\n\n### Creating release\n\n- run `mex release RULE` to release a new version where RULE determines which part of\n  the version to update and is one of `major`, `minor`, `patch`.\n\n### Container workflow\n\n- build image with `make image`\n- run directly using docker `make run`\n- start with docker compose `make start`\n\n## Commands\n\n- run `uv run {command} --help` to print instructions\n- run `uv run {command} --debug` for interactive debugging\n\n### dagster\n\n- `uv run dagster dev` to launch a local dagster UI\n\n### all extractors\n\n- `uv run all-extractors` executes all extractors\n- execute only in local or dev environment\n\n### artificial extractor\n\n- `uv run artificial` creates deterministic artificial sample data\n- execute only in local or dev environment\n\n### biospecimen extractor\n\n- `uv run biospecimen` extracts sources from the Biospecimen excel files\n\n### blueant extractor\n\n- `uv run blueant` extracts sources from the Blue Ant project management software\n\n### confluence-vvt extractor\n\n- `uv run confluence-vvt` extracts sources from the VVT confluence page\n\n### consent-mailer\n\n- `uv run consent-mailer` send emails to collect publishing consents\n\n### contact-point\n\n- `uv run contact-point` extracts default contact points\n\n### datscha-web extractor\n\n- `uv run datscha-web` extracts sources from the datscha web app\n\n### endnote extractor\n\n- `uv run endnote` extracts from endnote XML files\n\n### ff-projects extractor\n\n- `uv run ff-projects` extracts sources from the FF Projects excel file\n\n### grippeweb extractor\n\n- `uv run grippeweb` extracts grippeweb metadata from grippeweb database\n\n### ifsg extractor\n\n- `uv run ifsg` extracts sources from the ifsg data base\n\n### international-projects extractor\n\n- `uv run international-projects` extracts sources from the international projects excel\n\n### kvis extractor\n\n- `uv run kvis` extracts KVIS metadata from KVIS database\n\n### odk extractor\n\n- `uv run odk` extracts ODK survey data from excel files\n\n### open-data extractor\n\n- `uv run open-data` extracts Open Data sources from the Zenodo API\n\n### seq-repo extractor\n\n- `uv run seq-repo` extracts sources from seq-repo JSON file\n\n### sumo extractor\n\n- `uv run sumo` extract sumo data from xlsx files\n\n### synopse extractor\n\n- `uv run synopse` extracts synopse data from report-server exports\n\n### voxco extractor\n\n- `uv run voxco` extracts voxco data from voxco JSON files\n\n### publisher\n\n- `uv run publisher` gets merged items from backend and publishes them into sink\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobert-koch-institut%2Fmex-extractors","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frobert-koch-institut%2Fmex-extractors","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frobert-koch-institut%2Fmex-extractors/lists"}