https://github.com/astrazeneca/magnus-extensions
Extensions packages for magnus
https://github.com/astrazeneca/magnus-extensions
Last synced: 8 months ago
JSON representation
Extensions packages for magnus
- Host: GitHub
- URL: https://github.com/astrazeneca/magnus-extensions
- Owner: AstraZeneca
- License: apache-2.0
- Created: 2022-02-03T16:20:34.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-08-28T05:02:29.000Z (almost 3 years ago)
- Last Synced: 2025-09-09T11:50:59.016Z (9 months ago)
- Language: Python
- Size: 2.74 MB
- Stars: 3
- Watchers: 1
- Forks: 1
- Open Issues: 5
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Welcome to Magnus Extensions
Documentation of the extensions are available at: https://astrazeneca.github.io/magnus-extensions/
This repository provides all the extensions to [magnus core package](https://github.com/AstraZeneca/magnus-core).
Magnus provides 5 essential services:
- Executor: A way to define and execute/transpile dag definition.
- Catalog: An artifact store used to log and store data files generated during a pipeline execution.
- Secrets Handler: A framework to handle secrets from different providers.
- Logging: A comprehensive and automatic logging framework to capture essential information of a pipeline execution.
- Experiment Tracking: A framework to interact with different experiment tracking tools.
Below is the table of all the available extensions to the above services:
| Service | Description | Availability |
| :---------: | :----------------------------------: | :-------------: |
| **Executors** | | |
| Local | Run the pipeline on local machine (default) | Part of Magnus core |
| Local Containers | Run the pipeline on local containers | Part of Magnus core |
| **Catalog** | | |
| Do Nothing | Provides no cataloging functionality | Part of Magnus core |
| File System | Uses local file system (default) | Part of Magnus core |
| S3 | Uses S3 as a catalog | magnus_extension_catalog_s3 |
| **Secrets** | | |
| Do Nothing | Provides no secrets handler (default) | Part of Magnus core |
| Dot Env | Uses a file as secrets | Part of Magnus core |
| Environment Variables | Gets secrets from Environmental variables | Part of Magnus core |
| **Logging** | | |
| Buffered | Uses the run time buffer as logger (default) | Part of Magnus core |
| File System | Uses a file system as run log store | Part of Magnus core |
| S3 | Uses S3 to store logs | magnus_extension_datastore_s3 |
| **Experiment Tracking** | | |
| Do Nothing | Provides no experiment tracking (default) | Part of Magnus core |