{"id":19756377,"url":"https://github.com/mara/mara-etl-tools","last_synced_at":"2025-02-28T01:51:57.321Z","repository":{"id":57439604,"uuid":"129076149","full_name":"mara/mara-etl-tools","owner":"mara","description":"Utilities for creating ETL pipelines with mara","archived":false,"fork":false,"pushed_at":"2022-05-20T21:05:54.000Z","size":56,"stargazers_count":36,"open_issues_count":0,"forks_count":5,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-02-11T09:08:42.554Z","etag":null,"topics":["data-integration","date-dimension","etl","sql","sql-utils"],"latest_commit_sha":null,"homepage":null,"language":"PLpgSQL","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mara.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-11T10:23:56.000Z","updated_at":"2024-07-10T12:05:02.000Z","dependencies_parsed_at":"2022-09-26T17:20:26.276Z","dependency_job_id":null,"html_url":"https://github.com/mara/mara-etl-tools","commit_stats":null,"previous_names":[],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-etl-tools","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-etl-tools/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-etl-tools/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mara%2Fmara-etl-tools/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mara","download_url":"https://codeload.github.com/mara/mara-etl-tools/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241086804,"owners_count":19907336,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-integration","date-dimension","etl","sql","sql-utils"],"created_at":"2024-11-12T03:15:44.276Z","updated_at":"2025-02-28T01:51:57.304Z","avatar_url":"https://github.com/mara.png","language":"PLpgSQL","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Mara ETL Tools\n\n[![Build Status](https://travis-ci.org/mara/mara-etl-tools.svg?branch=master)](https://travis-ci.org/mara/mara-etl-tools)\n[![PyPI - License](https://img.shields.io/pypi/l/mara-etl-tools.svg)](https://github.com/mara/mara-etl-tools/blob/master/LICENSE)\n[![PyPI version](https://badge.fury.io/py/mara-etl-tools.svg)](https://badge.fury.io/py/mara-etl-tools)\n[![Slack Status](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack\u0026style=social)](https://communityinviter.com/apps/mara-users/public-invite)\n\nA collection of utilities around [Project A](https://project-a.com/)'s best practices for creating [data integration pipelines](https://github.com/mara/mara-pipelines) with Mara. The package is intended as a start for new projects. Forks/ copies are preferred over PRs.\n\nFor more details on how to use this package, have a look at the [mara example project 1](https://github.com/mara/mara-example-project-1) and [mara example project 2](https://github.com/mara/mara-example-project-2).\n\n\nThe package consists of a number modules that all can be used independently from each other:\n\n## SQL utility functions\n\nFunction `initialize_utils` in [etl_tools/initialize_utils/__init__.py](etl_tools/initialize_utils/__init__.py) returns a pipeline that creates a `util` schema with a number of PostgreSQL functions for organizing data pipelines. Add to your root pipeline like this:\n\n```python\nfrom etl_tools import initialize_utils\n\nmy_pipeline.add(initialize_utils.utils_pipeline(with_hll=True, with_cstore_fdw=True))\n```\n\nPlease have a look at the .sql files in [etl_tools/initialize_utils](etl_tools/initialize_utils) for available functions.\n\n\n## Schema copying\n\nThe file The file [etl_tools/schema_copying.py](etl_tools/schema_copying.py) contains the function `add_schema_copying_to_pipeline` that copies a PostgreSQL database schema from on host to another at the end of a pipeline run. This is useful for running the ETL and frontend tools on different database servers so that a running ETL does not affect the performance of dashboard queries.\n\n\nGiven that there is a pipline `my_pipeline` that has a number of child pipelines with the `Schema` label set to the respective schema to copy, then this is how the schema copying can be added to those child pipelines.\n\n```python\nfrom mara_db import dbs\nfrom mara_pipelines.commands.sql import ExecuteSQL\nfrom mara_pipelines.pipelines import Task\nfrom etl_tools.schema_copying import add_schema_copying_to_pipeline\n\n# when etl und frontend db are different, add schema copying\nif dbs.db('mdwh-etl').database != dbs.db('mdwh-frontend').database \\\n        or dbs.db('mdwh-etl').host != dbs.db('mdwh-frontend').host:\n\n    # run some of the files from etl_tools/initalize_utils in frontend db\n    initialize_frontend_db_commands = [ExecuteSQL(\n        sql_statement=\"DROP SCHEMA IF EXISTS util CASCADE; CREATE SCHEMA util;\", db_alias='mdwh-frontend')]\n\n    for file_name in ['schema_switching.sql', 'data_sets.sql', 'hll.sql', 'cstore_fdw.sql']:\n        initialize_frontend_db_commands.append(\n            ExecuteSQL(sql_file_name=str(\n                my_pipeline.nodes['utils'].nodes['initialize_utils'].base_path() / file_name),\n                db_alias='mdwh-frontend'))\n\n    my_pipeline.nodes['utils'].add(\n        Task(id='initialize_frontend_db',\n             description='Adds some functions to the frontend db so that schema copying works',\n             commands=initialize_frontend_db_commands))\n\n    # Add schema copying for time schema\n    add_schema_copying_to_pipeline(pipeline=my_pipeline.nodes['utils'].nodes['create_time_dimensions'],\n                                   schema_name='time',\n                                   source_db_alias='dwh-etl', target_db_alias='dwh-frontend')\n\n    # Add schema copying to all root pipelines\n    for pipeline in my_pipeline.nodes.values():\n        if \"Schema\" in pipeline.labels:\n            schema = pipeline.labels['Schema']\n            add_schema_copying_to_pipeline(pipeline=pipeline, schema_name=schema + '_next',\n                                           source_db_alias='dwh-etl', target_db_alias='dwh-frontend')\n            pipeline.final_node.commands_after.append(\n                ExecuteSQL(sql_statement=f\"SELECT util.replace_schema('{schema}', '{schema}_next')\",\n                           db_alias='mdwh-frontend')\n            )\n```\n \n\n## Time dimensions\n\nThe file [etl_tools/create_time_dimensions/__init__.py](etl_tools/create_time_dimensions/__init__.py) defines a pipeline that creates and updates `time` schema with the tables `day` and `duration`:\n\n```\nselect * from time.day order by _date desc limit 10;\n     day_id  |     day_name     | year_id | iso_year_id | quarter_id | quarter_name | month_id | month_name | week_id |  week_name   | day_of_week_id | day_of_week_name | day_of_month_id |   _date    \n   ----------+------------------+---------+-------------+------------+--------------+----------+------------+---------+--------------+----------------+------------------+-----------------+------------\n    20190815 | Thu, Aug 15 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201933 | 2019 - CW 33 |              4 | Thursday         |              15 | 2019-08-15\n    20190814 | Wed, Aug 14 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201933 | 2019 - CW 33 |              3 | Wednesday        |              14 | 2019-08-14\n    20190813 | Tue, Aug 13 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201933 | 2019 - CW 33 |              2 | Tuesday          |              13 | 2019-08-13\n    20190812 | Mon, Aug 12 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201933 | 2019 - CW 33 |              1 | Monday           |              12 | 2019-08-12\n    20190811 | Sun, Aug 11 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201932 | 2019 - CW 32 |              7 | Sunday           |              11 | 2019-08-11\n    20190810 | Sat, Aug 10 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201932 | 2019 - CW 32 |              6 | Saturday         |              10 | 2019-08-10\n    20190809 | Fri, Aug 09 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201932 | 2019 - CW 32 |              5 | Friday           |               9 | 2019-08-09\n    20190808 | Thu, Aug 08 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201932 | 2019 - CW 32 |              4 | Thursday         |               8 | 2019-08-08\n    20190807 | Wed, Aug 07 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201932 | 2019 - CW 32 |              3 | Wednesday        |               7 | 2019-08-07\n    20190806 | Tue, Aug 06 2019 |    2019 |        2019 |      20193 | 2019 Q3      |   201908 | 2019 Aug   |  201932 | 2019 - CW 32 |              2 | Tuesday          |               6 | 2019-08-06\n```\n\n```\nselect * from time.duration where duration_id \u003e= 0 order by duration_id limit 10;\n duration_id | days | days_name | weeks | weeks_name | four_weeks | four_weeks_name | months | months_name | sixth_years | sixth_years_name | half_years | half_years_name | years | years_name \n-------------+------+-----------+-------+------------+------------+-----------------+--------+-------------+-------------+------------------+------------+-----------------+-------+------------\n           0 |    0 | 0 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           1 |    1 | 1 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           2 |    2 | 2 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           3 |    3 | 3 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           4 |    4 | 4 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           5 |    5 | 5 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           6 |    6 | 6 days    |     0 | 0-6 days   |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           7 |    7 | 7 days    |     1 | 7-13 days  |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           8 |    8 | 8 days    |     1 | 7-13 days  |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n           9 |    9 | 9 days    |     1 | 7-13 days  |          0 | 0-27 days       |      0 | 0-29 days   |           0 | 0-59 days        |          0 | 0-179 days      |     0 | 0-359 days\n\n```\n\nAdd the pipeline to your project with \n\n```bash\nfrom etl_tools import create_time_dimensions\n\nmy_pipeline.add(create_time_dimensions.pipeline)\n```\n\nSet min and max dates by overwriting the `first_date_in_time_dimensions` and `last_date_in_time_dimensions` in [etl_tools/config.py](etl_tools/config.py).\n\n\n## Euro currency exchange rates\n\nThe file [etl_tools/load_euro_exchange_rates/__init__.py](etl_tools/create_time_dimensions/__init__.py) contains a pipeline that loads (historic) Euro exchange rates from the European central bank. \n\n\nAdd to your pipeline with \n\n```bash\nfrom etl_tools import load_euro_exchange_rates\n\nmy_pipeline.add(load_euro_exchange_rates.euro_exchange_rates_pipeline('db-alias'))\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmara%2Fmara-etl-tools","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmara%2Fmara-etl-tools","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmara%2Fmara-etl-tools/lists"}