{"id":31766069,"url":"https://github.com/mtes-mct/data-preparation-plugin","last_synced_at":"2025-10-10T00:29:09.052Z","repository":{"id":43453082,"uuid":"200813303","full_name":"MTES-MCT/data-preparation-plugin","owner":"MTES-MCT","description":"Airflow plugin with database hooks and operators for common data preparation tasks","archived":false,"fork":false,"pushed_at":"2023-05-08T20:27:51.000Z","size":107,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2023-08-04T12:11:18.452Z","etag":null,"topics":["airflow","data-science","etl","plugins"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/MTES-MCT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-08-06T08:50:28.000Z","updated_at":"2023-08-04T12:11:18.452Z","dependencies_parsed_at":"2023-01-23T01:46:05.148Z","dependency_job_id":null,"html_url":"https://github.com/MTES-MCT/data-preparation-plugin","commit_stats":null,"previous_names":[],"tags_count":9,"template":null,"template_full_name":null,"purl":"pkg:github/MTES-MCT/data-preparation-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MTES-MCT%2Fdata-preparation-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MTES-MCT%2Fdata-preparation-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MTES-MCT%2Fdata-preparation-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MTES-MCT%2Fdata-preparation-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/MTES-MCT","download_url":"https://codeload.github.com/MTES-MCT/data-preparation-plugin/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/MTES-MCT%2Fdata-preparation-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002358,"owners_count":26083356,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["airflow","data-science","etl","plugins"],"created_at":"2025-10-10T00:28:52.669Z","updated_at":"2025-10-10T00:29:09.044Z","avatar_url":"https://github.com/MTES-MCT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# data-preparation-plugin\nAirflow plugin with database hooks and operators for common data preparation tasks\n\n## Data preparation\n\nMost data preparation jobs can be seen as direct acyclic graph (DAG) of tasks where tasks can be of\ndifferent sorts:\n\n* downloading data from web urls into local filesystem\n* loading files from local filesystem into a PostgreSQL table\n* transforming data using Python or SQL functions:\n  * filtering\n  * calculating new columns\n  * formatting columns\n  * regex extraction\n  * ...\n* joining tables together\n\n\n![](./docs/images/data-preparation.png)\n\nThis plugin offers differents hooks and operators on top of Apache Airflow that makes this work easier.\n\n## Hooks\n\nIn Airflow, hooks are any kind of interface to external platforms and databases.\n\n### PostgresDataset\n\n#### Definition\n\nThis plugin defines a hook called `PostgresDataset` to interact with a PostgreSQL table in different ways:\n\n* read the schema of a dataset\n* read a dataset as Pandas dataframe\n* read a dataset row by row\n* write the schema of a dataset\n* write a Pandas dataframe to a dataset\n* write to a dataset row by row\n* reflect the content of a dataset into an SQL Alchemy mapper class\n\nThis last piece (reflection) is really useful when you want to use the ORM without declaring all fields explicitly\n\nReading and writing schema can be used to derive the schema of an output dataset from the schema of an input dataset in a transformation step. It makes everything DRY (don't repeat yourself). We only need to declare the schema of the first dataset explicitly.\n\n#### Examples\n\n*Transform a dataset using Pandas*\n\n```python\n\nconn_id = \"my_conn_id\"\nschema = \"my_schema\"\n\n# Input dataset\ndataset_1 = PostgresDataset(\n  name=\"table_1\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\n# Output dataset\ndataset_2 = PostgresDataset(\n  name=\"table_2\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\n# read dataset_1 into a Pandas dataframe\ndf = dataset_1.get_dataframe()\n\n# apply some transformation like adding a column\ndf[\"column_3\"] = df[\"column_1\"] + df[\"column_2\"]\n\n# read schema of input dataset\ndtype = dataset_1.read_dtype()\ncolumn_3 = Column(\"column_3\", Integer())\n\noutput_dtype = [*dtype, column_3]\n\n# write dataframe with dtype\ndataset_2.write_from_dataframe(\n  df,\n  dtype=output_dtype)\n```\n\n*Transform a dataset using plain Python*\n```python\nconn_id = \"my_conn_id\"\nschema = \"my_schema\"\n\n# Input dataset\ndataset_1 = PostgresDataset(\n  name=\"table_1\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\n# Output dataset\ndataset_2 = PostgresDataset(\n  name=\"table_2\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\n# keep onlu those columns, discarding others\nkeep = [\"column_1\", \"column_2\"]\n\n# transform schema\ndtype = dataset_1.read_dtype()\noutput_dtype = [\n  column for column\n  in dtype if column.name\n  in keep]\n\n# write schema\ndataset_2.write_dtype(output_dtype)\n\n# write data\nwith dataset_2.get_writer() as writer:\n  for row in dataset_1.iter_rows()\n    output_row = dict(\n      (column, row[column])\n      for column in row\n      if column in keep)\n    writer.write_row_dict(output_row)\n```\n\n\n*Use reflection and SQLAlchemy to perform a join*\n```python\n\nconn_id = \"my_conn_id\"\nschema = \"my_schema\"\n\n# Input datasets\ndataset_1 = PostgresDataset(\n  name=\"table_1\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\ndataset_2 = PostgresDataset(\n  name=\"table_2\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\n# Output dataset\ndataset_3 = PostgresDataset(\n  name=\"table_3\",\n  schema=schema,\n  postgres_conn_id=conn_id)\n\n# Write output schema\ndtype = [*dataset_1.read_dtype(), *dataset_2.read_dtype()]\ndtype = merge_dtype(dtype)\n\ndataset_3.write_schema(dtype)\n\n# Reflect input datasets to get ORM Mappers\nDataset1 = dataset_1.reflect()\nDataset2 = dataset_2.reflect()\n\n# Perform a join\nsession = dataset_1.get_session()\nq = session \\\n  .query(Dataset1, Dataset2)\n  .join(Dataset2, Dataset1.id == Dataset2.id)\n  .all()\n\n# Write records to output dataset\nwith dataset_3.get_writer() as writer:\n  for (row1, row2) in q:\n    output_row = {**row2dict(row1), **row2dict(row2)}\n    writer.write_row_dict(output_row)\n```\n\n## Operators\n\n### EmbulkOperator\n\nTODO\n\n### DownloadUnzipOperator\n\nTODO\n\n## Run the tests\n\nCreate a virtualenv and install dependencies\n\n```\npip install -r requirements.txt\n```\n\nInitialize Airflow metastore\n\n```\nairflow initdb\n```\n\nLaunch a local PostgreSQL instance\n\n```\ndocker run -p 5432:5432 postgres:9.6\n```\n\nRun the tests\n\n```\npython -m unittest\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtes-mct%2Fdata-preparation-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmtes-mct%2Fdata-preparation-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmtes-mct%2Fdata-preparation-plugin/lists"}