{"id":19714894,"url":"https://github.com/ajschofield/etl-project","last_synced_at":"2025-09-07T15:40:36.587Z","repository":{"id":255854694,"uuid":"851688883","full_name":"ajschofield/ETL-Project","owner":"ajschofield","description":"A solo continuation of the ETL pipeline project during the Data Engineering course at Northcoders","archived":false,"fork":false,"pushed_at":"2024-09-07T14:09:05.000Z","size":42,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"stable","last_synced_at":"2025-07-02T04:42:35.439Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://plane.ajschof.me/spaces/issues/7e21ffb65fb94dd989a107d045c3974b","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ajschofield.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-09-03T14:46:34.000Z","updated_at":"2024-09-08T18:32:27.000Z","dependencies_parsed_at":"2024-09-07T14:13:02.684Z","dependency_job_id":null,"html_url":"https://github.com/ajschofield/ETL-Project","commit_stats":null,"previous_names":["ajschofield/etl-project"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ajschofield/ETL-Project","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajschofield%2FETL-Project","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajschofield%2FETL-Project/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajschofield%2FETL-Project/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajschofield%2FETL-Project/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ajschofield","download_url":"https://codeload.github.com/ajschofield/ETL-Project/tar.gz/refs/heads/stable","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ajschofield%2FETL-Project/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":274058750,"owners_count":25215197,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-07T02:00:09.463Z","response_time":67,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T22:36:29.593Z","updated_at":"2025-09-07T15:40:36.410Z","avatar_url":"https://github.com/ajschofield.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# ETL-Project\n[![Python](https://img.shields.io/badge/Python-FFD43B?style=for-the-badge\u0026logo=python\u0026logoColor=blue)](https://www.python.org/)\n![Azure](https://img.shields.io/badge/azure-%230072C6.svg?style=for-the-badge\u0026logo=microsoftazure\u0026logoColor=white)\n[![Terraform](https://img.shields.io/badge/Terraform-7B42BC?style=for-the-badge\u0026logo=terraform\u0026logoColor=white)](https://www.terraform.io/)\n[![Postgresql](https://img.shields.io/badge/PostgreSQL-316192?style=for-the-badge\u0026logo=postgresql\u0026logoColor=white)](https://www.postgresql.org/)\n[![GitHub Actions](https://img.shields.io/badge/GitHub_Actions-2088FF?style=for-the-badge\u0026logo=github-actions\u0026logoColor=white)](https://github.com/features/actions)\n\nFor the original ToteSys project, please see [here](https://github.com/ajschofield/de-project-bentley).\n\nThis project implements a robust, serverless data processing platform that extracts data from an operational database, archives it in a data lake, and transforms it to be loaded into an easily accessible OLAP data warehouse. It is designed to be reliable, scalable and fully automated.\n\nThis platform includes the following key functions:\n- Extracts data from a PostgreSQL database at regular intervals\n- Stores raw data in a data lake for archival purposes\n- Transforms the data to conform to a star schema optimised for analytical queries\n- Loads the transformed data into a cloud-based data warehouse\n- Ensures data consistency, with a maximum delay of 30 minutes from source to warehouse\n\nThe original solution used Amazon Web Services, but this solo iteration will be using Azure, requiring a rewrite of the Terraform configuration and Python code.\n\nThe deadline for completion is the **end of September**.\n\n# Original Contributors\nBelow are the contributors to the original Totesys project.\n\u003ctable\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/ellsymonds\"\u003e\n        \u003cimg src=\"https://github.com/ellsymonds.png\" width=\"100px;\" alt=\"ellsymonds\"/\u003e\n        \u003cbr /\u003e\n        \u003csub\u003e\u003cb\u003eEllie Symonds\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/lian-manonog\"\u003e\n        \u003cimg src=\"https://github.com/lian-manonog.png\" width=\"100px;\" alt=\"lian-manonog\"/\u003e\n        \u003cbr /\u003e\n        \u003csub\u003e\u003cb\u003eLianmei Manon-og\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/T-Aji\"\u003e\n        \u003cimg src=\"https://github.com/T-Aji.png\" width=\"100px;\" alt=\"T-Aji\"/\u003e\n        \u003cbr /\u003e\n        \u003csub\u003e\u003cb\u003eTolu Ajibade\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n  \u003ctr\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/HastarTara\"\u003e\n        \u003cimg src=\"https://github.com/HastarTara.png\" width=\"100px;\" alt=\"HastarTara\"/\u003e\n        \u003cbr /\u003e\n        \u003csub\u003e\u003cb\u003eJoslin Rashleigh\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/bulve-ad\"\u003e\n        \u003cimg src=\"https://github.com/bulve-ad.png\" width=\"100px;\" alt=\"bulve-ad\"/\u003e\n        \u003cbr /\u003e\n        \u003csub\u003e\u003cb\u003eAnzelika Belotelova\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n    \u003ctd align=\"center\"\u003e\n      \u003ca href=\"https://github.com/ajschofield\"\u003e\n        \u003cimg src=\"https://github.com/ajschofield.png\" width=\"100px;\" alt=\"ajschofield\"/\u003e\n        \u003cbr /\u003e\n        \u003csub\u003e\u003cb\u003eAlex Schofield\u003c/b\u003e\u003c/sub\u003e\n      \u003c/a\u003e\n    \u003c/td\u003e\n  \u003c/tr\u003e\n\u003c/table\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajschofield%2Fetl-project","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fajschofield%2Fetl-project","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fajschofield%2Fetl-project/lists"}