{"id":27153954,"url":"https://github.com/tyriek-cloud/nyc-dca-etl","last_synced_at":"2026-01-21T21:35:24.354Z","repository":{"id":157671684,"uuid":"613441910","full_name":"Tyriek-cloud/NYC-DCA-ETL","owner":"Tyriek-cloud","description":"Created an ETL pipeline to merge two CSV files (converted to JSON) into a parquet file using Azure Data Factory, The data was extracted from NYC Open Data: https://opendata.cityofnewyork.us/ and I created a Blob Container within an existing storage account.","archived":false,"fork":false,"pushed_at":"2023-03-14T14:27:03.000Z","size":28,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"DCA-ETL","last_synced_at":"2025-04-08T16:19:13.684Z","etag":null,"topics":["azure","azure-data-factory","blob-storage","data","data-engineering","etl-pipeline"],"latest_commit_sha":null,"homepage":"https://opendata.cityofnewyork.us/","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Tyriek-cloud.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-13T15:19:27.000Z","updated_at":"2024-03-31T13:26:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"d9b62429-349b-4fba-a6f6-e656bfbf8173","html_url":"https://github.com/Tyriek-cloud/NYC-DCA-ETL","commit_stats":null,"previous_names":["tyriek-cloud/nyc-dca-etl"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tyriek-cloud%2FNYC-DCA-ETL","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tyriek-cloud%2FNYC-DCA-ETL/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tyriek-cloud%2FNYC-DCA-ETL/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Tyriek-cloud%2FNYC-DCA-ETL/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Tyriek-cloud","download_url":"https://codeload.github.com/Tyriek-cloud/NYC-DCA-ETL/tar.gz/refs/heads/DCA-ETL","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247877963,"owners_count":21011158,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["azure","azure-data-factory","blob-storage","data","data-engineering","etl-pipeline"],"created_at":"2025-04-08T16:19:14.448Z","updated_at":"2026-01-21T21:35:24.314Z","avatar_url":"https://github.com/Tyriek-cloud.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# NYC DCA ETL\n\nThis was a project done in Azure Data Factory. \n\nI began by extracting data from New York City Open Data: https://opendata.cityofnewyork.us/\n\nFrom there, I created a Blob Container within an existing storage account. Then I initialized Azure Data Factory to do a series of T-SQL transformations on CSV files. I ultimately wanted to load data into a parquet file. The dataflow looks like this:\n\n![image](https://user-images.githubusercontent.com/62261407/224825767-5fed9d29-175a-45cb-b914-6cea558afa56.png)\n\nThe final, loaded result of the ETL process resulted in the creation of a parquet file hosted within a generated blob in the container:\n\n![image](https://user-images.githubusercontent.com/62261407/224824921-bf381f03-1c8f-4f73-bc84-ba38e659d3ab.png)\n\nI then went back into my container to look for any issues to trouble shoot. There were no issues to resolve, so I monitored activity in my container in a private dashboard:\n\n![image](https://user-images.githubusercontent.com/62261407/225027680-e834c12a-e771-4284-bb86-89f2433afdbd.png)\n![image](https://user-images.githubusercontent.com/62261407/225027790-1646533d-1a54-4b3c-abd2-552dfc35a519.png)\n![image](https://user-images.githubusercontent.com/62261407/225027867-d180b4d3-7a78-4fb0-9275-c18ad4148b9e.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftyriek-cloud%2Fnyc-dca-etl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftyriek-cloud%2Fnyc-dca-etl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftyriek-cloud%2Fnyc-dca-etl/lists"}