{"id":21131208,"url":"https://github.com/derak-isaack/nyc-taxi-analytics","last_synced_at":"2026-05-07T09:31:42.329Z","repository":{"id":243020695,"uuid":"811240023","full_name":"derak-isaack/NYC-Taxi-Analytics","owner":"derak-isaack","description":"Data engineering ETL project using OLAP databases and DBT to perform analysis on NYC taxi data. ","archived":false,"fork":false,"pushed_at":"2024-07-30T19:33:19.000Z","size":1114,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-11-08T05:03:06.268Z","etag":null,"topics":["data-pipeline","dbt","duckdb","etl","olap-database","powerbi","prefect","python3","sql"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/derak-isaack.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-06T08:01:42.000Z","updated_at":"2024-07-30T19:34:54.000Z","dependencies_parsed_at":"2024-07-30T20:34:40.000Z","dependency_job_id":"42b8a2c1-ee4e-4b5b-9c00-58f6fc369142","html_url":"https://github.com/derak-isaack/NYC-Taxi-Analytics","commit_stats":null,"previous_names":["derak-isaack/nyc-taxi-analytics"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/derak-isaack/NYC-Taxi-Analytics","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derak-isaack%2FNYC-Taxi-Analytics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derak-isaack%2FNYC-Taxi-Analytics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derak-isaack%2FNYC-Taxi-Analytics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derak-isaack%2FNYC-Taxi-Analytics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/derak-isaack","download_url":"https://codeload.github.com/derak-isaack/NYC-Taxi-Analytics/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/derak-isaack%2FNYC-Taxi-Analytics/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32731233,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-07T02:14:30.463Z","status":"ssl_error","status_checked_at":"2026-05-07T02:14:29.405Z","response_time":62,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-pipeline","dbt","duckdb","etl","olap-database","powerbi","prefect","python3","sql"],"created_at":"2024-11-20T05:50:54.514Z","updated_at":"2026-05-07T09:31:42.310Z","avatar_url":"https://github.com/derak-isaack.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n## \u003cdiv style=\"padding: 35px;color:white;margin:10;font-size:200%;text-align:center;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://images.pexels.com/photos/7078619/pexels-photo-7078619.jpeg?auto=compress\u0026cs=tinysrgb\u0026w=1260\u0026h=750\u0026dpr=1)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e\u003cstrong\u003e NEW YORK CITY TLC TAXI DATA PIPELINE  \u003c/strong\u003e\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e \n\n![dbt](https://img.shields.io/badge/dbt-FF694B?logo=dbt\u0026logoColor=fff\u0026style=for-the-badge)\n![Duckdb](https://img.shields.io/badge/DuckDB-FFF000?logo=duckdb\u0026logoColor=000\u0026style=for-the-badge)\n![Apache-parquet](https://img.shields.io/badge/Apache%20Parquet-50ABF1?logo=apacheparquet\u0026logoColor=fff\u0026style=for-the-badge)\n\n\n### \u003cdiv style=\"padding: 20px;color:white;margin:10;font-size:90%;text-align:left;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://w0.peakpx.com/wallpaper/957/661/HD-wallpaper-white-marble-white-stone-texture-marble-stone-background-white-stone.jpg)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e Project Overview\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e\n\nThis is an ETL(`Extract-Transform-Load`) data pipeline using `DUCKDB` for extraction \u0026 loading and `DBT` for transformation. The data to be transformed is from the [NYC-TLC-website](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page#:~:text=Yellow%20and%20green%20taxi%20trip,and%20driver%2Dreported%20passenger%20counts.) for the month of May 2024. The data columns description can be found [here](data_dictionary_trip_records_green.pdf). \n\n### \u003cdiv style=\"padding: 20px;color:white;margin:10;font-size:90%;text-align:left;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://w0.peakpx.com/wallpaper/957/661/HD-wallpaper-white-marble-white-stone-texture-marble-stone-background-white-stone.jpg)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e Objectives\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e\n\nThe transformation objectives include building various transformation models for further analysis. \n\n1. `Route traffic model` using the `Pick-Up` \u0026 `Drop-off` locations. \n\n2. `Hourly daily Server outage model` using the `Drop-Off` location. Look for `drop off` locations that are prone to server outages in terms of sending trip detals to the server. They are marked as `N`. This is for analysis to get which hours of the day are mostly affected by severe server outages and might need further action. \n\n3. `Tip amount model`. Analyze the tips by different customers to different vendors. \n\n4. `Daily-hourly traffic model`. Model to analyze passenger count for `every 24hrs per day` for further passenger trend analysis. \n\n5. `Pick-up trend model` for assesing passenger counts in various pick up locations. \n\n### \u003cdiv style=\"padding: 20px;color:white;margin:10;font-size:90%;text-align:left;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://w0.peakpx.com/wallpaper/957/661/HD-wallpaper-white-marble-white-stone-texture-marble-stone-background-white-stone.jpg)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e Data Extraction \u0026 Loading\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e\n\nFor the data extraction, [DuckDB](https://duckdb.org/docs/data/parquet/overview) has extensive options for performing `data extraction` explicitly. Of importance is to use the `fetch_df()` in the `SQL` queries when seeking to find the data structure and format. How the final transformation models would look like can be found [here](taxi.ipynb). \n\n[DuckDB](https://duckdb.org/docs/installation/index?version=stable\u0026environment=cli\u0026platform=win\u0026download_method=package_manager) will also be used for Loading the transformed data in table formart as will be defined in the `transformation models` using the `{{config(materialized='table')}} command. \n\n### \u003cdiv style=\"padding: 20px;color:white;margin:10;font-size:90%;text-align:left;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://w0.peakpx.com/wallpaper/957/661/HD-wallpaper-white-marble-white-stone-texture-marble-stone-background-white-stone.jpg)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e Data Transformation\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e\n\nFor the transformation, [DBT](https://docs.getdbt.com/docs/introduction)(Data Build Tool) comes in very handy in handling the transformation logic using the normal `SQL` syntax. The [transformation-models](TLC_NYC/models) are all chained to the first model for further analysis of the data. \n\nTo initialize a `DBT project` together with `DuckDB OLAP database`, the following commands are to be performed in order.\n\n* `pip install dbt-duckdb`\n\n* `dbt init`\n\n* `dbt debug` to test that everything is working fine before proceeding. \n\n* `dbt run` after defining the transformation models. Incase of any error the `logs` should be checked. \n\nFor a succesfull dbt model, the following should be printed on the terminal:\n\n![dbt-final-screenshot](\u003cdbt screenshot.png\u003e)\n\n### \u003cdiv style=\"padding: 20px;color:white;margin:10;font-size:90%;text-align:left;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://w0.peakpx.com/wallpaper/957/661/HD-wallpaper-white-marble-white-stone-texture-marble-stone-background-white-stone.jpg)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e Dashboard\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e\n\nThe transformation models are then visualized using `Power BI` which offers quick interactive visualization charts with the key `KPI's`. \n\n![Dashboard](\u003cPowerBidashboard.png\u003e)\n\n### \u003cdiv style=\"padding: 20px;color:white;margin:10;font-size:90%;text-align:left;display:fill;border-radius:10px;overflow:hidden;background-image: url(https://w0.peakpx.com/wallpaper/957/661/HD-wallpaper-white-marble-white-stone-texture-marble-stone-background-white-stone.jpg)\"\u003e\u003cb\u003e\u003cspan style='color:black'\u003e Prefect Integration\u003c/span\u003e\u003c/b\u003e \u003c/div\u003e\n\n`Prefect-dbt-flow` [library](https://github.com/datarootsio/prefect-dbt-flow) offers quick simple integration with orchestration and pipeline monitoring. \n\n![Orchestration-prefect](prefect-flow.png)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderak-isaack%2Fnyc-taxi-analytics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fderak-isaack%2Fnyc-taxi-analytics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fderak-isaack%2Fnyc-taxi-analytics/lists"}