{"id":23740164,"url":"https://github.com/harshnarayanjha/taxi_analysis","last_synced_at":"2026-05-20T05:01:40.778Z","repository":{"id":269423215,"uuid":"907298774","full_name":"HarshNarayanJha/taxi_analysis","owner":"HarshNarayanJha","description":"NYC Taxi fare analysis using data science/machine learning","archived":false,"fork":false,"pushed_at":"2024-12-25T07:34:16.000Z","size":1050,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-05-24T11:08:40.899Z","etag":null,"topics":["data-science","machine-learning","python","taxifare"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/HarshNarayanJha.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-12-23T09:18:45.000Z","updated_at":"2025-01-04T03:03:58.000Z","dependencies_parsed_at":null,"dependency_job_id":"ec829809-15e3-41d0-a322-ee50df4bb1ed","html_url":"https://github.com/HarshNarayanJha/taxi_analysis","commit_stats":null,"previous_names":["harshnarayanjha/taxi_analysis"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/HarshNarayanJha/taxi_analysis","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarshNarayanJha%2Ftaxi_analysis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarshNarayanJha%2Ftaxi_analysis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarshNarayanJha%2Ftaxi_analysis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarshNarayanJha%2Ftaxi_analysis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/HarshNarayanJha","download_url":"https://codeload.github.com/HarshNarayanJha/taxi_analysis/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/HarshNarayanJha%2Ftaxi_analysis/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267141024,"owners_count":24041978,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-26T02:00:08.937Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","machine-learning","python","taxifare"],"created_at":"2024-12-31T09:47:21.245Z","updated_at":"2025-10-06T22:27:03.848Z","avatar_url":"https://github.com/HarshNarayanJha.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# New York Taxi Analysis Project\n\nIn this Data Analysis/Machine Learning project I play with the New York Taxi dataset.\n\nThere is a streamlit app to test the model and predict outputs. It can be started using\n\n```shell\nstreamlit run web_app.py\n```\n\n## Problem Definition\n\nPredict the average money spent on taxi rides for each region of New York per given day and hour.\n\nThis problem is a Supervised Regression problem.\n\n- Supervised: We have the actual values that we are trying to predict (in the train dataset)\n- Regression: We are trying to predict a continuous value (as opossed to categorical)\n\n## The Dataset\n\nThis project uses the `TLC Trip Record Data` dataset from `nyc.gov` (The NYC Taxi and Limousine Commission aka TLC).\nThe public dataset is available for download at [nyc.gov](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page)\n\nThe dataset contains a total of 3833771 records and 19 columns.\n\n```shell\n(3833771, 19)\n```\n\nThey use `parquet` data format for increased efficiency. See [Apache Parquet documentation](https://parquet.apache.org/docs/) for more details.\n\nRelavent supporting data can be found on the same page as for the dataset download\n\n- [Trip Record User Guide](https://www.nyc.gov/assets/tlc/downloads/pdf/trip_record_user_guide.pdf)\n- [Meaning of Column names](https://www.nyc.gov/assets/tlc/downloads/pdf/data_dictionary_trip_records_yellow.pdf)\n- [Taxi Zone Lookup Table](https://d37ci6vzurychx.cloudfront.net/misc/taxi_zone_lookup.csv)\n\n## Problems with raw datasets\n\nHere is the histogram of the raw data read from the `data/yellow_tripdata_2024-10.parquet`\n\n![Initial histogram of data](images/initial_hist.png)\n\nIt is clear that the dataset contains various illegal values, such as dates earlier than Oct 2024, despite being a Oct 2024 dataset.\nAlso, `passenger_count` is 0 for some records, which seems to be nonsense at first.\nA few trips have distances over 50,000 miles which is just ridiculous.\nAnd the most interesting part, `total_amount` is 0 and even negative in some records.\n\nThis calls for Data Cleaning and Preprocessing, and it requires a bit of domain knowledge.\nBy looking at the `Meaning of Column names` document linked above, we see that there is a column called `payment_type`\n\n```\nA numeric code signifying how the passenger paid for the trip.\n1= Credit card\n2= Cash\n3= No charge\n4= Dispute\n5= Unknown\n6= Voided trip\n```\n\nSo there are some rides with disputes and voided trips and even no charge. This justifies 0 and -ve `total_amount`s it seems.\n\n## Data Cleaning\n\nBy looking at the number of records having a -ve `total_amount`\n\n```python\nprint(taxi_data[taxi_data[\"total_amount\"] \u003c 0].shape)\n```\n\n```shell\n(61426, 9)\n```\n\nWe see that it is a mere 61k compared to 3 Million records we have. We explore the data a bit more and find out that most of the -ve\n`total_amount` are from `payment_type` 3 and 4. Hence we simply drop those records.\n\nBut what about very high values, like someone paid $2000 for a taxi trip!\nAgain by analyzing the data, only 1350 records have payments above $250, so we drop them\n\nOnly 22 records have `trip_distance` \u003e 100 miles (that's a lot for taxi). We drop them.\n\nIt is very important to not have outliers and false values in the dataset, as it can confuse the model later.\nThis is why we always collect as much data as we can since real life data will always have \"noise\" like this\n\nDatasets may also contain missing values. There are cetain approaches to handle those values.\nWe can fill them in with averages, interpolate existing values, or simply drop them if count is negligible, which is for out case.\n\nNext, we convert the columns to appropriate types such as `datetime`, `float64`, `int32/64`, `boolean` or `object` as suitable for better training results.\nIt is useful to have `day`, `month` and `hour` columns instead of one `datetime` column, since it yields more features.\n\nData aggregation is an important step to reduce the number of data points wherever meaningful to provide more information.\n\n## Benchmark Model\n\nBefore actually spinning up a ML model training process, we might want to first benchmark out data by making a benchmark model.\nThis helps us to pre-identify overfitting, underfitting and other unfavorable scenarios.\n\nFeatures for benchmarking\n\n```python\ncategorical_features_benchmark = [\n    \"PULocationID\",\n    \"transaction_month\",\n    \"transaction_day\",\n    \"transaction_hour\",\n]\n# do NOT include trip_distance here, since it directly coorelates with total_amount, and that's cheating actually!\ninput_features_benchmark = categorical_features_benchmark\ntarget_feature_benchmark = \"total_amount\"\n```\n\nNote that we don't have `trip_distance` as out input features, since it directly co-relates with `total_amount`, the output.\nIt will act as a cheat for the model and overfit it.\n\nWe one-hot encode the input data points.\nThen we split the dataset into X_test, X_train, y_test and y_train, and fit a `DecisionTreeRegressor` on the cleaned dataset.\n\nThe results were as follows\n\n```shell\nmean absolute error 12.616090392101823\nmean squared error 357.68911713379254\nroot mean squared error 18.91267080911082\nr2 0.2347719420857315\n```\n\nAs we can see it actually performed quite well on the benchmark model, but not too good.\nHence we need to actually introduce some new features to the dataset.\n\n## Feature Engineering\n\nWe add new features to the dataset such as `weekday`, `weekend` and `is_holiday` to help the model draw more info from the data points.\nThis helps the model to understand the date values much more.\nFeature engineering is cruicial since it requires real world knowlegde about the domain to introduce new and meaningful features to the dataset.\n\nHere, the weekday or whether the day was a weekend or a holiday will affect out result to a high degree since people tend to go out on holidays\nor office workers won't take a morning taxi on weekdays.\nWe can use the `USFederalHolidayCalendar` from `pandas.tseries.holiday` to automatically insert the column into out dataset.\n\nWe can add more features to the dataset in a similar way.\n\n## Model Training\n\nNow we actually begin to train out model.\n\nFinal Features for training.\n\n```python\ncategorical_features = [\n    \"PULocationID\",\n    \"transaction_month\",\n    \"transaction_day\",\n    \"transaction_hour\",\n    \"transaction_week_day\",\n    \"weekend\",\n    \"is_holiday\",\n    \"Borough\",\n]\ninput_features = categorical_features\ntarget_feature = \"total_amount\"\n```\n\nFinal training results (models used, mean absolute error, root mean square error and r2 scores) are tabulated below\"\n\n| Algorithm              |   MAE |  RMSE |   R2 |\n| :--------------------- | ----: | ----: | ---: |\n| Benchmark Model        | 12.61 | 18.90 | 0.23 |\n| Decision Tree          | 10.78 | 17.64 | 0.36 |\n| Random Forest          |  9.82 | 17.04 | 0.40 |\n| Hist Gradient Boosting |  9.62 | 15.95 | 0.47 |\n\nWe can iterate over on any of the steps again to improve the model, but this is a good checkpoint.\nSo we save the currenly best performing model to use it in other applications.\n\n```python\nwith open(\"taxi_regression_model.pkl\", \"wb\") as file:\n    pickle.dump(model_at_hand, file)\n```\n\nFurther steps include fine tuning the model by finding the best hyperparameters, which I may do at some point.\n\n## A Classification Problem\n\nThis problem can be turned into a whole new classification problem.\n\n### Problem Definition\n\nClassify the earnings of taxi drivers for each ride for each region of New York per given day and hour.\n\nClassification into\n\n- Low (\u003c $25)\n- High (\u003e= $25)\n\nThis problem is a Supervised Classification problem.\n\n- Supervised: We have the actual values that we are trying to predict (in the train dataset)\n- Classification: We are trying to predict a discrete value (0 or 1)\n\nWe can simply add a new feature called `earning_class` containing 0 or 1 if `total_amount` is \u003c 25 or \u003e= 25.\n\n```python\nnyc_class[\"earning_class\"] = data_with_new_features[\"total_amount\"].apply(lambda x: \"low\" if x \u003c= 25 else \"high\")\nnyc_class[\"earning_class_binary\"] = nyc_class[\"earning_class\"].apply(lambda x: 0 if x == \"low\" else 1)\n```\n\nWe used the `total_amount` histogram for finding the appropriate split point.\n\n![Total Amount Histogram](images/total_amount_dist.png)\n\n$25 results in a very even split\n\n```shell\nearning_class\nhigh    45671\nlow     34165\nName: count, dtype: int64\n```\n\nNow we train a classification model on these features, such as `RandomForestClassifier`\n\n```python\ncategorical_features = [\n    \"PULocationID\",\n    \"transaction_month\",\n    \"transaction_day\",\n    \"transaction_hour\",\n    \"transaction_week_day\",\n    \"weekend\",\n    \"is_holiday\",\n    \"Borough\",\n]\ninput_features = categorical_features\ntarget_feature = \"earning_class_binary\"\n```\n\nThis results in an overall nice score\n\n```shell\n[[ 7600  3717]\n [ 3313 11716]]\naccuracy 0.7331663250588325\nprecision 0.759152465496015\nrecall 0.7795595182646883\n```\n\nand finally we save the model for later use\n\n```python\nwith open(\"taxi_classification_model.pkl\", \"wb\") as file:\n    pickle.dump(clf, file)\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshnarayanjha%2Ftaxi_analysis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fharshnarayanjha%2Ftaxi_analysis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fharshnarayanjha%2Ftaxi_analysis/lists"}