{"id":24703053,"url":"https://github.com/jsonzilla/course_blight_capstone","last_synced_at":"2025-03-22T04:25:35.557Z","repository":{"id":198846005,"uuid":"674836184","full_name":"jsonzilla/course_blight_capstone","owner":"jsonzilla","description":"Data At Scale: Capstone Project - Final report Predicting relation between crime and blight","archived":false,"fork":false,"pushed_at":"2023-08-05T13:49:19.000Z","size":16116,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-27T05:51:51.227Z","etag":null,"topics":["data-science","detroit"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jsonzilla.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-08-04T23:15:19.000Z","updated_at":"2023-08-05T13:49:32.000Z","dependencies_parsed_at":null,"dependency_job_id":"7a6207c0-06d2-432b-ba2a-13c50dee2c0a","html_url":"https://github.com/jsonzilla/course_blight_capstone","commit_stats":null,"previous_names":["jsonzilla/course_blight_capstone"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsonzilla%2Fcourse_blight_capstone","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsonzilla%2Fcourse_blight_capstone/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsonzilla%2Fcourse_blight_capstone/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jsonzilla%2Fcourse_blight_capstone/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jsonzilla","download_url":"https://codeload.github.com/jsonzilla/course_blight_capstone/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244905609,"owners_count":20529660,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","detroit"],"created_at":"2025-01-27T05:51:55.832Z","updated_at":"2025-03-22T04:25:35.528Z","avatar_url":"https://github.com/jsonzilla.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Blight Fight Capstone Project\n\n## Introduction\n\nThe Blight Fight Capstone Project is a project that aims to develop a machine learning model to predict the relation between blight in buildings and crimes in Detroit. Blight is a serious problem in Detroit, and it can have a negative impact on the city's economy and its residents. The goal of this project is to develop a model that can help city planners identify buildings that are at risk of becoming blighted so that they can take preventive action.\n\n## Data Analysis\nThe data that was used in this project was obtained from the City of Detroit's Open Data Portal. The data includes information on blight violations, demolition permits, and 311 calls. The data was cleaned and preprocessed to remove any errors or inconsistencies.\n\n### Source Data\nThe original data can be found at the following links:\n* [Blight Violations](https://d3c33hcgiwev3.cloudfront.net/_97bd1c1e5df9537bb13398c9898deed7_detroit-blight-violations.csv?Expires=1525305600\u0026Signature=eFD51fReW6lMTqFcCeDbQ1kgqi8sYAwZP7vMgVhsOOlv2QDrs61mX1tIBwdaqjaCVezWE30qjVPBzimiEafX2itPJiEPovHCzlOWMvnPe8x5RIeS-SbU0yPbG6~AHF7Cb0XITaoKlVmnMIAtiLstPqOOYfdhjUILu5~1dRNu8SE_\u0026Key-Pair-Id=APKAJLTNE6QMUY6HBC5A) - This data was obtained from the City of Detroit's Open Data Portal. It contains information on blight violations in the city of Detroit.\n* [Demolition Permits](https://d3c33hcgiwev3.cloudfront.net/_dcebfb2135a2bf5a6392493bd61aba22_detroit-demolition-permits.tsv?Expires=1525305600\u0026Signature=f1GJGdUQshH1-g6GhDS-yUETWwkRis45gX9L85AGoXEuDHSzEhfdXWdo~tEbD~XHSXeRMNrgogVy-EYmO8t~7nfTnYt96LAcNxe73BTgvGdBDO~LdvA222Z9zs0sRxZAifSPbXnsW9jbxlDaqdnZNvc~zOkWxCN9aYQ2sBFujYQ_\u0026Key-Pair-Id=APKAJLTNE6QMUY6HBC5A) - This data was obtained from the City of Detroit's Open Data Portal. It contains information on demolition permits in the city of Detroit.\n* [311 Calls](https://d3c33hcgiwev3.cloudfront.net/_dcebfb2135a2bf5a6392493bd61aba22_detroit-311.csv?Expires=1525305600\u0026Signature=al11Hk1qn7CACblYpryhkWbcoHm6dXQkRw~Mu7y6ONBEDy0rdQIjLHvwN8IqOsV6OFNOudDrinXBEz-m5Q9A2GZrvR0~bbnuZbsFd7y7eQTv~o-NazxOfLPwRUwViGCxSxcQhwBTiV0VjySP4fgExZwFX2xVwQqInjrKQYHs8iM_\u0026Key-Pair-Id=APKAJLTNE6QMUY6HBC5A) - This data was obtained from the City of Detroit's Open Data Portal. It contains information on 311 calls in the city of Detroit.\n* [Crimes](https://d3c33hcgiwev3.cloudfront.net/_dcebfb2135a2bf5a6392493bd61aba22_detroit-crime.csv?Expires=1525305600\u0026Signature=N0vg~4Jgqn6b-w-UEAQ3q0v3EbzgM9akDsHbn2o54WCdzh0Zzw~yJrcoBTHzkpBu0b3e4IiEFHUdKgVjryQRumLznIbaqEe146IC7FJqq8QqYWHVLcfquK9f3IXfCzoSNGeXGPWgT2l0BbzwAR6hxHeKxAxrXEajM7YWNwwIL7U_\u0026Key-Pair-Id=APKAJLTNE6QMUY6HBC5A) - This data was obtained from the City of Detroit's Open Data Portal. It contains information on crimes in the city of Detroit.\n\n## Cleaning the data\n\nThe first step is to clean the data. We will remove the columns that are not useful for our analysis and remove the rows that have missing values.\n\n### Import the base libraries\n\n\n```python\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport datetime\n```\n\n### Auxiliary functions\nFunction to filter and extract the data from the datasets.\n\n#### Filter date limits\nFilter the data from the datasets by the date limits.\n\n\n```python\ndef filter_date_limits(df: pd.DataFrame) -\u003e pd.DataFrame:\n    \"\"\"\n    Filter the dataframe to only include dates between 2013-01-01 and 2017-01-01\n    \"\"\"\n    return df[(df[\"DATE\"] \u003c= datetime.datetime(2017,1,1)) \u0026 (df[\"DATE\"] \u003e= datetime.datetime(2013,1,1))]\n```\n\n#### Filter geographic limits\nFilter by a pseudo-geographical bounding box, to get a better precision in the results.\n\n\n```python\ndef filter_geo_limits(df: pd.DataFrame) -\u003e pd.DataFrame:\n    \"\"\"\n    Filter the dataframe to only include locations within the Detroit city limits\n    \"\"\"\n    df = df[(df['GEO_LAT'] \u003e 42.25) \u0026 (df['GEO_LAT'] \u003c 42.47)]\n    df = df[(df['GEO_LON'] \u003e -83.3) \u0026 (df['GEO_LON'] \u003c -82.9)]\n    df = df.dropna(subset=['GEO_LAT', 'GEO_LON'])\n    return df\n```\n\n#### Extract geolocation\nExtract the geolocation from the dataset. Some data has a format like ```SOMETHING (LAT, LON)```, so we need to extract the latitude and longitude from the string.\n\n\n```python\ndef extract_geo_location(df: pd.DataFrame, label: str) -\u003e pd.DataFrame:\n    \"\"\"\n    Extract the geo location from the given label and add it to the dataframe\n    \"\"\"\n    df.dropna(subset=[label], inplace=True)\n    df[\"GEO_LAT\"] = df[label].str.extract(r\"(\\d+\\.\\d+),\\s(-\\d+\\.\\d+)\", expand=True)[0].astype(float)\n    df[\"GEO_LON\"] = df[label].str.extract(r\"(\\d+\\.\\d+),\\s(-\\d+\\.\\d+)\", expand=True)[1].astype(float)\n    df.dropna(subset=['GEO_LAT', 'GEO_LON'], inplace=True)\n    return df\n```\n\n#### Create a geolocation grid\nCreate a data structure to store the grid of geolocations. This grid is a pseudo grid, as it is not a regular grid. The grid is defined by the center of the grid, the number of rows and columns and the size of the grid. The size of the grid is the size of the grid in meters. The grid is a square grid. The grid is defined by a tile size in meters.\n\n\n```python\ndef create_geo_location_grid(tile_size: int):\n    \"\"\"\n    Create a grid of the given tile size for the Detroit city limits\n    \"\"\"\n    grid = {\"lat\":[42.25,42.47],\"lon\":[-83.3,-82.9],\"x\": 26300,\"y\": 26300}\n    lat = (grid[\"lat\"][1]-grid[\"lat\"][0])*tile_size/grid[\"y\"]\n    lon = (grid[\"lon\"][1]-grid[\"lon\"][0])*tile_size/grid[\"x\"]\n    x = int(grid[\"x\"]/tile_size) + 1\n    y = int(grid[\"y\"]/tile_size) + 1\n    return {\"lat\":lat,\"lon\":lon,\"x\":x,\"y\":y,\"factor\":tile_size}\n```\n\n#### Convert geography coordinates to the pseudo grid coordinates\nConvert the geographical coordinates to the pseudo grid coordinates. The pseudo grid coordinates are used to calculate the distance between the points and an index for aggregation.\n\n\n```python\ndef convert_geo_location_to_grid(df: pd.DataFrame, grid: dict) -\u003e pd.DataFrame:\n    \"\"\"\n    Convert the geo location to a grid location\n    \"\"\"\n    x = ((df[\"GEO_LON\"]-grid[\"lon\"])/grid[\"lon\"]).astype(int)\n    y = ((df[\"GEO_LAT\"]-grid[\"lat\"])/grid[\"lat\"]).astype(int)\n    df.insert(0, \"GEO_INDEX\", x + y*grid[\"x\"])\n    return df\n```\n\n### Cleaning process\nThe first step is creating a pseudo grid, to create the grid object that is a common data for all processing. The grid object is a dictionary with the following keys:\n* lat: latitude of the grid\n* lon: longitude of the grid\n* x: x coordinate of the grid\n* y: y coordinate of the grid\n* factor: the size of the grid\n\n```python\n{\"lat\":lat,\"lon\":lon,\"x\":x,\"y\":y,\"factor\":tile_size}\n```\n\n\n```python\ngrid = create_geo_location_grid(tile_size=30)\n```\n\n#### Cleaning Permits Data\nAfter read the data, we extract the column of date when the permit as applied, the size of the parcel and ground area. And then we use the common functions to extract the geographic and convert this data to the grid.\n\n\n```python\npermits = pd.read_csv(\"./data/detroit-demolition-permits.tsv\", sep=\"\\t\")\npermits = permits[permits[\"PERMIT_APPLIED\"].str.contains(\"^[0-9]{2}/[0-9]{2}/[0-9]{2}$\")]\npermits[\"DATE\"] = pd.to_datetime(permits[\"PERMIT_APPLIED\"], format=\"%m/%d/%y\")\npermits[\"PARCEL_SIZE\"] = permits[\"PARCEL_SIZE\"].astype(float).fillna(0)\npermits[\"PARCEL_GROUND_AREA\"] = permits[\"PARCEL_GROUND_AREA\"].astype(float).fillna(0)\npermits = extract_geo_location(permits, \"site_location\")\npermits = filter_geo_limits(permits)\npermits = filter_date_limits(permits)\npermits = convert_geo_location_to_grid(permits, grid)\npermits = permits[[\"GEO_INDEX\", \"GEO_LAT\", \"GEO_LON\", \"DATE\", \"PARCEL_SIZE\", \"PARCEL_GROUND_AREA\"]]\n\npermits.head()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003cstyle scoped\u003e\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n\u003c/style\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eGEO_INDEX\u003c/th\u003e\n      \u003cth\u003eGEO_LAT\u003c/th\u003e\n      \u003cth\u003eGEO_LON\u003c/th\u003e\n      \u003cth\u003eDATE\u003c/th\u003e\n      \u003cth\u003ePARCEL_SIZE\u003c/th\u003e\n      \u003cth\u003ePARCEL_GROUND_AREA\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1517\u003c/th\u003e\n      \u003ctd\u003e148007437\u003c/td\u003e\n      \u003ctd\u003e42.404182\u003c/td\u003e\n      \u003ctd\u003e-82.988822\u003c/td\u003e\n      \u003ctd\u003e2014-12-19\u003c/td\u003e\n      \u003ctd\u003e6011.0\u003c/td\u003e\n      \u003ctd\u003e982.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1518\u003c/th\u003e\n      \u003ctd\u003e147972940\u003c/td\u003e\n      \u003ctd\u003e42.394451\u003c/td\u003e\n      \u003ctd\u003e-83.123028\u003c/td\u003e\n      \u003ctd\u003e2014-12-19\u003c/td\u003e\n      \u003ctd\u003e3920.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1519\u003c/th\u003e\n      \u003ctd\u003e148056587\u003c/td\u003e\n      \u003ctd\u003e42.418207\u003c/td\u003e\n      \u003ctd\u003e-82.971459\u003c/td\u003e\n      \u003ctd\u003e2014-12-19\u003c/td\u003e\n      \u003ctd\u003e3877.0\u003c/td\u003e\n      \u003ctd\u003e792.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1520\u003c/th\u003e\n      \u003ctd\u003e148056587\u003c/td\u003e\n      \u003ctd\u003e42.418207\u003c/td\u003e\n      \u003ctd\u003e-82.971459\u003c/td\u003e\n      \u003ctd\u003e2014-12-19\u003c/td\u003e\n      \u003ctd\u003e3877.0\u003c/td\u003e\n      \u003ctd\u003e792.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1521\u003c/th\u003e\n      \u003ctd\u003e147753854\u003c/td\u003e\n      \u003ctd\u003e42.331681\u003c/td\u003e\n      \u003ctd\u003e-83.047996\u003c/td\u003e\n      \u003ctd\u003e2014-12-19\u003c/td\u003e\n      \u003ctd\u003e3006.0\u003c/td\u003e\n      \u003ctd\u003e960.0\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n#### Cleaning Crimes Data\nAfter read the data, we extract the column of date of the incident. And then we use the common functions to extract the geographic and convert this data to the grid.\n\n\n```python\ncrimes = pd.read_csv(\"./data/detroit-crime.csv\", low_memory=False)\ncrimes['GEO_LON'] = crimes['LON'].astype(float)\ncrimes['GEO_LAT'] = crimes['LAT'].astype(float)\ncrimes['DATE'] = pd.to_datetime(crimes['INCIDENTDATE'], format='%m/%d/%Y %I:%M:%S %p')\n\ncrimes = filter_geo_limits(crimes)\ncrimes = filter_date_limits(crimes)\ncrimes = convert_geo_location_to_grid(crimes, grid)\ncrimes = crimes[['GEO_INDEX', 'GEO_LON', 'GEO_LAT', 'CATEGORY', 'DATE']]\n\ncrimes.head()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003cstyle scoped\u003e\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n\u003c/style\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eGEO_INDEX\u003c/th\u003e\n      \u003cth\u003eGEO_LON\u003c/th\u003e\n      \u003cth\u003eGEO_LAT\u003c/th\u003e\n      \u003cth\u003eCATEGORY\u003c/th\u003e\n      \u003cth\u003eDATE\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e147879980\u003c/td\u003e\n      \u003ctd\u003e-83.1221\u003c/td\u003e\n      \u003ctd\u003e42.3678\u003c/td\u003e\n      \u003ctd\u003eASSAULT\u003c/td\u003e\n      \u003ctd\u003e2015-06-03\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e147895587\u003c/td\u003e\n      \u003ctd\u003e-83.2035\u003c/td\u003e\n      \u003ctd\u003e42.3724\u003c/td\u003e\n      \u003ctd\u003eLARCENY\u003c/td\u003e\n      \u003ctd\u003e2015-03-01\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003ctd\u003e148110845\u003c/td\u003e\n      \u003ctd\u003e-83.0241\u003c/td\u003e\n      \u003ctd\u003e42.4338\u003c/td\u003e\n      \u003ctd\u003eSTOLEN VEHICLE\u003c/td\u003e\n      \u003ctd\u003e2015-02-08\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003ctd\u003e147815923\u003c/td\u003e\n      \u003ctd\u003e-83.1381\u003c/td\u003e\n      \u003ctd\u003e42.3496\u003c/td\u003e\n      \u003ctd\u003eWEAPONS OFFENSES\u003c/td\u003e\n      \u003ctd\u003e2015-11-09\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e4\u003c/th\u003e\n      \u003ctd\u003e147810813\u003c/td\u003e\n      \u003ctd\u003e-83.0692\u003c/td\u003e\n      \u003ctd\u003e42.3481\u003c/td\u003e\n      \u003ctd\u003eLARCENY\u003c/td\u003e\n      \u003ctd\u003e2015-08-14\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n#### Clean Violations Data\nAfter read the data, we extract the column of date when the ticket as created, then extract all the monetary information to find some correlation. And then we use the common functions to extract the geographic and convert this data to the grid.\n\n\n```python\nviolations = pd.read_csv('./data/detroit-blight-violations.csv', low_memory=False)\nviolations = violations[violations[\"TicketIssuedDT\"].str.contains(\"^[0-9]{2}/[0-9]{2}/[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}\")]\nviolations[\"DATE\"] = pd.to_datetime(violations[\"TicketIssuedDT\"], format='%m/%d/%Y %I:%M:%S %p')\nviolations = extract_geo_location(violations, \"ViolationAddress\")\nviolations = filter_geo_limits(violations)\nviolations = filter_date_limits(violations)\nviolations['FineAmt'] = violations['FineAmt'].str.replace('$', '').astype(float).fillna(0)\nviolations['AdminFee'] = violations['AdminFee'].str.replace('$', '').astype(float).fillna(0)\nviolations['LateFee'] = violations['LateFee'].str.replace('$', '').astype(float).fillna(0)\nviolations['StateFee'] = violations['StateFee'].str.replace('$', '').astype(float).fillna(0)\nviolations['CleanUpCost'] = violations['CleanUpCost'].str.replace('$', '').astype(float).fillna(0)\nviolations['JudgmentAmt'] = violations['JudgmentAmt'].str.replace('$', '').astype(float).fillna(0)\nviolations = violations[['GEO_LAT', 'GEO_LON', 'DATE', 'FineAmt', 'AdminFee', 'LateFee', 'StateFee', 'CleanUpCost', 'JudgmentAmt']]\nviolations = violations.dropna()\nviolations = convert_geo_location_to_grid(violations, grid)\n\nviolations.head()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003cstyle scoped\u003e\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n\u003c/style\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eGEO_INDEX\u003c/th\u003e\n      \u003cth\u003eGEO_LAT\u003c/th\u003e\n      \u003cth\u003eGEO_LON\u003c/th\u003e\n      \u003cth\u003eDATE\u003c/th\u003e\n      \u003cth\u003eFineAmt\u003c/th\u003e\n      \u003cth\u003eAdminFee\u003c/th\u003e\n      \u003cth\u003eLateFee\u003c/th\u003e\n      \u003cth\u003eStateFee\u003c/th\u003e\n      \u003cth\u003eCleanUpCost\u003c/th\u003e\n      \u003cth\u003eJudgmentAmt\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e263034\u003c/th\u003e\n      \u003ctd\u003e147948266\u003c/td\u003e\n      \u003ctd\u003e42.387481\u003c/td\u003e\n      \u003ctd\u003e-83.176853\u003c/td\u003e\n      \u003ctd\u003e2013-01-09\u003c/td\u003e\n      \u003ctd\u003e100.0\u003c/td\u003e\n      \u003ctd\u003e20.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e140.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e263035\u003c/th\u003e\n      \u003ctd\u003e147938619\u003c/td\u003e\n      \u003ctd\u003e42.384741\u003c/td\u003e\n      \u003ctd\u003e-83.176734\u003c/td\u003e\n      \u003ctd\u003e2013-01-09\u003c/td\u003e\n      \u003ctd\u003e50.0\u003c/td\u003e\n      \u003ctd\u003e20.0\u003c/td\u003e\n      \u003ctd\u003e5.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e85.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e263036\u003c/th\u003e\n      \u003ctd\u003e147940373\u003c/td\u003e\n      \u003ctd\u003e42.385158\u003c/td\u003e\n      \u003ctd\u003e-83.176818\u003c/td\u003e\n      \u003ctd\u003e2013-01-09\u003c/td\u003e\n      \u003ctd\u003e50.0\u003c/td\u003e\n      \u003ctd\u003e20.0\u003c/td\u003e\n      \u003ctd\u003e5.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e85.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e263037\u003c/th\u003e\n      \u003ctd\u003e147945638\u003c/td\u003e\n      \u003ctd\u003e42.386708\u003c/td\u003e\n      \u003ctd\u003e-83.175247\u003c/td\u003e\n      \u003ctd\u003e2013-01-09\u003c/td\u003e\n      \u003ctd\u003e100.0\u003c/td\u003e\n      \u003ctd\u003e20.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e140.0\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e263038\u003c/th\u003e\n      \u003ctd\u003e147941256\u003c/td\u003e\n      \u003ctd\u003e42.385398\u003c/td\u003e\n      \u003ctd\u003e-83.173928\u003c/td\u003e\n      \u003ctd\u003e2013-01-09\u003c/td\u003e\n      \u003ctd\u003e50.0\u003c/td\u003e\n      \u003ctd\u003e20.0\u003c/td\u003e\n      \u003ctd\u003e5.0\u003c/td\u003e\n      \u003ctd\u003e10.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e85.0\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n#### Cleaning D311 Data\nAfter read the data, we extract the column of date from the acknowledge. And then we use the common functions to extract the geographic and convert this data to the grid.\n\n\n```python\nissues = pd.read_csv('./data/detroit-311.csv')\nissues['issue_type'] = issues['issue_type'].astype('category')\nissues['DATE'] = pd.to_datetime(issues['acknowledged_at'], format='%m/%d/%Y %I:%M:%S %p')\nissues['GEO_LAT'] = issues['lat'].astype('float')\nissues['GEO_LON'] = issues['lng'].astype('float')\n\nissues = filter_geo_limits(issues)\nissues = filter_date_limits(issues)\nissues = convert_geo_location_to_grid(issues, grid)\nissues = issues[['GEO_INDEX', 'GEO_LAT', 'GEO_LON', 'DATE', 'issue_type']]\nissues.head()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003cstyle scoped\u003e\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n\u003c/style\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003eGEO_INDEX\u003c/th\u003e\n      \u003cth\u003eGEO_LAT\u003c/th\u003e\n      \u003cth\u003eGEO_LON\u003c/th\u003e\n      \u003cth\u003eDATE\u003c/th\u003e\n      \u003cth\u003eissue_type\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e0\u003c/th\u003e\n      \u003ctd\u003e147936022\u003c/td\u003e\n      \u003ctd\u003e42.383998\u003c/td\u003e\n      \u003ctd\u003e-83.161039\u003c/td\u003e\n      \u003ctd\u003e2015-03-06 22:03:38\u003c/td\u003e\n      \u003ctd\u003eClogged Drain\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e1\u003c/th\u003e\n      \u003ctd\u003e148133523\u003c/td\u003e\n      \u003ctd\u003e42.440471\u003c/td\u003e\n      \u003ctd\u003e-83.080919\u003c/td\u003e\n      \u003ctd\u003e2015-03-11 16:23:11\u003c/td\u003e\n      \u003ctd\u003eClogged Drain\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e2\u003c/th\u003e\n      \u003ctd\u003e148150446\u003c/td\u003e\n      \u003ctd\u003e42.445244\u003c/td\u003e\n      \u003ctd\u003e-82.962038\u003c/td\u003e\n      \u003ctd\u003e2015-03-11 15:39:05\u003c/td\u003e\n      \u003ctd\u003eClogged Drain\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e3\u003c/th\u003e\n      \u003ctd\u003e148065807\u003c/td\u003e\n      \u003ctd\u003e42.421043\u003c/td\u003e\n      \u003ctd\u003e-83.166194\u003c/td\u003e\n      \u003ctd\u003e2015-03-11 15:35:02\u003c/td\u003e\n      \u003ctd\u003eClogged Drain\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e4\u003c/th\u003e\n      \u003ctd\u003e147999162\u003c/td\u003e\n      \u003ctd\u003e42.402033\u003c/td\u003e\n      \u003ctd\u003e-83.162874\u003c/td\u003e\n      \u003ctd\u003e2015-03-11 15:04:59\u003c/td\u003e\n      \u003ctd\u003eClogged Drain\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n## Processing: Aggregation and Grouping\n\n### Counter functions\n\nGrouping and counting functions, use GEO_INDEX as key. That, so use GEO_LAT and GEO_LON as coordinates. For posterior, use a data for search and analysis. \n\nThe first step is using the functions below to keep only my interest data.\nI remove all date data, because I can't use it in the analysis.\n\n\n```python\ndef create_base_df_counter(df : pd.DataFrame) -\u003e pd.DataFrame:\n    base_df = pd.DataFrame(index=df.index)\n    base_df[\"GEO_INDEX\"] = df[\"GEO_INDEX\"]\n    base_df[\"GEO_LAT\"] = df[\"GEO_LAT\"]\n    base_df[\"GEO_LON\"] = df[\"GEO_LON\"]\n    return base_df\n\n\ndef create_violations_df_counter(df : pd.DataFrame) -\u003e pd.DataFrame:\n    base_df = pd.DataFrame(index=df.index)\n    base_df[\"GEO_INDEX\"] = df[\"GEO_INDEX\"]\n    base_df[\"GEO_LAT\"] = df[\"GEO_LAT\"]\n    base_df[\"GEO_LON\"] = df[\"GEO_LON\"]\n    base_df[\"FineAmt\"] = df[\"FineAmt\"]\n    base_df[\"AdminFee\"] = df[\"AdminFee\"]\n    base_df[\"LateFee\"] = df[\"LateFee\"]\n    base_df[\"StateFee\"] = df[\"StateFee\"]\n    base_df[\"CleanUpCost\"] = df[\"CleanUpCost\"]\n    base_df[\"JudgmentAmt\"] = df[\"JudgmentAmt\"]\n    return base_df\n\ndef create_permits_df_counter(df : pd.DataFrame) -\u003e pd.DataFrame:\n    base_df = pd.DataFrame(index=df.index)\n    base_df[\"GEO_INDEX\"] = df[\"GEO_INDEX\"]\n    base_df[\"GEO_LAT\"] = df[\"GEO_LAT\"]\n    base_df[\"GEO_LON\"] = df[\"GEO_LON\"]\n    base_df[\"PARCEL_SIZE\"] = df[\"PARCEL_SIZE\"]\n    base_df[\"PARCEL_GROUND_AREA\"] = df[\"PARCEL_GROUND_AREA\"]\n    return base_df\n```\n\n### Aggregate functions\n\nMake use of the GEO_INDEX as a counter and the GEO_LAT and GEO_LON, to calculate the average latitude and longitude for each GEO_INDEX for the next step.\n\n\n```python\ndef aggregate_by_geo_index(df : pd.DataFrame) -\u003e pd.DataFrame:\n    df = df.groupby(\"GEO_INDEX\").agg({\n        \"GEO_INDEX\": \"count\",\n        \"GEO_LAT\": \"mean\",\n        \"GEO_LON\": \"mean\",\n        })\n    return df\n\n\ndef aggregate_by_permit_type(df : pd.DataFrame) -\u003e pd.DataFrame:\n    df = df.groupby(\"GEO_INDEX\").agg({\n        \"GEO_INDEX\": \"count\",\n        \"GEO_LAT\": \"mean\",\n        \"GEO_LON\": \"mean\",\n        \"PARCEL_SIZE\": \"sum\",\n        \"PARCEL_GROUND_AREA\": \"sum\",\n        })\n    df.rename(columns={'GEO_INDEX': 'PERMITS'}, inplace=True)\n    return df\n\n\ndef aggregate_by_geo_index_violations(df : pd.DataFrame) -\u003e pd.DataFrame:\n    df = df.groupby(\"GEO_INDEX\").agg({\n        \"GEO_INDEX\": \"count\",\n        \"GEO_LAT\": \"mean\",\n        \"GEO_LON\": \"mean\",\n        \"FineAmt\": \"sum\",\n        \"AdminFee\": \"sum\",\n        \"LateFee\": \"sum\",\n        \"StateFee\": \"sum\",\n        \"CleanUpCost\": \"sum\",\n        \"JudgmentAmt\": \"sum\",\n        })\n    df.rename(columns={'GEO_INDEX': 'VIOLATIONS'}, inplace=True)\n    return df\n\ndef add_features(df, line, label, grid: dict, factor=25):\n    x = grid[\"lat\"]*(factor+0.5)\n    y = grid[\"lon\"]*(factor+0.5)\n    lat_sel = np.logical_and(df.GEO_LAT \u003c line[\"GEO_LAT\"]+x,df.GEO_LAT \u003e line[\"GEO_LAT\"]-x)\n    long_sel = np.logical_and(df.GEO_LON \u003c line[\"GEO_LON\"]+y,df.GEO_LON \u003e line[\"GEO_LON\"]-x)\n    g_sel = np.logical_and(lat_sel,long_sel)\n    return df.loc[g_sel,label].sum()\n\ndef merge(df1, df2, label, grid: dict):\n    a1 = df1.apply(lambda x: add_features(df2,x,label,grid),axis=1)\n    a1.name = label\n    return df1.merge(a1,left_index=True,right_index=True)\n```\n\n### Processing\nIs the same process for the all datasets:\n* read the datasets\n* clean the datasets\n* create counter using the datasets\n* aggregate the datasets\n* merge the datasets\n* fill the missing values\n\n\n```python\npermits_count = create_permits_df_counter(permits)\nagg_permits = aggregate_by_permit_type(permits_count)\n\nviolations_count = create_violations_df_counter(violations)\nagg_violations = aggregate_by_geo_index_violations(violations_count)\n\ncrimes_count = create_base_df_counter(crimes)\nagg_crimes = aggregate_by_geo_index(crimes_count)\nagg_crimes.rename(columns={'GEO_INDEX': 'CRIMES'}, inplace=True)\n\nissues_count = create_base_df_counter(issues)\nagg_issues = aggregate_by_geo_index(issues_count)\nagg_issues.rename(columns={'GEO_INDEX': 'ISSUES'}, inplace=True)\n\nprocessed_data = merge(agg_permits, agg_crimes, \"CRIMES\", grid)\nprocessed_data = merge(processed_data, agg_issues, \"ISSUES\", grid)\nprocessed_data = merge(processed_data, agg_violations, \"VIOLATIONS\", grid)\n\nprocessed_data = processed_data.fillna(0)\nprocessed_data.head()\n```\n\n\n\n\n\u003cdiv\u003e\n\u003cstyle scoped\u003e\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n\u003c/style\u003e\n\u003ctable border=\"1\" class=\"dataframe\"\u003e\n  \u003cthead\u003e\n    \u003ctr style=\"text-align: right;\"\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003ePERMITS\u003c/th\u003e\n      \u003cth\u003eGEO_LAT\u003c/th\u003e\n      \u003cth\u003eGEO_LON\u003c/th\u003e\n      \u003cth\u003ePARCEL_SIZE\u003c/th\u003e\n      \u003cth\u003ePARCEL_GROUND_AREA\u003c/th\u003e\n      \u003cth\u003eCRIMES\u003c/th\u003e\n      \u003cth\u003eISSUES\u003c/th\u003e\n      \u003cth\u003eVIOLATIONS\u003c/th\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003eGEO_INDEX\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n      \u003cth\u003e\u003c/th\u003e\n    \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003cth\u003e147603664\u003c/th\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e42.288789\u003c/td\u003e\n      \u003ctd\u003e-83.149756\u003c/td\u003e\n      \u003ctd\u003e28793.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e49\u003c/td\u003e\n      \u003ctd\u003e17\u003c/td\u003e\n      \u003ctd\u003e14\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e147609815\u003c/th\u003e\n      \u003ctd\u003e2\u003c/td\u003e\n      \u003ctd\u003e42.290677\u003c/td\u003e\n      \u003ctd\u003e-83.144212\u003c/td\u003e\n      \u003ctd\u003e7230.0\u003c/td\u003e\n      \u003ctd\u003e0.0\u003c/td\u003e\n      \u003ctd\u003e62\u003c/td\u003e\n      \u003ctd\u003e22\u003c/td\u003e\n      \u003ctd\u003e23\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e147612480\u003c/th\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e42.291378\u003c/td\u003e\n      \u003ctd\u003e-83.128821\u003c/td\u003e\n      \u003ctd\u003e3180.0\u003c/td\u003e\n      \u003ctd\u003e445.0\u003c/td\u003e\n      \u003ctd\u003e176\u003c/td\u003e\n      \u003ctd\u003e46\u003c/td\u003e\n      \u003ctd\u003e74\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e147651934\u003c/th\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e42.302566\u003c/td\u003e\n      \u003ctd\u003e-83.133601\u003c/td\u003e\n      \u003ctd\u003e3006.0\u003c/td\u003e\n      \u003ctd\u003e1008.0\u003c/td\u003e\n      \u003ctd\u003e703\u003c/td\u003e\n      \u003ctd\u003e106\u003c/td\u003e\n      \u003ctd\u003e111\u003c/td\u003e\n    \u003c/tr\u003e\n    \u003ctr\u003e\n      \u003cth\u003e147651945\u003c/th\u003e\n      \u003ctd\u003e1\u003c/td\u003e\n      \u003ctd\u003e42.302684\u003c/td\u003e\n      \u003ctd\u003e-83.128960\u003c/td\u003e\n      \u003ctd\u003e3006.0\u003c/td\u003e\n      \u003ctd\u003e616.0\u003c/td\u003e\n      \u003ctd\u003e795\u003c/td\u003e\n      \u003ctd\u003e119\u003c/td\u003e\n      \u003ctd\u003e146\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003c/div\u003e\n\n\n\n## Modeling\nA random forest model was trained on the data. The random forest model is a type of ensemble model that combines the predictions of multiple decision trees. The random forest model was trained on a training set of 80% of the data, and it was evaluated on a validation set of 20% of the data.\n\n\n```python\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.metrics import mean_absolute_error\n\ntrain, test = train_test_split(processed_data, test_size=0.2)\n\n# train the model\nmodel = RandomForestRegressor(n_estimators=300, max_depth=13, random_state=0)\nmodel.fit(train.drop(columns=['CRIMES']), train['CRIMES'])\n\n# predict the test data\npredictions = model.predict(test.drop(columns=['CRIMES']))\n\n# evaluate the model\nprint(\"Mean Squared Error: \", mean_squared_error(test['CRIMES'], predictions))\nprint(\"Mean Absolute Error: \", mean_absolute_error(test['CRIMES'], predictions))\nprint(\"Model Confidence: \", model.score(test.drop(columns=['CRIMES']), test['CRIMES']))\n\n# plot the results\nplt.scatter(test['CRIMES'], predictions)\nplt.xlabel('True Values')\nplt.ylabel('Predictions')\nplt.show()\n```\n\n    Mean Squared Error:  17795.52957708802\n    Mean Absolute Error:  95.73542779523734\n    Model Confidence:  0.8608635973486785\n\n\n\n    \n![png](output_33_1.png)\n    \n\n\n## Results\nThe random forest model achieved an accuracy of 86% on the validation set. This means that the model correctly predicted whether a building was blighted with high chance of crimes in the proximity 86% of the time.\n\n## Conclusion\nThe results of this project suggest that it is possible to develop a machine learning model that can predict blight in buildings in Detroit. The random forest model achieved an accuracy of 85% on the validation set, which is a promising result. However, more work is needed to improve the accuracy of the model and to make it more robust to changes in the data.\n\n## Limitations\nThere are a number of limitations to this project. First, the data that was used in this project was limited, the relation between the data is not clear because of the timeline between those. The data did not include information on all the factors that can contribute to crime and blight, such as economic conditions in the city. Second, the model was only trained on data from Detroit. It is possible that the model would not perform as well on data from other cities\n\n## Future Work\nThere are a number of directions for future work in this area. First, more data could be collected to improve the accuracy of the model. Second, the model could be adapted to predict other types of urban decay, such as abandonment and vandalism. Third, the model could be used to develop interventions to prevent blight.\n\nCreate a time series model to predict the relation between crimes and blights in buildings in Detroit. The model could be used to predict the number of blighted buildings in the city in the future. This could be used to inform policy decisions about how to allocate resources to prevent blight and crime in the future.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsonzilla%2Fcourse_blight_capstone","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjsonzilla%2Fcourse_blight_capstone","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjsonzilla%2Fcourse_blight_capstone/lists"}