{"id":20618695,"url":"https://github.com/notjoemartinez/rowdy_datathon","last_synced_at":"2025-08-03T19:08:09.903Z","repository":{"id":204313844,"uuid":"711249657","full_name":"NotJoeMartinez/rowdy_datathon","owner":"NotJoeMartinez","description":null,"archived":false,"fork":false,"pushed_at":"2023-10-29T11:46:17.000Z","size":1831,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-07-30T10:08:03.884Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NotJoeMartinez.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-28T16:56:53.000Z","updated_at":"2023-10-29T15:34:15.000Z","dependencies_parsed_at":null,"dependency_job_id":"09427bb5-f191-4467-8df7-6ba497d129db","html_url":"https://github.com/NotJoeMartinez/rowdy_datathon","commit_stats":null,"previous_names":["notjoemartinez/rowdy_datathon"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/NotJoeMartinez/rowdy_datathon","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotJoeMartinez%2Frowdy_datathon","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotJoeMartinez%2Frowdy_datathon/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotJoeMartinez%2Frowdy_datathon/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotJoeMartinez%2Frowdy_datathon/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NotJoeMartinez","download_url":"https://codeload.github.com/NotJoeMartinez/rowdy_datathon/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NotJoeMartinez%2Frowdy_datathon/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268596956,"owners_count":24275966,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-03T02:00:12.545Z","response_time":2577,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T12:09:12.631Z","updated_at":"2025-08-03T19:08:09.766Z","avatar_url":"https://github.com/NotJoeMartinez.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n### 2.2.3  Answer 1\n\u003e The “Text Import Wizard ” would guide you through the process of identifying variables by column. Import it and determine the total number of school districts.\n\nThere were 17604 school districts between 2017-2018 according to the excel file \n\n\t\t\n### 2.2.3 Answer 2\n\u003e The previous file counts the total number of rows in the file. Does this number coincide with the number produced by Excel? \n\nThe python method of counting rows does not accurately count the rows in the dataset because it interprets newlines wrapped in double quotes as a new row of data. This adds 9 new rows of data which are counted as school districts.\n\nWe used chat gpt to write a python script that could find these data anomalies an example \nof one of these data anomalies can be found online 14041 of the `LEA Characteristics.csv` data file :\n```txt\n\",\"jswan@reach.connectionsacademy.org\",\"JANE\",\"SWAN\",\"7177048437\n\",\"jswan@reach.connectionsacademy.org\",\"JANE\",\"SWAN\",\"7177048437\n\",\"jswan@reach.connectionsacademy.org\",\"No\",\"Yes\",\"Yes\",\"www.connexus.com\",\"No\"\n```\n\n\n###  2.2.3  Answer 3\n\u003e Why would you need encoding=“cp1252” for US data?\n\ncp1252 is needed for encoding spanish characters like ñ on cell `D2797`\n\n```txt\nCA\tCALIFORNIA\t06CC340\tOpportunities for Learning ñ Duarte\nCA\tCALIFORNIA\t06CC341\tOptions for Youth ñ San Bernardino\n```\n\n\n### 2.2.3 Answer 4\t\n\u003e This number does not match the CRDB database, but both databases claim to have the list of school districts in the US in the period 2017-18. Explain why\n\nWhile we we're unable to find out exactly why this discrepancy exists we we're able \nisolate the extra entries from the larger dataset by creating two sqlite tables with \neach csv file, then running the following query. \n\n```shell\nsqlite-utils lea_vs_edge.db \"SELECT e.*\nFROM edge e\nLEFT JOIN lea l ON e.leaid = l.leaid\nWHERE l.leaid IS NULL;\" \u003e\u003e in_edge_not_lea.json\n```\nThis creates 2100 line json file containing rows not found in `LEA Characteristics.csv`. \n\n\n\n## 2.4 Intermediate Challenge\n\n### Importing Data From CSV to a Postgres DB\n\nWe used pandas and sqlalchemy to import the CRDC csv files to our\npostgresql database, the file names were used as table names.\n```python\nvalid_csvs = glob.glob(f\"GRF17/csv/**/*.csv\", recursive=True)\nfor csv in valid_csvs:\n\ttable_name = csv.split(\"/\")[-1].split(\".\")[0]\n\ttable_name = table_name.replace(\" \", \"_\").replace(\"-\", \"_\")\n\tcsv_to_sql(csv, table_name, engine, encoding=\"cp1252\")\n```\n\n\n### Correlation between poverty and total enrollment \nOnly about 5513 of the 17614 LEA datapoints had usable poverty statistics\nfrom the Small Area Income and Poverty Estimates dataset. Plotting the \nzip codes out on a map of the US it revealed that several states were missing\nfrom the dataset as well. \n\nThe following scatter plot shows the number of 5-17 year olds in a school District \nEstimated to be in poverty over the total enrollment numbers for that district.\nThe graph seems to scale proportionally to the over all enrollment size.\n\n![d88d1eb20b96e8759093b32f14d2ca8e.png](imgs/c1170707d8cc4002894a469b9309983a.png)\n\n\n\n\n### Census income report correlation with highschool accademic performance \n\u003e school districts to census tracts and determine whether the income reported in the Opportunity Atlas has correlation with academic performance in high school\n\nScatter plot showing the relationship between total number of students in Bexar count who passed  algebra 1 and their predicted income at the age of  35 by Opportunity Atlas.\n\n![b2b99373d4daa8d65f1bab999d6ce5d8.png](imgs/011c964fc40a48259e7c8446bb82ccba.png)\n\nUsing algebra 1 as a metric for measuring success of a person when they \nare 35 is inherently, flawed however we used it because it was the only \ndatapoint that contained a definitive \"pass/fail\" score. Ideally we would \nhave preferred using ACT/SAT scores.  \n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotjoemartinez%2Frowdy_datathon","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnotjoemartinez%2Frowdy_datathon","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnotjoemartinez%2Frowdy_datathon/lists"}