{"id":15066773,"url":"https://github.com/sunlightpolicy/sql4housing","last_synced_at":"2025-08-18T19:31:39.106Z","repository":{"id":35061818,"uuid":"194924414","full_name":"sunlightpolicy/sql4housing","owner":"sunlightpolicy","description":"Create housing databases","archived":false,"fork":false,"pushed_at":"2022-06-21T22:31:39.000Z","size":593,"stargazers_count":6,"open_issues_count":2,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-12-14T00:32:54.073Z","etag":null,"topics":["census-data","cities","civic-tech","databases","housing","housing-advocates","housing-data","hud","postgis","socrata-open-data-api"],"latest_commit_sha":null,"homepage":"https://sunlightfoundation.com/2019/07/22/hacking-for-housing-how-open-data-and-civic-hacking-creates-wins-for-housing-advocates/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sunlightpolicy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"license.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-07-02T19:38:53.000Z","updated_at":"2022-03-15T04:19:44.000Z","dependencies_parsed_at":"2022-09-19T10:02:03.693Z","dependency_job_id":null,"html_url":"https://github.com/sunlightpolicy/sql4housing","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunlightpolicy%2Fsql4housing","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunlightpolicy%2Fsql4housing/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunlightpolicy%2Fsql4housing/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunlightpolicy%2Fsql4housing/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sunlightpolicy","download_url":"https://codeload.github.com/sunlightpolicy/sql4housing/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230268748,"owners_count":18199806,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["census-data","cities","civic-tech","databases","housing","housing-advocates","housing-data","hud","postgis","socrata-open-data-api"],"created_at":"2024-09-25T01:12:01.531Z","updated_at":"2024-12-18T12:13:18.467Z","avatar_url":"https://github.com/sunlightpolicy.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# sql4housing\n\n## Background\n\nSql4housing is based on a broader effort to encourage collaboration between civic hackers and housing advocates. Read more about this work on our blog here:\n\n[Hacking for Housing: How open data and civic hacking creates wins for housing advocates](https://sunlightfoundation.com/2019/07/22/hacking-for-housing-how-open-data-and-civic-hacking-creates-wins-for-housing-advocates/) \u003cbr\u003e\n\n[Ownership, evictions, and violations: an overview of housing data use cases\n](https://sunlightfoundation.com/2019/08/20/ownership-evictions-and-violations-an-overview-of-housing-data-use-cases/)\n\n\n## Introduction\n\nSql4housing is based on a cloned copy of Dallas Morning News' [socrata2sql](https://github.com/DallasMorningNews/socrata2sql). Socrata2sql is a tool which allows you to import any dataset on the Socrata API and copy it into a SQL database of your choice using a command line interface. Here, I aim to adapt socrata2sql to be able to import datasets from the following sources:\n\n-[HUD's Open Data Portal](https://hudgis-hud.opendata.arcgis.com/)\u003cbr\u003e\n-Any locally saved Excel file or Excel download hyperlink\u003cbr\u003e\n-Any locally saved .csv file or .csv download hyperlink\u003cbr\u003e\n-Any locally saved .shp file or .zip download hyperlink containing a .shp file\u003cbr\u003e\n-Any locally saved .geojson file or .geojson download hyperlink\u003cbr\u003e\n-Any dataset on a Socrata open data portal\u003cbr\u003e\n-Census variables within the 5-year American Community Survey or Decennial Census\n\n## Requirements\n\n- Python 3.x\u003cbr\u003e\n- Any database supported by SQLAlchemy\u003cbr\u003e\n- Download package via: `pip install sql4housing`\n\n## Usage\n\nChanges in usage will be periodically updated and documented within the docstring of [cli.py](https://github.com/sunlightpolicy/sql4housing/blob/master/sql4housing/cli.py)\n\nSee [/chicago_examples](https://github.com/sunlightpolicy/sql4housing/tree/master/chicago_example) for a detailed use case.\n\n    \"\"\"Housing data to SQL database loader\n\n    Load a dataset directly from an API (Socrata, HUD) or file (csv or shp)\n    into a SQL database. The loader supports any database supported by SQLalchemy.\n    This file is adapted from a forked copy of DallasMorningNews/socrata2sql\n\n    Usage:\n      sql4housing bulk_load\n      sql4housing hud \u003csite\u003e [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing socrata \u003csite\u003e \u003cdataset_id\u003e [--a=\u003capp_token\u003e] [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing csv \u003clocation\u003e [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing excel \u003clocation\u003e [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing shp \u003clocation\u003e [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing geojson \u003clocation\u003e [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing census (decennial2010 | (acs [--y=\u003cyear\u003e])) \u003cvariables\u003e (--m=\u003cmsa\u003e | --c=\u003ccsa\u003e | --n=\u003ccounty\u003e | --s=\u003cstate\u003e | --p=\u003cplace\u003e) [--l=\u003clevel\u003e] [--d=\u003cdatabase_url\u003e] [--t=\u003ctable_name\u003e]\n      sql4housing (-h | --help)\n      sql4housing (-v | --version)\n\n    Options:\n      \u003cbulk_load\u003e        Loads all datasets documented within a file entitled bulk_load.yaml.\n                         Must be run in the same folder where bulk_load.yaml is saved.\n      \u003csite\u003e             The domain for the open data site. For Socrata, this is the\n                         URL to the open data portal (Ex: www.dallasopendata.com).\n                         For HUD, this is the Query URL as created in the API\n                         Explorer portion of each dataset's page on the site\n                         https://hudgis-hud.opendata.arcgis.com. See example use cases\n                         for detailed instructions.\n      \u003cdataset_id\u003e       The ID of the dataset on Socrata's open data site. This is\n                         usually a few characters, separated by a hyphen, at the end\n                         of the URL. Ex: 64pp-jeba\n      \u003clocation\u003e         Either the path or download URL where the file can be accessed.\n      --d=\u003cdatabase_url\u003e Database connection string for destination database as\n                         diacdlect+driver://username:password@host:port/database.\n                         Default: \"postgresql:///mydb\"\n      --t=\u003ctable_name\u003e   Destination table in the database. Defaults to a sanitized\n                         version of the dataset or file's name.\n      --a=\u003capp_token\u003e    App token for the Socrata site. Only necessary for\n                         high-volume requests. Default: None\n      --y=\u003cyear\u003e         Optional year specification for the 5-year American Community\n                         survey. Defaults to 2017.\n      --m=\u003cmsa\u003e          The metropolitan statistical area to include. \n                         Ex: --m=\"new york-newark-jersey city\"\n      --c=\u003ccsa\u003e          The combined statistical area to include.\n                         Ex: --c=\"New York-Newark, NY-NJ-CT-PA\"\n      --n=\u003ccounty\u003e       The county to include.\n                         Ex: --n=\"cook county, IL\"\n      --s=\u003cstate\u003e        The state to include.\n                         Ex: --s=\"illinois\"\n      --p=\u003cplace\u003e        The census place to include.\n                         Ex: --p=\"chicago, IL\"\n      --l=\u003clevel\u003e        The geographic level at which to extract data. i.e. tract,\n                         block, county, region, division. Reference cenpy documentation\n                         to learn more: https://github.com/cenpy-devs/cenpy\n      -h --help          Show this screen.\n      -v --version       Show version.\n\n    Examples:\n\n      Load the Dallas check register into a local SQLite file (file name chosen\n      from the dataset name):\n      $ sql4housing socrata www.dallasopendata.com 64pp-jeba\n\n      Load it into a PostgreSQL database called mydb:\n      $ sql4housing socrata www.dallasopendata.com 64pp-jeba -d\"postgresql:///mydb\"\n\n      Load Public Housing Buildings from HUD into a PostgreSQL database called mydb:\n      $ sql4housing hud \"https://services.arcgis.com/VTyQ9soqVukalItT/arcgis/rest/services/Public_Housing_Buildings/FeatureServer/0/query?outFields=*\u0026where=1%3D1\" -d=postgresql:///mydb\n\n      Load Public Housing Physical Inspection scores into a PostgreSQL database called housingdb:\n      $ sql4housing excel \"http://www.huduser.org/portal/datasets/pis/public_housing_physical_inspection_scores.xlsx\" -d=postgresql:///housingdb\n    \"\"\"\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunlightpolicy%2Fsql4housing","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsunlightpolicy%2Fsql4housing","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunlightpolicy%2Fsql4housing/lists"}