{"id":13557705,"url":"https://github.com/GreenInfo-Network/nyc-crash-mapper-etl-script","last_synced_at":"2025-04-03T12:31:03.800Z","repository":{"id":74814513,"uuid":"81900617","full_name":"GreenInfo-Network/nyc-crash-mapper-etl-script","owner":"GreenInfo-Network","description":"Extract, Transform, and Load script for fetching new data from the NYC Open Data Portal's vehicle collision data and loading into the NYC Crash Mapper table on CARTO.","archived":false,"fork":false,"pushed_at":"2024-12-19T22:19:41.000Z","size":4516,"stargazers_count":3,"open_issues_count":3,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-19T23:22:22.242Z","etag":null,"topics":["carto","carto-api","extract-transform-load","heroku-scheduler","nyc","open-data","python","socrata-api"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/GreenInfo-Network.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-02-14T03:29:45.000Z","updated_at":"2024-12-19T22:19:45.000Z","dependencies_parsed_at":"2024-01-14T17:55:56.391Z","dependency_job_id":null,"html_url":"https://github.com/GreenInfo-Network/nyc-crash-mapper-etl-script","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GreenInfo-Network%2Fnyc-crash-mapper-etl-script","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GreenInfo-Network%2Fnyc-crash-mapper-etl-script/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GreenInfo-Network%2Fnyc-crash-mapper-etl-script/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/GreenInfo-Network%2Fnyc-crash-mapper-etl-script/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/GreenInfo-Network","download_url":"https://codeload.github.com/GreenInfo-Network/nyc-crash-mapper-etl-script/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247002185,"owners_count":20867421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["carto","carto-api","extract-transform-load","heroku-scheduler","nyc","open-data","python","socrata-api"],"created_at":"2024-08-01T12:04:30.020Z","updated_at":"2025-04-03T12:30:59.451Z","avatar_url":"https://github.com/GreenInfo-Network.png","language":"Python","funding_links":[],"categories":["Python","open-data"],"sub_categories":[],"readme":"# ETL SCRIPT FOR NYC CRASH MAPPER\n\nExtract, Transform, and Load script for fetching new data from the NYC Open Data Portal's vehicle collision data and loading into the NYC Crash Mapper table on CARTO.\n\n\n## Python\n\nThis script is written for Python 3.8 The `python` and `pip` commands below reflect this.\n\n\n## Setup\n\nSet the following environment variables in your shell. Copy in the values from the Heroku panel, or from `heroku config -a nyc-crash-mapper-etl` if you use the Heroku CLI.\n\n```\nexport CARTO_API_KEY='\u003credacted\u003e'\nexport CARTO_MASTER_KEY='\u003credacted\u003e'\nexport SOCRATA_APP_TOKEN_SECRET='\u003credacted\u003e'\nexport SOCRATA_APP_TOKEN_PUBLIC='\u003credacted\u003e'\nexport SENDGRID_API_KEY='\u003credacted\u003e'\nexport SENDGRID_USERNAME='\u003credacted\u003e'\nexport SENDGRID_TO_EMAIL=\"\u003credacted\u003e\"\n```\n\nYou may find it useful to create a file called `.env` which contains these commands, then to use `source .env` to load those variables into your shell.\n\nDouble check that the variable was set and is in your environment:\n```\necho $SENDGRID_USERNAME\n```\n\nInstall Python requirements:\n\n```\npip3.8 install -r requirements.txt\n```\n\n\n## Running Locally\n\nRun the script using Python 2.7 by doing:\n\n```\npython3.8 main.py\n```\n\n\n## Running via a Heroku Scheduler\n\nTo run on Heroku, fill in the values and send them to Heroku via commands such as these. Include all of the variables in that environment variable list described above.\n\n```\nheroku git:remote -a nyc-crash-mapper-etl\n\nheroku config:set CARTO_API_KEY=\u003credacted\u003e\nheroku config:set CARTO_MASTER_KEY=\u003credacted\u003e\nheroku config:set SOCRATA_APP_TOKEN_SECRET=\u003credacted\u003e\nheroku config:set SOCRATA_APP_TOKEN_PUBLIC=\u003credacted\u003e\nheroku config:set SENDGRID_API_KEY='\u003credacted\u003e'\nheroku config:set SENDGRID_USERNAME='\u003credacted\u003e'\nheroku config:set SENDGRID_TO_EMAIL=\"\u003credacted\u003e\"\n```\n\nThen provision the Heroku Scheduler, and add a job simply with the following command:\n\n```\npython3.8 main.py\n```\n\n\n## Deploying the Scheduled Task\n\nAfter making changes to the script, you will want to push these changes to Heroku scheduler so the script is used the next day.\n\nTo deploy the site to the Heroku scheduler, push the code to the Heroku remote:\n\n```\nheroku git:remote -a nyc-crash-mapper-etl\n```\n\n```\ngit push heroku master\n```\n\n## Note about qgtunnel\n\nIn 2023, we needed to have a static IP for this service, so that it could be safelisted for use with a MySQL database the client is using for the Walkmapper project. Heroku does not offer static IPs itself, but there's an addon for it. the `.qgtunnel` file in the root of this repo is the config for that. Settings and docs are reachable via the add-on section of the control panel on heroku.com.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGreenInfo-Network%2Fnyc-crash-mapper-etl-script","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FGreenInfo-Network%2Fnyc-crash-mapper-etl-script","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FGreenInfo-Network%2Fnyc-crash-mapper-etl-script/lists"}