{"id":16442259,"url":"https://github.com/sungchun12/schedule-python-script-using-google-cloud","last_synced_at":"2025-09-01T03:34:07.206Z","repository":{"id":101328625,"uuid":"122375432","full_name":"sungchun12/schedule-python-script-using-Google-Cloud","owner":"sungchun12","description":":clock4: Schedules a Python script to append data into Bigquery using Google Cloud's App Engine with a cron job","archived":false,"fork":false,"pushed_at":"2024-04-16T23:42:49.000Z","size":26,"stargazers_count":13,"open_issues_count":3,"forks_count":3,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T14:11:24.567Z","etag":null,"topics":["appengine-python","bigquery","chicago-traffic","cron","google-cloud","python-script"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sungchun12.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-02-21T18:24:50.000Z","updated_at":"2025-03-09T08:49:55.000Z","dependencies_parsed_at":"2024-10-28T20:47:32.423Z","dependency_job_id":null,"html_url":"https://github.com/sungchun12/schedule-python-script-using-Google-Cloud","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sungchun12/schedule-python-script-using-Google-Cloud","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Fschedule-python-script-using-Google-Cloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Fschedule-python-script-using-Google-Cloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Fschedule-python-script-using-Google-Cloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Fschedule-python-script-using-Google-Cloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sungchun12","download_url":"https://codeload.github.com/sungchun12/schedule-python-script-using-Google-Cloud/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sungchun12%2Fschedule-python-script-using-Google-Cloud/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273069699,"owners_count":25040103,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["appengine-python","bigquery","chicago-traffic","cron","google-cloud","python-script"],"created_at":"2024-10-11T09:16:49.496Z","updated_at":"2025-09-01T03:34:07.166Z","avatar_url":"https://github.com/sungchun12.png","language":"Python","readme":"# schedule-python-script-using-Google-Cloud\n**Use Case**: Automates live Chicago traffic data and flows it into BigQuery for interactive real-time analysis\n\n**Technical Concept**: Schedules a simple Python script to append data into BigQuery using Google Cloud's App Engine with a cron job.\n\n**Source Data**: https://data.cityofchicago.org/Transportation/Chicago-Traffic-Tracker-Congestion-Estimates-by-Se/n4j6-wkkf\n\n**Architecture Reference**: http://zablo.net/blog/post/python-apache-beam-google-dataflow-cron\n\nShout out to Mylin Ackerman for all his help. Saved me weeks of research with his personal touch.\nhttps://www.linkedin.com/in/mylin-ackermann-25a00445/\n\nCheck me out on LinkedIn: https://www.linkedin.com/in/sungwonchung1/\n\n**Setup Prerequisites**:\n1. Signup for Google Cloud account and enable billing\n2. Enable BigQuery API, Stackdriver API, Google Cloud Deployment Manager V2 API, Google Compute Engine API\n\n**Order of Operations**:\n1. Develop scripts with Google cloud shell or SDK\n2. Deploy on appengine\n3. Deploy cron job\n4. Check BigQuery\n5. Connect with dataviz tool such as Tableau\n\n**Development Instructions**:\n1. Copy github repository into SDK or Google cloud shell(thankfully it has persistent storage, so you don't have to recopy the folder structure): git clone https://github.com/sungchun12/schedule-python-script-using-Google-Cloud.git\n2. Create BigQuery dataset: \"chicago_traffic\"\n\n**Deploy Instructions**:\n1. Remember to put __init__.py files into all local packages\n2. Change directory: cd ~/chicago-traffic\n3. Install all required packages into local lib folder: pip install -r requirements.txt -t lib\n4. To deploy App Engine app, run: gcloud app deploy app.yaml\n5. To deploy App Engine CRON, run: gcloud app deploy cron.yaml\n\n**Folder Structure**:\n\n![alt text](https://storage.googleapis.com/demos-sung.appspot.com/Folder%20Structure.PNG \"Using Google Cloud Shell\")\n\ninit.py needed to properly deploy within App Engine\n\nappend_data.py - call the Chicago live traffic API and appends it into BigQuery\n\napp.yaml - definition of Google App Engine application\n\nappengine_config.py adds dependencies to locally installed packages (from lib folder)\n\ncron.yaml - definition of Google App Engine CRON job\n\nmain.py - entry point for the web application and calls the function contained within \"append_data.py\"\n\nrequirements.txt - file for pip package manager, which contains list of all required packages to run the application and the pipeline\n\nlib - local folder with all pip-installed packages from requirements.txt file\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsungchun12%2Fschedule-python-script-using-google-cloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsungchun12%2Fschedule-python-script-using-google-cloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsungchun12%2Fschedule-python-script-using-google-cloud/lists"}