{"id":21148045,"url":"https://github.com/civictechto/ttc_subway_times","last_synced_at":"2025-07-09T07:33:38.861Z","repository":{"id":15653204,"uuid":"78588072","full_name":"CivicTechTO/ttc_subway_times","owner":"CivicTechTO","description":"A scraper to grab and publish TTC subway arrival times.","archived":false,"fork":false,"pushed_at":"2022-12-09T11:47:54.000Z","size":7725,"stargazers_count":40,"open_issues_count":35,"forks_count":32,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-04-17T12:20:12.438Z","etag":null,"topics":["scraper","transit","ttc"],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CivicTechTO.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":".github/CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-01-11T00:40:58.000Z","updated_at":"2023-07-28T22:24:07.000Z","dependencies_parsed_at":"2023-01-11T20:23:43.884Z","dependency_job_id":null,"html_url":"https://github.com/CivicTechTO/ttc_subway_times","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicTechTO%2Fttc_subway_times","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicTechTO%2Fttc_subway_times/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicTechTO%2Fttc_subway_times/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CivicTechTO%2Fttc_subway_times/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CivicTechTO","download_url":"https://codeload.github.com/CivicTechTO/ttc_subway_times/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225508888,"owners_count":17483515,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["scraper","transit","ttc"],"created_at":"2024-11-20T09:20:31.865Z","updated_at":"2024-11-20T09:20:33.938Z","avatar_url":"https://github.com/CivicTechTO.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TTC Subway times\n\nThis is a project implementing a ‘scraper’ to grab and publish TTC subway arrival times against citizen-led performance metrics analyzing service timeliness and reliability.  The end goal is to maintain and publish this performance dashboard building consensus on service quality betterment over public transit in Toronto.\n\n# State of the project\n\nWe have a Python scraper running on AWS saving predicted subway arrivals since late February 2017.  We need to process this data to generate observed station arrival times for each train at each station.  There was a couple month hiatus in data scraping because the database was full between August and November, but the scraper is now continuing to hum along nicely.\n\nWe're still trying to process the predicted arrival times obtained from the API into reasonably reliable state.  This work is happening in a Jupyter Notebook [Filtering Observed Arrivals.ipynb](doc/filtering_observed_arrivals.ipynb); it is mostly in SQL, despite being in a Python notebook.\n\nTake a look at [How to Get Involved](#how-to-get-involved) with your expertise. Feel free to follow along; your informed feedback will surely lead to better data.\n\n# First Step Metric\n\n**Buffer Time** - A measure of reliability useful for the rider:\n\n- how much extra time he/she needs to buffer in order for their trip to be on time 1 trip out of 20.\n\n# Documentation\n\nReview the [doc/](doc/) folder for the [Jupyter Notebooks](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html#id5) explaining how we explored developing this project and understanding the data. Jupyter notebooks are a way of having code, descriptive text, and output together in one document to present a narrative around data exploration. \n\n**If you're interested in a Data Dictionary**, [click here](doc/DataDictionary.md) (it's also in the `doc/` folder).\n\n## Presentations\n\nYou can watch Raphael give a presentation to CivicTech Toronto January 15, 2019 on [this blurry Youtube](https://youtu.be/DtNR5whN9gg?t=728) and the slides are [here](https://radumas.info/presentations/20190115#/)\n\n# Data Flow and Data Structure\n\nThe scraper runs every minute during TTC service hours.  Each of these runs is termed a `polls`\nand has a start / end time.  During one run of the scraper, each station gets its predicted arrivals returned as a `request`, noting which station it is using `stationid` and `lineid`. \nFor each request, 3 predicted arrivals are recorded for each line and direction at that station. These responses include the train's `traindirection` and its unique ID `trainid`, the time until the train's arrival `timint` and a `train_message`, whether the train is arriving, at station, or is delayed.\n\nFor more info: review the API exploration notebook under [`doc/API_exploration.ipynb`](https://github.com/CivicTechTO/ttc_subway_times/blob/master/doc/API_exploration.ipynb)\n\n# Analyzing the Data\n\n:warning: **We're not entirely sure how to make this work, you can get data by finding us in the `#transportation` channel on [Slack](https://civictechto-slack-invite.herokuapp.com/). If you think you can help with this, check out [issue #59](https://github.com/CivicTechTO/ttc_subway_times/issues/59)**\n\nHistorical data is stored in the s3://ttc.scrape bucket, the fetch_s3.py script can be used to automatically fetch and assemble this data. Its usage is\n\n```shell\npython3 fetch_s3.py --bucket ttc.scrape --start_date 2018-12-02 --end_date 2018-12-05 --output_dir out/\n```\n\nIf end date is omitted it is taken to be the latest available date. This script can only select by day.\n\nThis will generate three CSVs in the output_dir which can be copied into the Postgres database with the following SQL commands\n\n```SQL\nCOPY polls FROM '/path/to/polls.csv' DELIMITER ',' CSV HEADER;\nCOPY requests FROM '/path/to/requests.csv' DELIMITER ',' CSV HEADER;\nCOPY ntas_data FROM '/path/to/responses.csv' DELIMITER ',' CSV HEADER;\n```\n\n# Automating the Scraper Runs\n\nThere are two ways that the scraper can be automated, via Docker/cron, or via Serverless with AWS Lambda. No matter how you choose to run it the scraper runs through ttc_scraper_api.py\n\n## Storage Backends\n\nThere are two ways the data can be stored once it has been scraped, AWS S3 and Postgres.\n\nAWS S3 stores each scrape in a JSON collected by service day (see Consolidate function). This requires an AWS account. This can be enabled by with the `--s3` flag. The advantage of S3 is that it requires no persistant server, and is extremely cheap. Its main disadvantage is that the data is not as easily queryable as an SQL database and some steps are required before it can be queried in SQL (see [Analyzing the Data](#analyzing-the-data)). This storage method is well suited to the AWS Lambda scraping mode.\n\nPostgres requires a running Postgres instance. While the data is immediately queryable, it requires a Postgres server to be always running, which increases the work, risk and cost of the project. This can be selected with the `--postgres` flag and is most often used with the Docker/Cron scraping mode.\n\nBoth modes store the same data, in largely the same structure (nested JSONs vs tables with joinable keys).\n\n## AWS Lambda Scraping\n\nThere is a mode which will allow scraping via AWS Lambda with logging added to AWS Cloudwatch. This mode uses the Serverless framework.\n\nThe [Serverless](https://serverless.com/) framework is a suite of tooling which allows the easy deployment and management of serverless code.\n\nThis allows us to run this code without having to spin up/monitor for an instance manually. And since we only pay for the code when it is running the compute costs are nearly zero.\n\n### Setup\n\nIn addition to installing the Python [requirements](requirements.txt) we need to install the [Serverless framework](https://serverless.com/) with [npm](https://www.npmjs.com/get-npm) ([install instructions for Ubuntu](https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04) by running `npm install` in the project root. \n\nOn Ubuntu you'll need to add the `serverless` binary to your `PATH` variable with `PATH=\"$PATH:/[PATH/TO]/ttc_subway_times/node_modules/serverless/bin/\"` (replace `[PATH/TO]` with the absolute path to the repository). This will only be temporary for your session, if you're going to use `serverless` for other projects, you should probably install it globally with `sudo npm install -g`.\n\nTell Serverless which AWS creds you would like to use with:\n\n`serverless config credentials --provider aws --key \u003cAWS_ACCESS_KEY\u003e --secret \u003cAWS_SECRET_KEY\u003e`\n\nCreating these credentials must be done through your AWS account. A good guide to this\nprocess can be found on the [Serverless Website](https://serverless.com/framework/docs/providers/aws/guide/credentials/)\n\n#### Configure the `serverless.yml`\n\nThe template `serverless.yml` is configured to use dev and prod environments. This will create functions with `dev-` or `prod-` prepended to them, and will also direct output to the buckets that are defined in the custom section\n\nMove `serverless.yml.template` to `serverless.yml`.\n\n[Create an S3 bucket](https://s3.console.aws.amazon.com/s3/home). Replace the angle bracketed bucket names (under the `custom:` properties at the end of the document with the **names** of the buckets you created.\n\nAt the time of writing the schedule line in serverless.yml is set as\n\n```yaml\n    rate: cron(* 0-2,5-23 * * ? *)\n```\nwhich means that it should run every minute from 5am to 2am every day. More information on this cron line can be found on the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html), in this documentation references to UTC should be ignored, we use the 'serverless-local-schedule' package which allows us to specify crons in local time rather than UTC (otherwise the behaviour would change during daylight savings time).\n\n### Deploy\n\nYou'll need to install docker for this step ([Ubuntu install instructions](https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04)).\n\nFinally deploy the function with \n```shell\nserverless deploy -v\n```\nThis command will deploy to the dev environment by default, the environment can be specified on the command line with the `--stage` flag (acceptable values for this project are `dev` and `prod`)\n\nLogs are automatically persisted to Cloudwatch.\n\n### Consolidate Function\nIf you are scraping with the AWS Lambda function with the S3 data destination it will write a JSON to `s3://\u003cBUCKET\u003e/\u003cSERVICE DATE\u003e`. As the scraper runs once per minute, this results in a very large number of small files, which are inefficient and expensive to store and transfer.\n\nTo remedy this there is a second 'consolidate' serverless function which runs at the end of every service day and combines the previous day into a .tar.gz file, storing it at `s3://\u003cBUCKET\u003e/\u003cSERVICE_DATE\u003e.tar.gz`.\n\nThis isn't relevant if you are storing the data in Postgres with the Lambda scraper, but this configuration will require you to modify serverless.yml.\n\n## Docker/Cron Scraping\n\nAnother way to run the scraper is to run it periodically a Cron in a Docker container. This is useful if you would like to scrape from a computer that is always left on.\n\nFollow the below command to set up a Python 3 environment and install requirements.\n\n\u003cdetails\u003e\u003csummary\u003eRunning inside docker\u003c/summary\u003e\nFollow the instructions [here](https://docs.docker.com/compose/install/) to get `docker-compose`.\n\nAll you need to do is `docker-compose run --rm scraper`.\nThis will setup a database container, initialize the tables, and then run the initial scrape.\n\nTo have cli access to the data you can use `docker-compose exec db psql -U postgres -d ttc`.\nCommands of interest:\n- `\\?`: list the help information for all the special commands\n- `\\h`: list all the sql commands that are available\n- `\\q`: quit the console\n\u003c/details\u003e\n\n### Linux/Unix\n\nTo use Mac or Linux, add the following to cron. Don't forget to change `/path/to/ttc_api_scraper.py`.\n\nNote: These commands assume postgres as a data destination.\n\n```bash\n# m h  dom mon dow   command\n* 5-23 * * 1-5 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --postgres\n* 0-1 * * 1-5 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --postgres\n* 5-23 * * 6-7 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --postgres\n* 0-2 * * 6-7 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --postgres\n```\nOr to run every 20s while filtering out any \"arriving\" records\n```bash\n* 5-23 * * 1-5 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres\n* 0-1 * * 1-5 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres\n* 5-23 * * 6-7 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres\n* 0-2 * * 6-7 cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres\n* 5-23 * * 1-5 (sleep 20; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered  --postgres)\n* 0-1 * * 1-5 (sleep 20; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered  --postgres)\n* 5-23 * * 6-7 (sleep 20; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres)\n* 0-2 * * 6-7 (sleep 20; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres)\n* 5-23 * * 1-5 (sleep 40; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres)\n* 0-1 * * 1-5 (sleep 40; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres)\n* 5-23 * * 6-7 (sleep 40; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered)\n* 0-2 * * 6-7 (sleep 40; cd /path/to/repo/ttc_subway_times/ \u0026\u0026 python3 ttc_api_scraper.py --filter --schemaname filtered --postgres)\n```\n\n### Windows users\n\nUse Task Scheduler.\n\n### cronic.py\n\nIf the above sounds complicated, here's a simple looping script that calls `ttc_api_scraper.py` every minute during the TTC's operating hours. Just start it in your command line with\n```shell\npython cronic.py\n```\n\nAnd let it collect the data.\n\n# Database setup\nIf you would like to use Postgres as a data repository, the database engine used to store the data is PostgreSQL. Instructions to obtain the latest and greatest version are [here](https://www.postgresql.org/). After setting up your database, you can run the contents of `create_tables.sql` in a pgAdmin query window (or run it as a sql query).\n\nYou will also need to edit `db.cfg`\n```ini\n[DBSETTINGS]\ndatabase=ttc\nhost=host.ip.address\nuser=yourusername\npassword=pw\n```\n\n\n# How to Get Involved\n\nWe discuss the project on [CivicTechTO's Slack Team](https://civictechto-slack-invite.herokuapp.com/) on the `#transportation` channel. This is probably the best place to introduce yourself and ask how you can participate. There are links in that channel to get access to ~1 month of the raw data in a `csv` or a PostgreSQL dump. You can also ask about getting read access to the database. Alternatively you can set up the scraper yourself and play with your own archive locally, hack away!\n\nIf you're exploring the data, please write up your exploration in a Jupyter Notebook/RMarkdown/Markdown file and place it in the `doc/` folder and submit a Pull Request with details on what you explored.\n\nOtherwise have a look at [open issues](https://github.com/CivicTechTO/ttc_subway_times/issues) and comment on any thing you think you could contribute to or open your own issue if you notice something to improve upon in the code.\n\n# Sources of Inspiration\n\nBoldly following in [others' footsteps](https://blog.sammdot.ca/pockettrack-tracking-subway-trains-is-hard-9c8fdfb7fd3c?source=collection_home---4------0----------)\nSee more on the [Resources page](https://github.com/CivicTechTO/ttc_subway_times/wiki/Resources)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcivictechto%2Fttc_subway_times","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcivictechto%2Fttc_subway_times","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcivictechto%2Fttc_subway_times/lists"}