{"id":17047635,"url":"https://github.com/residentmario/subway-explorer-api","last_synced_at":"2026-04-19T13:31:51.272Z","repository":{"id":66134386,"uuid":"117621090","full_name":"ResidentMario/subway-explorer-api","owner":"ResidentMario","description":"API for ground-truth MTA subway arrival and departure times.","archived":false,"fork":false,"pushed_at":"2018-08-30T02:05:13.000Z","size":75,"stargazers_count":3,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-22T21:32:22.253Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ResidentMario.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-01-16T02:35:12.000Z","updated_at":"2023-02-23T14:27:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"91689b57-dca0-45ed-aa9f-7ad886f9485a","html_url":"https://github.com/ResidentMario/subway-explorer-api","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ResidentMario/subway-explorer-api","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fsubway-explorer-api","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fsubway-explorer-api/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fsubway-explorer-api/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fsubway-explorer-api/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ResidentMario","download_url":"https://codeload.github.com/ResidentMario/subway-explorer-api/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ResidentMario%2Fsubway-explorer-api/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32009088,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-18T20:23:30.271Z","status":"online","status_checked_at":"2026-04-19T02:00:07.110Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-14T09:49:53.762Z","updated_at":"2026-04-19T13:31:51.256Z","avatar_url":"https://github.com/ResidentMario.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## About \n\nThe [Metropolitan Transit Authority](https://en.wikipedia.org/wiki/Metropolitan_Transportation_Authority) is the \nprimary public transportation authority for the greater New York City region. It provides real-time information about \nits buses, subway trains, and track trains using a bundle of what are called [GTFS-Realtime \nfeeds](https://developers.google.com/transit/gtfs-realtime/). Each GTFS-RT feed represents a snapshot of a slice of the \nMTA's service jurisdiction at a certain timestamp.\n\nThis repository comprises:\n\n* Scripts for pouring archived MTA GTFS-Realtime data into a `sqlite` database of station arrival and departure times (using \nthe [`gtfs-tripify`](https://github.com/ResidentMario/gtfs-tripify) library and some glue code).\n* An Express Node.JS application serving an API based on that data.\n\nThe [`subway-explorer-webapp`](https://github.com/ResidentMario/subway-explorer-webapp) repository defines a model web application using this API.\n\n## Installation\n\nYou will need to have [Node.JS](https://nodejs.org/en/) and [Python](https://www.python.org/) installed and configured.\n\nClone this repository:\n\n```sh\ngit clone https://github.com/ResidentMario/subway-explorer-api\n```\n\nCurrently this API is in proof-of-concept, and is hard-coded to serve a specific range of data (a selection of a dates from a handful of days in February 2018). The database serving said data is stored in [Quilt](https://github.com/quiltdata/quilt), and you can get it locally by doing the following:\n\n```\npip install quilt\nquilt install ResidentMario/subway_explorer_data_package\nquilt export ResidentMario/subway_explorer_data_package\n```\n\nThis will create a `logbooks.sqlite` database wherever you ran the command. Now create an `.env` file in the root folder whose contents is a pointer to this file:\n\n```sh\nDATABASE_FILEPATH=/path/to/your/database/logbooks.sqlite\n```\n\nTo install the necessary packages and then spin the service up, run:\n\n```sh\nnpm install\nnode index.js\n```\n\nThe API will now listen for input on `http://localhost:3000/`.\n\n## Endpoints\n\nThere are two routes in the API. The first is `locate-stations`, which simply returns the station nearest a given \nlatitude and longitude. If you visit the following URL in your local browser:\n\n```\nhttp://localhost:3000/locate-stations/json?line=1\u0026x=-74.01\u0026y=40.70\u0026heading=N\u0026time=2018-01-18T14:00\n```\n\nYou will be served the following as `text/json`:\n\n```\n{\"taxicab_dist\":0.005731999999994741,\n \"stop_id\":\"142N\",\"stop_name\":\"South Ferry\",\n \"stop_lat\":40.702068,\n \"stop_lon\":-74.013664,\n \"authority_start_time\":1514782800,\n \"authority_end_time\":1522555200}\n```\n\nThe second and more interesting route is `poll-travel-times`. This route returns the most efficient trip between two \nstops (on the same line) possible at given timestamps. For example, if you visit the following URL:\n\n```\nhttp://localhost:3000/poll-travel-times/json?line=2\u0026start=247N\u0026end=220N\u0026timestamps=2018-01-18T02:00\n```\n\nYou will see:\n\n```\n[{\"status\":\"OK\",\n  \"results\": \n    [\n        {\"event_id\":10306,\n         \"trip_id\":\"014350_2..N08X010\",\n         \"unique_trip_id\":\"014350_2..N08X010_426\",\n         \"route_id\":\"2\",\n         \"action\":\"STOPPED_OR_SKIPPED\",\n         \"minimum_time\":1516260172,\n         \"maximum_time\":1516260232,\n         \"stop_id\":\"247N\",\n         \"latest_information_time\":\"1516260232\"},\n        {...},\n        ...\n    ]\n}]\n```\n\nTo inspect multiple timestamps simultaneously, separate them using a pipe character `|`:\n\n```\nhttp://localhost:3000/poll-travel-times/json?line=2\u0026start=247N\u0026end=220N\u0026timestamps=2018-01-18T02:00|2018-01-18T14:00\n```\n\nYou will notice that the stops inputted (and returned) by the API are MTA-defined stop IDs, *not* human-readable station names. You might use the `locate-station` API route to resolve these, or resolve the names manually or in your own code using the `stops.txt` record in the [official MTA GTFS record](http://web.mta.info/developers/data/nyct/subway/google_transit.zip).\n\nThe first stop in the sequence may have a `minimum_time` of `null`. More rarely, intermediate stops in the sequence may also have this property.\n\nIf the last stop in the sequence corresponds with the last stop on the route, it may have a `maximum_time` far away from the `minimum_time` (the train might be \"parked\" at the station). In this case you will find the `minimum_time` to be more informative.\n\nTo shut the API down, enter `Ctrl+C` into the terminal console you launched it in.\n\n## Rolling your own\n\nThe `Installation` example uses a simple example database that was prepared in advance. To use this API properly, you will need to generate a proper database yourself (undocumented: and also change the hard-coded time lookups in the code).\n\nBefore you start, make sure you have [Python](https://www.python.org/) installed on your machine. Then use `pip` to install the following required packages:\n\n```sh\npip install git+git://github.com/ResidentMario/gtfs-tripify.git@master\npip install click\n```\n\n### Localizing data\n\nThe MTA publishes GTFS-Realtime data for every line in its system. This data is split up across several different feeds, each one of which updated every 30 seconds. This data is the raw input for this project, and if you want to do something similar you need to archive it somehow.\n\nUnfortunately no public archives up-to-date archives exist; if you want to use this API for current data, you will need to archive the data yourself.\n\nI've done this using an AWS Lambda function feeding into a packet of AWS S3 buckets, triggered by a AWS CloudWatch cron job running once a minute. A copy of the script doing this work is available as `aws-lambda-archiver.py` in the `scripts` folder. You can use this script to archive feeds up on AWS yourself.\n\nThe `localize-gtfs-r-records.py` script can download batches of relevant GTFS-Realtime fields:\n\n    python localize-gtfs-r-records.py '2018-01-19' ~/Desktop/subway-explorer-datastore/\n    \nYou must have the [AWS CLI](https://aws.amazon.com/cli/) installed (and [configured with your credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)) to run this script.\n  \nNote that the scripts and instructions in this section up to here are AWS specific. Any other cloud provider you prefer will do the job, but you will need to write your own pipeline, using this existing one as a template.\n\n### Feeding in the realtime data\n\nThen, the `compile-gtfs-feed-to-db.py` script writes that data to a database:\n\n    python compile-logbooks-to-db.py ~/Desktop/subway-explorer-datastore/ '2018-01-18T00:00' '2018-01-18T12:00' .\n\nRunning this script (actually a thin wrapper on the `gt.io.stream_to_sql` module method) will add complete stop sequences which started within the inputted time period to the database.\n\nThe table entries created are contiguous, meaning that we can follow this run (on the first half of January 18, 2018) with a second run (on the second half of January 18, 2018) and get the same result we would have gotten if we had parameterized the script with the whole day (all of January 18, 2018) to begin win with.\n\nTo achieve this, the script \"looks ahead\" and parses data up to three hours after the given end date. So for the endpoint timestamp `2018-01-18T12:00`, the script will actually parse trips all the way up to `2018-01-18T15:00`. This is done to ensure that trips that started before the `2018-01-18T12:00` cutoff but ended some time after that are populated with their complete stop sequence (e.g. no projected `EXPECTED_TO_ARRIVE_AT` records).\n\nIn the future I may implement streaming support for `gtfs-tripify`, which would de-necessitate this ugly workaround (and unlock some other interesting use cases for this data stream besides). For now, keep in mind when running this script that you must have three extra hours of data available in order for the run to be successful.\n\nBecause of this high fixed cost, it is optimal to run this script on time chunks as large as your RAM limitations allow. 12-hour chunks are approximately optimal for a 16 GB machine. This may change in the future, but for now running this script is very expensive!\n\n### Feeding in station identities\n\nThe database will also need relevant `GTFS` station records.\n\nThese are used to populate the station lookup code path in the API. The challenge is that the names and IDs of stations may change over time. So `stops.txt` from a `GTFS` roll-up must be written to the database with an `authority_start_time` and `authority_end_time`, to make sure that the given station is correct for the given time period. You can write this data using the `compile-gtfs-feed-to-db.py` script:\n\n    python compile-gtfs-feed-to-db.py ~/Downloads/google_transit.zip \"2018-01-01T00:00\" \"2018-04-01T00:00\" logbooks.sqlite\n\nOnce you have all this your database will be ready to use. To run it, see the instructions in the [Quickstart](#Quickstart).\n\n## Running the tests\n\nTo run the tests locally, `npm install mocha` if you haven't already, then run `npx mocha` from the root folder.\n\n## Using the container\n\nThis repo contains a Docker file bundled with Node.JS and this application. I found the [Node.JS-Docker integration \nquickstart](https://nodejs.org/en/docs/guides/nodejs-docker-webapp/) very helpful in getting that set up, and it's a \ngood reference on how to work with the containerized application.\n\nTo build the container image, run the following from the root folder:\n\n    docker build -t residentmario/subway-explorer-api .\n\nThen, to run the container (pointing it to `localhost:49160`):\n\n    docker run -p 49160:3000 --env-file .env -d residentmario/subway-explorer-api\n\nYou can visit the following (port-forwarded) URI in the browser to verify that the connection is being served:\n\n```\nhttp://localhost:49160/locate-stations/json?line=1\u0026x=-74.01\u0026y=40.70\u0026heading=N\u0026time=2018-01-18T14:00\n```\n\n\nYou can also run the tests by jumping inside the container and running `npx mocha`, using the following command:\n\n    docker exec -it 949cc5d81abe /bin/bash\n\nReplacing the name with the name of the running image (discoverable via `docker ps`).\n\nThis Docker container does not come with a database attached, so the API can't do anything useful right off the bat. To make it do something useful you can mount the database you created as a volume using Docker controls, or rely on similar tools in a container management platform like Kubernetes.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresidentmario%2Fsubway-explorer-api","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fresidentmario%2Fsubway-explorer-api","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fresidentmario%2Fsubway-explorer-api/lists"}