{"id":18174919,"url":"https://github.com/kylebarron/all-transit","last_synced_at":"2025-09-06T16:33:28.542Z","repository":{"id":37873399,"uuid":"239042442","full_name":"kylebarron/all-transit","owner":"kylebarron","description":"Interactive visualization of all transit in the Transitland database","archived":false,"fork":false,"pushed_at":"2023-01-11T20:30:54.000Z","size":36958,"stargazers_count":27,"open_issues_count":25,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-12-23T07:27:45.159Z","etag":null,"topics":["animation","mapbox-gl-js","transit","transit-data","transitland","vector-tiles","webgl"],"latest_commit_sha":null,"homepage":"https://all-transit.com","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kylebarron.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-02-07T23:47:31.000Z","updated_at":"2024-12-06T17:17:00.000Z","dependencies_parsed_at":"2023-02-09T06:01:57.912Z","dependency_job_id":null,"html_url":"https://github.com/kylebarron/all-transit","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fall-transit","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fall-transit/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fall-transit/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kylebarron%2Fall-transit/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kylebarron","download_url":"https://codeload.github.com/kylebarron/all-transit/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":232133629,"owners_count":18477294,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["animation","mapbox-gl-js","transit","transit-data","transitland","vector-tiles","webgl"],"created_at":"2024-11-02T16:08:09.325Z","updated_at":"2025-01-01T22:45:10.333Z","avatar_url":"https://github.com/kylebarron.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# All Transit\n\n[![Build Status](https://travis-ci.org/kylebarron/all-transit.svg?branch=master)](https://travis-ci.org/kylebarron/all-transit)\n\n[![Static image of US Transit](static_image/us.png)](https://all-transit.com)\n\n[Website: https://kylebarron.dev/all-transit](https://kylebarron.dev/all-transit)\n\nAll transit, as reported by the [Transitland][transitland] database. Inspired by\n[_All Streets_][all_streets]. I have a blog post [here][blog_post] detailing\nmore information about the project.\n\n[transitland]: https://transit.land\n[all_streets]: https://benfry.com/allstreets/map5.html\n[blog_post]: https://kylebarron.dev/blog/all-transit\n\n## Website\n\nThe code for the website is in `site/`. It uses React, Gatsby, Deck.gl, and\nReact Map GL/Mapbox GL JS.\n\n## Static SVG/PNG\n\nThe `static_image` folder contains code to generate an SVG and PNG of all the\nroutes in the U.S. It uses `d3` and\n[`geo2svg`](https://github.com/d3/d3-geo-projection/blob/master/README.md#geo2svg).\n\n## Data\n\nMost of the data-generating code for this project is done in Bash,\n[`jq`](https://stedolan.github.io/jq/), GNU Parallel, SQLite, and a couple\nPython scripts. Data is kept in _newline-delimited JSON_ and _newline-delimited\nGeoJSON_ for all intermediate steps to facilitate streaming and keep memory use\nlow.\n\n### Data Download\n\nClone this Git repository and install the Python package I wrote to easily\naccess the Transitland API.\n```bash\ngit clone https://github.com/kylebarron/all-transit\ncd all-transit\npip install transitland-wrapper\nmkdir -p data\n```\n\nEach of the API endpoints allows for a bounding box. At first, I tried to just\npass a bounding box of the entire United States to these APIs and page through\nthe results. Unsurprisingly, that method isn't successful for the endpoints that\nhave more data to return, like stops and schedules. I found that for the\nschedules endpoint, the API was really slow and occasionally timed out when I\nwas trying to request something with `offset=100000`, because presumably it\ntakes a lot of time to find the 100,000th row of a given query.\n\nBecause of this, I found it best in general to split API queries into smaller\npieces, by using e.g. operator ids or route ids.\n\n#### Operators\n\nDownload all operators whose service area intersects the continental US, and\nthen extract their identifiers.\n```bash\n# All operators\ntransitland operators --page-all \u003e data/operators_new.geojson\n\n# All operator `onestop_id`s\ncat data/operators.geojson \\\n    | jq '.properties.onestop_id' \\\n    | uniq \\\n    | \\\n    tr -d \\\" \\\n    \u003e data/operator_onestop_ids.txt\n```\n\n#### Routes\n\nI downloaded routes by the geometry of the US, and then later found it best to\nsplit the response into separate files by operator. If I were to run this\ndownload again, I'd just download routes by operator to begin with.\n\n```bash\n# All routes\nrm -rf data/routes\nmkdir -p data/routes\ncat data/operator_onestop_ids.txt | while read operator_id\ndo\n    transitland routes \\\n        --page-all \\\n        --operated-by $operator_id \\\n        --per-page 1000 \\\n        \u003e data/routes/$operator_id.geojson\ndone\n```\n\nNow that the routes are downloaded, I extract the identifiers for all\n`RouteStopPattern`s and `Route`s.\n```bash\nmkdir -p data/route_stop_patterns_by_onestop_id/\ncat data/operator_onestop_ids.txt | while read operator_id\ndo\n    cat data/routes/$operator_id.geojson \\\n        | jq '.properties.route_stop_patterns_by_onestop_id[]' \\\n        | uniq \\\n        | tr -d \\\" \\\n        \u003e data/route_stop_patterns_by_onestop_id/$operator_id.txt\ndone\n\nmkdir -p data/routes_onestop_ids/\ncat data/operator_onestop_ids.txt | while read operator_id\ndo\n    cat data/routes/$operator_id.geojson \\\n        | jq '.properties.onestop_id' \\\n        | uniq \\\n        | tr -d \\\" \\\n        \u003e data/routes_onestop_ids/$operator_id.txt\ndone\n```\n\nIn order to split up how I later call the `ScheduleStopPairs` API endpoint, I\nsplit the `Route` identifiers into sections. There are just shy of 15,000 route\nidentifiers, so I split into 5 files of roughly equal 3,000 route identifiers.\n```bash\n# Split into fifths so that I can call the ScheduleStopPairs API in sections\ncat routes_onestop_ids.txt \\\n    | sed -n '1,2999p;3000q' \\\n    \u003e routes_onestop_ids_1.txt\ncat routes_onestop_ids.txt \\\n    | sed -n '3000,5999p;6000q' \\\n    \u003e routes_onestop_ids_2.txt\ncat routes_onestop_ids.txt \\\n    | sed -n '6000,8999p;9000q' \\\n    \u003e routes_onestop_ids_3.txt\ncat routes_onestop_ids.txt \\\n    | sed -n '9000,11999p;12000q' \\\n    \u003e routes_onestop_ids_4.txt\ncat routes_onestop_ids.txt \\\n    | sed -n '12000,15000p;15000q' \\\n    \u003e routes_onestop_ids_5.txt\n```\n\n#### Stops\n\n`Stops` are points along a `Route` or `RouteStopPattern` where passengers may\nget on or off.\n\nDownloading stops by operator was necessary to keep the server from paging\nthrough too long of results. I was stupid and concatenated them all into a\nsingle file, which I later saw that I needed to split with `jq`. If I were\ndownloading these again, I'd write each `Stops` response into a file named by\noperator.\n```bash\n# All stops\nrm -rf data/stops\nmkdir -p data/stops\ncat data/operator_onestop_ids_new.txt | while read operator_id\ndo\n    transitland stops \\\n        --page-all \\\n        --served-by $operator_id \\\n        --per-page 1000 \\\n        \u003e data/stops/$operator_id.geojson\ndone\n```\n\n#### Route Stop Patterns\n\n`RouteStopPattern`s are portions of a route. I think an easy way to think of the\ndifference is the a `Route` can be a MultiLineString, while a `RouteStopPattern`\nis always a LineString.\n\nSo far I haven't actually needed to use `RouteStopPattern`s for anything. I\nwould've ideally matched `ScheduleStopPair`s to `RouteStopPattern`s instead of\nto `Route`s, but I found that some `ScheduleStopPair` have missing\n`RouteStopPattern`s, while `Route` is apparently never missing.\n\n```bash\nmkdir -p data/route_stop_patterns/\ncat data/operator_onestop_ids.txt | while read operator_id\ndo\n    transitland onestop-id \\\n        --page-all \\\n        --file data/route_stop_patterns_by_onestop_id/$operator_id.txt \\\n        \u003e data/route_stop_patterns/$operator_id.json\ndone\n```\n\n#### Schedule Stop Pairs\n\n`ScheduleStopPair`s are edges along a `Route` or `RouteStopPattern` that define\na single instance of transit moving between a pair of stops along the route.\n\nI at first tried to download this by `operator_id`, but even that stalled the\nserver because some operators in big cities have millions of different\n`ScheduleStopPair`s. Instead I downloaded by `route_id`.\n\nApparently you can only download by `Route` and not by `RouteStopPattern`, or\nelse I probably would've chosen the latter, which might've made associating\n`ScheduleStopPair`s to geometries easier.\n\nI used each fifth of the `Route` identifiers from earlier so that I could make\nsure each portion was correctly downloaded.\n```bash\n# All schedule-stop-pairs\n# Best to loop over route_id, not operator_id\nmkdir -p data/ssp/\ncat data/operator_onestop_ids_new.txt | while read operator_id\ndo\n    cat data/routes_onestop_ids/$operator_id.txt | while read route_id\n    do\n        transitland schedule-stop-pairs \\\n            --page-all \\\n            --route-onestop-id $route_id \\\n            --per-page 1000 \\\n            --active \\\n            | gzip \u003e\u003e data/ssp/$operator_id.json.gz\n        touch data/ssp/$operator_id.finished\n    done\ndone\n\nfor i in {1..5}; do\n    cat data/routes_onestop_ids_${i}.txt | while read route_id\n    do\n        transitland schedule-stop-pairs \\\n        --page-all \\\n        --route-onestop-id $route_id \\\n        --per-page 1000 --active \\\n        | gzip \u003e\u003e data/ssp/ssp${i}.json.gz\n    done\ndone\n```\n\n### Vector tiles for Operators, Routes, Stops\n\nI generate vector tiles for the routes, operators, and stops. I have `jq`\nfilters in `code/jq/` to reshape the GeoJSON into the format I want, so that the\ncorrect properties are included in the vector tiles.\n\nIn order to keep the size of the vector tiles small:\n\n- The `stops` layer is only included at zoom 11\n- The `routes` layer only includes metadata about the identifiers of the stops\n  that it passes at zoom 11\n\n```bash\n# Writes mbtiles to data/mbtiles/routes.mbtiles\n# The -c is important so that each feature gets output onto a single line\nfind data/routes -type f -name '*.geojson' -exec cat {} \\; \\\n    `# Apply jq filter at code/jq/routes.jq` \\\n    | jq -c -f code/jq/routes.jq \\\n    | bash code/tippecanoe/routes.sh\n\n# Writes mbtiles to data/mbtiles/operators.mbtiles\nbash code/tippecanoe/operators.sh data/operators.geojson\n\n# Writes mbtiles to data/mbtiles/stops.mbtiles\n# The -c is important so that each feature gets output onto a single line\nfind data/stops -type f -name '*.geojson' -exec cat {} \\; \\\n    | jq -c -f code/jq/stops.jq \\\n    | bash code/tippecanoe/stops.sh\n```\n\nCombine into single mbtiles\n```bash\ntile-join \\\n    -o data/mbtiles/all.mbtiles \\\n    `# Don't enforce size limits;` \\\n    `# Size limits already enforced individually for each sublayer` \\\n    --no-tile-size-limit \\\n    `# Overwrite existing mbtiles` \\\n    --force \\\n    `# Input files` \\\n    data/mbtiles/stops.mbtiles \\\n    data/mbtiles/operators.mbtiles \\\n    data/mbtiles/routes.mbtiles\n```\n\nThen publish! Host on a small server with\n[`mbtileserver`](https://github.com/consbio/mbtileserver) or export the\n`mbtiles` to a directory of individual tiles with\n[`mb-util`](https://github.com/mapbox/mbutil) and upload the individual files to\nS3.\n\nI'll upload this to S3:\n\nExport mbtiles to a directory\n```bash\nmb-util \\\n    `# Existing mbtiles` \\\n    data/all.mbtiles \\\n    `# New directory` \\\n    data/all \\\n    `# Set file extension to pbf` \\\n    --image_format=pbf\n```\n\nThen upload to S3\n```bash\n# First the tile.json\naws s3 cp \\\n    code/tile/op_rt_st.json s3://data.kylebarron.dev/all-transit/op_rt_st/tile.json \\\n    --content-type application/json \\\n    `# Set to public read access` \\\n    --acl public-read\naws s3 cp \\\n    data/all s3://data.kylebarron.dev/all-transit/op_rt_st/ \\\n    --recursive \\\n    --content-type application/x-protobuf \\\n    --content-encoding gzip \\\n    `# Set to public read access` \\\n    --acl public-read \\\n    `# 6 hour cache; one day swr` \\\n    --cache-control \"public, max-age=21600, stale-while-revalidate=86400\"\n```\n\n### Schedules\n\nThe schedule component is my favorite part of the project. You can see streaks\nmoving around that correspond to transit vehicles: trains, buses, ferries. This\ndata comes from actual schedule information from the Transitland API and matches\nit to route geometries. (Though it's not real-time info, so it doesn't reflect\ndelays).\n\nI use the deck.gl\n[`TripsLayer`](https://deck.gl/#/documentation/deckgl-api-reference/layers/trips-layer)\nto render the schedule data as an animation. That means that I need to figure\nout the best way to transport three-dimensional `LineStrings` (where the third\ndimension refers to time) to the client. Unfortunately, at this time Tippecanoe\n[doesn't support three-dimensional\ncoordinates](https://github.com/mapbox/tippecanoe/issues/714). The\nrecommendation in that thread was to reformat to have individual points with\nproperties. That would make it harder to associate the points to lines, however.\nI eventually decided it was best to pack the data into tiled\ngzipped-minified-GeoJSON. And since I know that all features are `LineStrings`,\nand since I have no properties that I care about, I take only the coordinates,\nso that the data the client receives is like:\n\n```json\n[\n    [\n        [\n            0, 1, 2\n        ],\n        [\n            1, 2, 3\n        ]\n    ],\n    [\n        []\n        ...\n    ]\n]\n```\n\nI currently store the third coordinate as seconds of the day. So that 4pm is `16\n* 60 * 60 = 57000`.\n\nIn order to make the data download manageable, I cut each GeoJSON into xyz map\ntiles, so that only data pertaining to the current viewport is loaded. For dense\ncities like Washington DC and New York City, some of the LineStrings are very\ndense, so I cut the schedule tiles into full resolution at zoom 13, and then\ngenerate overview tiles for lower zooms that contain a fraction of the features\nof their child tiles.\n\nI generated tiles in this manner down to zoom 2, but discovered that performance\nwas very poor on lower-powered devices like my phone. Because of that, I think\nit's best to have the schedule feature disabled by default.\n\n#### Data Processing\n\nI originally tried to do everything with `jq`, but the schedule data for all\nroutes in the US as uncompressed JSON is \u003e100GB and things were too slow. I\ntried SQLite and it's pretty amazing.\n\nTo import `ScheduleStopPair` data into SQLite, I first converted the JSON files\nto CSV:\n```bash\n# Create CSV file with data\nmkdir -p data/ssp_sqlite/\nfor i in {1..5}; do\n    # header line\n    gunzip -c data/ssp/ssp${i}.json.gz \\\n        | head -n 1 \\\n        | jq -rf code/ssp/ssp_keys.jq \\\n        | gzip \\\n        \u003e data/ssp_sqlite/ssp${i}.csv.gz\n    # Data\n    gunzip -c data/ssp/ssp${i}.json.gz \\\n        | jq -rf code/ssp/ssp_values.jq \\\n        | gzip \\\n        \u003e\u003e data/ssp_sqlite/ssp${i}.csv.gz\ndone\n```\n\nThen import the CSV files into SQLite:\n```bash\nfor i in {1..5}; do\n    gunzip -c data/ssp_sqlite/ssp${i}.csv.gz \\\n        | sqlite3 -csv data/ssp_sqlite/ssp.db '.import /dev/stdin ssp'\ndone\n```\n\nCreate SQLite index on `route_id`\n```bash\nsqlite3 data/ssp_sqlite/ssp.db \\\n    'CREATE INDEX route_onestop_id_idx ON ssp(route_onestop_id);'\n```\n\nI found it best to loop over `route_id`s when matching schedules to route\ngeometries. Here I create a crosswalk with the operator id for each route, so\nthat I can pass to my Python script 1) `ScheduleStopPair`s pertaining to a\nroute, 2) `Stops` by operator and 3) `Routes` by operator.\n```bash\n# Make xw with route_id: operator_id\ncat data/routes/*.geojson \\\n    | jq -c '{route_id: .properties.onestop_id, operator_id: .properties.operated_by_onestop_id}' \\\n    \u003e data/route_operator_xw.json\n```\n\nHere's the meat of connecting schedules to route geometries. The bash script\ncalls `code/schedules/ssp_geom.py`, and the general process of that script is:\n\n1. Load stops, routes, and route stop patterns for the operator\n2. Load provided `ScheduleStopPair`s from stdin\n3. Iterate over every `ScheduleStopPair`. For each pair, try to find the route stop pattern it's associated with. If it exists, use the linear stop distances contained in the `ScheduleStopPair` and Shapely's linear referencing methods to take the substring of that `LineString`.\n4. If a route stop pattern isn't found directly, find the associated route, then find its associate route stop patterns, then try taking a substring of each of those, checking that the start/end points are very close to the start/end stops.\n5. As a fallback, skip route stop patterns entirely. Find the starting/ending `Point`s; find the nearest point on the route for each of those points, and take the line between them.\n5. Get the time at which the vehicle leaves the start stop and at which it\n    arrives at the destination stop. Then linearly interpolate this along\n    every coordinate of the `LineString`. This way, the finalized\n    `LineString`s have the same geometry as the original routes, and every\n    coordinate has a time.\n\n```bash\n# Loop over _routes_\nnum_cpu=12\nfor i in {1..5}; do\n    cat data/routes_onestop_ids_${i}.txt \\\n        | parallel -P $num_cpu bash code/schedules/ssp_geom.sh {}\ndone\n```\n\nNow in `data/ssp/geom` I have a newline-delimited GeoJSON file for every route.\nI take all these individual features and cut them into individual tiles for a\nzoom that has all the original data with no simplification, which I currently\nhave as zoom 13.\n```bash\nrm -rf data/ssp/tiles\nmkdir -p data/ssp/tiles\nfind data/ssp/geom/ -type f -name 'r-*.geojson' -exec cat {} \\; \\\n    | uniq \\\n    | python code/tile/tile_geojson.py \\\n            `# Set minimum and maximum tile zooms` \\\n            -z 13 -Z 13 \\\n            `# Only keep LineStrings` \\\n            --allowed-geom-type 'LineString' \\\n            `# Write tiles into the following root dir` \\\n            -d data/ssp/tiles\n```\n\nCreate overview tiles for lower zooms\n```bash\npython code/tile/create_overview_tiles.py \\\n    --min-zoom 10 \\\n    --existing-zoom 13 \\\n    --tile-dir data/ssp/tiles \\\n    --max-coords 150000\n```\n\nMake gzipped protobuf files from these tiles:\n```bash\nrm -rf data_us/ssp/pbf\nmkdir -p data_us/ssp/pbf\nnum_cpu=15\nfor zoom in {10..13}; do\n    find data_us/ssp_geom_tiles/${zoom} -type f -name '*.geojson' \\\n        | parallel -P $num_cpu bash code/tile/compress_tiles_pbf.sh {}\ndone\n```\n\nUpload to AWS\n```bash\naws s3 cp \\\n    data/ssp/pbf/13 s3://data.kylebarron.dev/all-transit/pbfv2/schedule/4_16-20/13 \\\n    --recursive \\\n    --content-type application/x-protobuf \\\n    --content-encoding gzip \\\n    `# Set to public read access` \\\n    --acl public-read\n```\n\n## Feed Attribution\n\nSeveral data providers wish to be accredited when you use their data.\n\nDownload all feed information:\n\n```bash\ntransitland feeds --page-all \u003e data/feeds.geojson\npython code/generate_attribution.py data/feeds.geojson \\\n    | gzip \\\n    \u003e data/attribution.json.gz\naws s3 cp \\\n    data/attribution.json.gz \\\n    s3://data.kylebarron.dev/all-transit/attribution.json \\\n    --content-type application/json \\\n    --content-encoding gzip \\\n    --acl public-read\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylebarron%2Fall-transit","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkylebarron%2Fall-transit","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkylebarron%2Fall-transit/lists"}