{"id":13545954,"url":"https://github.com/public-transport/gtfs-utils","last_synced_at":"2025-04-08T01:32:54.418Z","repository":{"id":42233801,"uuid":"121556423","full_name":"public-transport/gtfs-utils","owner":"public-transport","description":"Read \u0026 analyze GTFS datasets using Node.js.","archived":false,"fork":false,"pushed_at":"2022-02-26T16:27:26.000Z","size":601,"stargazers_count":41,"open_issues_count":16,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-30T22:24:10.768Z","etag":null,"topics":["gtfs","public-transport","transit"],"latest_commit_sha":null,"homepage":"https://github.com/public-transport/gtfs-utils#gtfs-utils","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/public-transport.png","metadata":{"files":{"readme":"readme.md","changelog":null,"contributing":null,"funding":null,"license":"license.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-02-14T20:13:34.000Z","updated_at":"2025-02-19T14:41:45.000Z","dependencies_parsed_at":"2022-09-11T16:32:02.714Z","dependency_job_id":null,"html_url":"https://github.com/public-transport/gtfs-utils","commit_stats":null,"previous_names":[],"tags_count":34,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/public-transport%2Fgtfs-utils","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/public-transport%2Fgtfs-utils/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/public-transport%2Fgtfs-utils/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/public-transport%2Fgtfs-utils/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/public-transport","download_url":"https://codeload.github.com/public-transport/gtfs-utils/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247325692,"owners_count":20920714,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gtfs","public-transport","transit"],"created_at":"2024-08-01T12:00:28.758Z","updated_at":"2025-04-08T01:32:54.036Z","avatar_url":"https://github.com/public-transport.png","language":"JavaScript","funding_links":["https://github.com/sponsors/derhuerst"],"categories":["Frameworks and Libraries","Producing Data","Uncategorized"],"sub_categories":["GTFS","Uncategorized"],"readme":"# gtfs-utils\n\n**Utilities to process [GTFS](https://gtfs.org/reference/static/) data sets.**\n\n[![npm version](https://img.shields.io/npm/v/gtfs-utils.svg)](https://www.npmjs.com/package/gtfs-utils)\n![ISC-licensed](https://img.shields.io/github/license/public-transport/gtfs-utils.svg)\n![minimum Node.js version](https://img.shields.io/node/v/gtfs-utils.svg)\n[![support me via GitHub Sponsors](https://img.shields.io/badge/support%20me-donate-fa7664.svg)](https://github.com/sponsors/derhuerst)\n[![chat with me on Twitter](https://img.shields.io/badge/chat%20with%20me-on%20Twitter-1da1f2.svg)](https://twitter.com/derhuerst)\n\n- ✅ supports `frequencies.txt`\n- ✅ works in the browser\n- ✅ fully asynchronous/streaming\n\n\n## Design goals\n\n### streaming/iterative on sorted data\n\nAs [public transportation systems will hopefully become more integrated](https://github.com/public-transport/why-linked-open-transit-data#why-linked-open-transit-data) over time, GTFS datasets will often be multiple GBs large. GTFS processing should work in memory-constrained Raspberry Pis or [FaaS](https://en.wikipedia.org/wiki/Function_as_a_service) environments as well.\n\nWhenever possible, all `gtfs-utils` tools will only read as little data into memory as possible. For this, the individual files in a GTFS dataset need to be [sorted in a way](#sorted-gtfs-files) that allows iterative processing.\n\nRead more in the [*performance* section](#performance).\n\n### data-source-agnostic\n\n`gtfs-utils` does not make assumptions about where you read the GTFS data from. Although it has a built-in tool to read CSV from files on disk, anything is possible: [`.zip` archives](docs/zip.md), [HTTP requests](docs/fetch.md), in-memory [buffers](https://nodejs.org/api/buffer.html), [dat](https://dat.foundation)/[IPFS](https://ipfs.io), etc.\n\nThere are too many half-done, slightly opinionated GTFS processing tools out there, so `gtfs-utils` tries to be as universal as possible.\n\n### correct\n\nAside from new features of the ever-expanded GTFS spec that change the expected behavior of old ones (and bugs of course), `gtfs-utils` tries to follow the spec closely.\n\nFor example, it will, when computing the absolute timestamp/instant of an arrival at a stop, always take into account `stop_timezone` or the user-supplied timezone, because [`stop_times.txt` uses \"wall clock time\"](https://gist.github.com/derhuerst/574edc94981a21ef0ce90713f1cff7f6).\n\n\n## Installing\n\n```shell\nnpm install gtfs-utils\n```\n\n\n## Usage\n\n[API documentation](docs/api.md)\n\n### sorted GTFS files\n\n**`gtfs-utils` assumes that the files in your GTFS dataset are sorted in a particular way**; This allows it to compute some data aggregations more memory-efficiently, which means that you can use it to process [very large](#performance) datasets. For example, if [`trips.txt`](https://gtfs.org/reference/static/#tripstxt) and [`stop_times.txt`](https://gtfs.org/reference/static/#stop_timestxt) are both sorted by `trip_id`, `computeStopovers()` can read each file incrementally, only rows for *one* `trip_id` at a time.\n\n[Miller](https://miller.readthedocs.io/) and [`sponge`](https://linux.die.net/man/1/sponge) work very well for this:\n\n```shell\nmlr --csv sort -f agency_id agency.txt | sponge agency.txt\nmlr --csv sort -f parent_station -nr location_type stops.txt | sponge stops.txt\nmlr --csv sort -f route_id routes.txt | sponge routes.txt\nmlr --csv sort -f trip_id trips.txt | sponge trips.txt\nmlr --csv sort -f trip_id -n stop_sequence stop_times.txt | sponge stop_times.txt\nmlr --csv sort -f service_id calendar.txt | sponge calendar.txt\nmlr --csv sort -f service_id,date calendar_dates.txt | sponge calendar_dates.txt\nmlr --csv sort -f trip_id,start_time frequencies.txt | sponge frequencies.txt\n```\n\nThere's also a [`sort.sh` script](sort.sh) included in the npm package, which executes the commands above.\n\n*Note:* For read-only sources (like HTTP requests), sorting the files is not an option. You can solve this by [spawning](https://nodejs.org/docs/latest-v12.x/api/child_process.html#child_process_child_process_spawn_command_args_options) `mlr` and piping data through it.\n\n*Note:* With a bit of extra code, you can also use `gtfs-utils` [with a `.zip` archive](docs/zip.md) or [with a *remote* feed](docs/fetch.md).\n\n### basic example\n\nGiven our [sample GTFS dataset](https://npmjs.com/package/sample-gtfs-feed), we'll answer the following question: **On a specific day, which vehicles of which lines stop at a specific station?**\n\nWe define a function `readFile` that reads our GTFS data into a [readable stream](https://nodejs.org/api/stream.html#stream_readable_streams)/[async iterable](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator). In this case we'll read CSV files from disk using the built-in `readCsv` helper:\n\n```js\nconst readCsv = require('gtfs-utils/read-csv')\n\nconst readFile = (file) =\u003e {\n\treturn readCsv(require.resolve('sample-gtfs-feed/gtfs/' + file + '.txt'))\n}\n```\n\n[`computerStopovers()`](docs/api.md#computestopovers) will read [`calendar.txt`](https://gtfs.org/reference/static/#calendartxt), [`calendar_dates.txt`](https://gtfs.org/reference/static/#calendar_datestxt), [`trips.txt`](https://gtfs.org/reference/static/#tripstxt), [`stop_times.txt`](https://gtfs.org/reference/static/#stop_timestxt) \u0026 [`frequencies.txt`](https://gtfs.org/reference/static/#frequenciestxt) and return all *stopovers* of all trips across the full time frame of the dataset.\n\nIt returns an [async generator function](https://javascript.info/async-iterators-generators#async-generators) (which thus is [async-iterable](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator)), so we can use `for await`.\n\nIn the following example, we're going to print all stopovers at `airport` on the 5th of May 2019:\n\n```js\nconst {DateTime} = require('luxon')\nconst computeStopovers = require('gtfs-utils/compute-stopovers')\n\nconst day = '2019-05-15'\nconst isOnDay = (t) =\u003e {\n\tconst iso = DateTime.fromMillis(t * 1000, {zone: 'Europe/Berlin'}).toISO()\n\treturn String(t).slice(0, day.length) === day\n}\n\nconst stopovers = await computeStopovers(readFile, 'Europe/Berlin')\nfor await (const stopover of stopovers) {\n\tif (stopover.stop_id !== 'airport') continue\n\tif (!isOnDay(stopover.arrival)) continue\n\tconsole.log(stopover)\n}\n```\n\n```js\n{\n\tstop_id: 'airport',\n\ttrip_id: 'a-downtown-all-day',\n\tservice_id: 'all-day',\n\troute_id: 'A',\n\tstart_of_trip: 1557871200,\n\tarrival: 1557926580,\n\tdeparture: 1557926640,\n}\n{\n\tstop_id: 'airport',\n\ttrip_id: 'a-outbound-all-day',\n\tservice_id: 'all-day',\n\troute_id: 'A',\n\tstart_of_trip: 1557871200,\n\tarrival: 1557933900,\n\tdeparture: 1557933960,\n}\n// …\n{\n\tstop_id: 'airport',\n\ttrip_id: 'c-downtown-all-day',\n\tservice_id: 'all-day',\n\troute_id: 'C',\n\tstart_of_trip: 1557871200,\n\tarrival: 1557926820,\n\tdeparture: 1557926880,\n}\n```\n\nFor more examples, check the [API documentation](docs/api.md).\n\n\n## Performance\n\nBy default, `gtfs-utils` verifies that the input files are sorted correctly. You can disable this to improve performance slightly by running with the `CHECK_GTFS_SORTING=false` environment variable.\n\n`gtfs-utils` should be fast enough for small to medium-sized GTFS datasets. It won't be as fast as other GTFS tools because it\n\n- uses [async iteration](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Symbol/asyncIterator) extensively for memory-efficiency and an easy-of-use, which [currently has significant performance penalties in v8](https://github.com/nodejs/node/issues/31979).\n- is written in JavaScript, so it cannot optimise the memory layout of its data structures.\n- parses all columns of a file it needs information from, into a JavaScript object.\n\nOn my [M1 Macbook Air](https://everymac.com/systems/apple/macbook-air/specs/macbook-air-m1-8-core-7-core-gpu-13-retina-display-2020-specs.html), with the [180mb `2022-02-03` *HVV* GTFS dataset](https://suche.transparenz.hamburg.de/dataset/hvv-fahrplandaten-gtfs-februar-2022-bis-dezember-2022) (17k `stops.txt` rows, 91k `trips.txt` rows, 2m `stop_times.txt` rows, ~500m stopovers), `computeStopovers` computes 18k stopovers per second, and finishes in several hours.\n\n*Note:* If you want a faster way to query and transform GTFS datasets, I suggest you to use [`gtfs-via-postgres`](https://github.com/derhuerst/gtfs-via-postgres) to leverage PostgreSQL's query optimizer. Once you have imported the data, it is usually orders of magnitude faster.\n\n\n## Related\n\n- [gtfstidy](https://github.com/patrickbr/gtfstidy) – Go command line tool for validating and tidying GTFS feeds.\n- [gtfs-stream](https://github.com/staeco/gtfs-stream) – Streaming GTFS and GTFS-RT parser for node\n- [mapzen-gtfs](https://github.com/transitland/mapzen-gtfs) – Python library for reading and writing GTFS feeds. (Python)\n- [gtfspy](https://github.com/CxAalto/gtfspy) – Public transport network analysis using Python\n- [extract-gtfs-shapes](https://github.com/derhuerst/extract-gtfs-shapes) – Command-line tool to extract shapes from a GTFS dataset.\n- [extract-gtfs-pathways](https://github.com/derhuerst/extract-gtfs-pathways) – Command-line tool to extract pathways from a GTFS dataset.\n- [Awesome GTFS: Frameworks and Libraries](https://github.com/andredarcie/awesome-gtfs#frameworks-and-libraries) – A collection of libraries for working with GTFS.\n\n## Contributing\n\nIf you have a question or have difficulties using `gtfs-utils`, please double-check your code and setup first. If you think you have found a bug or want to propose a feature, refer to [the issues page](https://github.com/public-transport/gtfs-utils/issues).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpublic-transport%2Fgtfs-utils","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpublic-transport%2Fgtfs-utils","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpublic-transport%2Fgtfs-utils/lists"}