{"id":28203350,"url":"https://github.com/tobilg/duckdb-gtfs","last_synced_at":"2026-02-13T15:32:37.014Z","repository":{"id":292385383,"uuid":"980645755","full_name":"tobilg/duckdb-gtfs","owner":"tobilg","description":"Loading and analyzing GTFS Schedule data with DuckDB","archived":false,"fork":false,"pushed_at":"2025-08-25T21:18:29.000Z","size":418,"stargazers_count":6,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-09-02T20:55:56.298Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tobilg.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-09T13:25:15.000Z","updated_at":"2025-08-25T21:18:33.000Z","dependencies_parsed_at":"2025-05-09T17:42:57.510Z","dependency_job_id":null,"html_url":"https://github.com/tobilg/duckdb-gtfs","commit_stats":null,"previous_names":["tobilg/duckdb-gtfs"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tobilg/duckdb-gtfs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fduckdb-gtfs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fduckdb-gtfs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fduckdb-gtfs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fduckdb-gtfs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tobilg","download_url":"https://codeload.github.com/tobilg/duckdb-gtfs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tobilg%2Fduckdb-gtfs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29411147,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-13T06:24:03.484Z","status":"ssl_error","status_checked_at":"2026-02-13T06:23:12.830Z","response_time":78,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-17T01:14:36.583Z","updated_at":"2026-02-13T15:32:37.006Z","avatar_url":"https://github.com/tobilg.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# duckdb-gtfs\nLoading and analyzing GTFS Schedule data with DuckDB.\n\n## GTFS Schedule DuckDB database\nThis projects provides an empty DuckDB database that contains the necessary tables and their relations for GTFS Schedule datasets. It can be found at [database/gtfs.duckdb](database/gtfs.duckdb).\n\nThe SQL script that created the DuckDB database can be found at [scripts/create_gtfs_database.sql](scripts/create_gtfs_database.sql). It's directly derived from the [official GTFS Schedule Field Definitions docs](https://gtfs.org/documentation/schedule/reference/#field-definitions).\n\nThe Entity Relationship Diagram looks like this:\n![GTFS Schedule ERD](docs/erd.png)\n\n## Loading an example dataset\nTo show the GTFS Schedule database can be used, this repo contains an example of how to transform and load a public dataset.\n\nThere’s a [German dataset](https://gtfs.de/de/feeds/de_full/) that contains the full train and local traffic for Germany. It contains more than 32 million stop times,  1.6 million trips, and 663 thousand stops.\n\n### Usage\nFirst, you need to clone this git repo locally. Then you can run the \n[scripts/providers/gtfs-de/full/download_data.sh](scripts/providers/gtfs-de/full/download_data.sh) script to download the data to the [source-data/providers/gtfs-de/full](source-data/providers/gtfs-de/full) directory.\n\nAfter that, you can load the data to a copy of the pre-existing GTFS database with the following command:\n\n```bash\nscripts/providers/gtfs-de/full/create_database.sh\n```\n\nThe database will be written to `exported-data/providers/gtfs-de/full/data.duckdb`.\n\nOnce this is done, you can open the DuckDB CLI to query the data:\n\n```bash\nduckdb exported-data/providers/gtfs-de/full/data.duckdb\n```\n\n### Creating a \"view database\"\nDuckDB can also be used to create a lightweight \"view database\" that contains only references to the actual data, hosted on object storage like S3 or R2.\n\nThe process to create such a \"view database\" consists of four main steps:\n\n1. Exporting the data from a database as Parquet files.\n2. Uploading the exported Parquet files to an object storage service.\n3. Creating the database file containing the views to the object storage files.\n4. Uploading the new database file to object storage as well, to be able to use the [ATTACH](https://duckdb.org/docs/stable/sql/statements/attach.html#attach) statement of DuckDB.\n\nWith our example dataset, if the above steps to create the basic database have been run, you can do the following:\n\n1. Run the `scripts/providers/gtfs-de/full/export_data.sh` script. This will export the data as one Parquet file per table.\n2. Upload the Parquet files to an object storage service.\n3. Edit the `scripts/providers/gtfs-de/full/create_view_database.sh` script to match the location of your object storage uploads, and run the script.\n4. Upload the resulting database file [exported-data/providers/gtfs-de/full/database.duckdb](exported-data/providers/gtfs-de/full/database.duckdb) to your object storage as well.\n\nThe resulting DuckDB database file will be around 250kb in size, and contains no actual data, but references to the data on object storage via views. \n\nYou can access the generated example database via [SQL Workbench](https://sql-workbench.com/#queries=v0,ATTACH-'https%3A%2F%2Fdata.openrailway.dev%2Fproviders%2Fgtfs%20de%2Ffull%2Fdatabase.duckdb'-as-gtfs_de_full-(READ_ONLY)~,SELECT-count(*)%3A%3AINT-AS-cnt-FROM-gtfs_de_full.stop_times~).\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fduckdb-gtfs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftobilg%2Fduckdb-gtfs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftobilg%2Fduckdb-gtfs/lists"}