{"id":15163348,"url":"https://github.com/costopoulos/realtime-transport-statistics","last_synced_at":"2026-02-28T12:05:24.537Z","repository":{"id":246083967,"uuid":"817916635","full_name":"Costopoulos/RealTime-Transport-Statistics","owner":"Costopoulos","description":"Real-time transport statistics application using data from Helsinki's Transportation Service API's MQTT broker","archived":false,"fork":false,"pushed_at":"2024-06-25T17:49:54.000Z","size":16873,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-02-10T05:33:13.518Z","etag":null,"topics":["common-table-expression","database-indexes","express","expressjs","mqtt","postgresql","transportation","trigger-functions"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Costopoulos.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-20T17:57:07.000Z","updated_at":"2024-06-25T17:49:58.000Z","dependencies_parsed_at":"2024-06-25T19:43:14.527Z","dependency_job_id":"c07a022d-7da6-4497-b12f-96b04c5b571e","html_url":"https://github.com/Costopoulos/RealTime-Transport-Statistics","commit_stats":null,"previous_names":["costopoulos/realtime-transport-statistics"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Costopoulos%2FRealTime-Transport-Statistics","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Costopoulos%2FRealTime-Transport-Statistics/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Costopoulos%2FRealTime-Transport-Statistics/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Costopoulos%2FRealTime-Transport-Statistics/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Costopoulos","download_url":"https://codeload.github.com/Costopoulos/RealTime-Transport-Statistics/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240035952,"owners_count":19737603,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["common-table-expression","database-indexes","express","expressjs","mqtt","postgresql","transportation","trigger-functions"],"created_at":"2024-09-27T02:23:42.227Z","updated_at":"2026-02-28T12:05:24.532Z","avatar_url":"https://github.com/Costopoulos.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RealTime-Transport-Statistics\nThis project provides a real-time transport statistics application using data from [Helsinki's Transportation Service API's MQTT broker](https://digitransit.fi/en/developers/apis/4-realtime-api/vehicle-positions/high-frequency-positioning/). The application ingests vehicle data, stores it in a PostgreSQL database, and provides endpoints to query the data.\n\n## Project Description\nThe task is defined in detail in the `dev-challenge-backend-transport-2024.pdf` file. Its endpoint-wise definition is as follows:\n\n### Ingest Data\n\n**POST** `/api/vehicles/ingest?duration={seconds}`\n\nIngests vehicle data from MQTT (after preprocessing them) for the specified duration (in seconds).\n\n### Get N Closest Vehicles\n\n**GET** `/api/vehicles/closest?latitude={lat}\u0026longitude={long}\u0026n={number}`\n\nRetrieves the N closest vehicles to the given latitude and longitude.\n\n### Get N Closest Unique Vehicles\n\n**GET** `/api/vehicles/unique-closest?latitude={lat}\u0026longitude={long}\u0026n={number}`\n\nRetrieves the N closest _distinct_ vehicles to the given latitude and longitude.\n\n### Get Average Speed Per Route\n\n**GET** `/api/vehicles/average-speed`\n\nCalculates the average speed of all vehicles for each route, **without the response time depending on the size of the data ingested so far.**\n\n### Get Metro Max Speeds Near Office\n\n**GET** `/api/vehicles/metro-speeds-timestamp`\n\nRetrieves the maximum speeds and the time since those speeds were achieved for metro vehicles near the office.\n\n### Get Unique Metro Max Speeds Near Office\n\n**GET** `/api/vehicles/unique-metro-speeds-timestamp`\n\nRetrieves the maximum speeds and the time since those speeds were achieved for _distinct_ metro vehicles near the office.\n\n## Design Process and Insights\n\n### Database Choice and Setup\nI chose PostgreSQL as the relational database for this project due to its high scalability and efficiency in handling large volumes of data. PostgreSQL's support for advanced indexing makes it an optimal choice for an application that manages data spanning multiple days.\n\nInitially, I set up a service to retrieve data from Digitransit's MQTT broker to inspect the incoming data. Based on the data structure, I designed the database schema and stored it in the `db/setup.sql` file, which I later refined as the project evolved. I connected to the `/hfp/v2/journey/#` topic by default to capture all relevant data without using multiple wildcard characters. The connection was then narrowed down to `ongoing` trips (`/hfp/v2/journey/ongoing/vp/#`) to align with the API's requirements for endpoints involving real-time data.\n\n### Schema Design\nKey fields from the `VP` event type (`desi` for route number, `veh` for vehicle number, `spd` for speed, `lat` and `long` for latitude and longitude, and `tst` for timestamp) were selected for the schema due to their relevance for the application's queries.\n\n### Error Handling in MQTT Messages\nFor error handling, I considered using `ON CONFLICT DO NOTHING` to ignore null values, but this approach would increase space and time complexity. Instead, I opted to drop messages if any of the critical fields (`desi`, `veh`, `spd`, `lat`, `long`, `tst`) were null or NaN. Using `==` for comparison allowed me to catch `null` and `NaN` values effectively. Notably, vehicle number `0` is considered valid (despite JavaScript \u0026 TypeScript considering it falsy), so it is ingested if all other fields are valid too.\n\n### Indexing Strategy\nIndexes were added to the following columns to optimize query performance:\n1. `timestamp`: For time-based queries.\n2. `route_number` and `vehicle_number`: For filtering and calculating statistics per route and vehicle.\n3. `latitude` and `longitude`: For finding the closest vehicles.\n\nI chose B-tree indexes for `timestamp`, `route_number`, and `vehicle_number` due to their suitability for high-cardinality (containing many unique values) columns. For geographical queries involving latitude and longitude, I selected `GiST` indexes over `SP-GiST` since `GiST` is well-suited for multi-dimensional data and supports distance queries, which are crucial for this application.\n\n### Justification for Index Choices\nIndexing has trade-offs, including:\n1. **Write Performance**: Index maintenance during insertions, updates, and deletions can slow down write operations.\n2. **Storage Space**: Each index requires additional storage.\n\nI did not index the `speed` column because it is primarily used for aggregation (calculating averages) rather than direct lookups or filtering. Indexing `speed` would be more appropriate if it were used in `WHERE` clauses, joins, or as sort keys.\n\n### Closest Vehicles Query\nFor the second endpoint (Get N Closest Vehicles), I encountered duplicate entries for the same vehicle at different timestamps by running the following query: \n```sql\nSELECT\n    route_number,\n    vehicle_number,\n    latitude,\n    longitude,\n    speed,\n    timestamp,\n    COUNT(*) AS duplicate_count\nFROM\n    vehicles\nGROUP BY\n    route_number,\n    vehicle_number,\n    latitude,\n    longitude,\n    speed,\n    timestamp\nHAVING COUNT(*) \u003e 1;\n\n```\nAdding a constraint on the fields (`route_number`, `vehicle_number`, `latitude`, `longitude`, `speed`) to prevent duplicates would not be feasible as it could exclude valid future entries, since public transportation vehicles follow the same routes and often travel at the same speeds. Instead, I thought of using `SELECT DISTINCT ON (vehicle_number)` to return unique vehicles. However, this approach can yield incorrect distances because `DISTINCT ON` selects the first row for each `vehicle_number` based on the order specified, not necessarily the closest distance.\n\nTo ensure the closest unique vehicles are retrieved, I implemented a `Common Table Expression (CTE)` to rank the distances and then select the closest entries. The `ranked_vehicles` CTE ranks each vehicle's distances using `ROW_NUMBER()` with a `PARTITION BY vehicle_number` clause, ensuring each vehicle is considered once based on the closest distance. The outer query selects rows with `rank = 1` and orders them by distance, ensuring the top `n` closest unique vehicles are returned.\n\n### Average Speed Per Route Endpoint\nFor the fourth endpoint (Get Average Speed Per Route), the response time must be independent of the total database size. There are several approaches to achieve this:\n1. **Summary (Aggregate) Table \u0026 Trigger Function with Triggers**\n2. **Materialized View \u0026 cronJob**\n3. **Materialized View \u0026 Trigger Function with Triggers**\n\nThe second solution, while easier, involves running the same query repeatedly, effectively caching the results, which is kinda cheating. The first solution stores average speed per route in a summary table, updated via triggers on insertions or updates. The third solution, which refreshes a materialized view, can be resource-intensive for large datasets.\n\nGiven the real-time nature of the data, the summary table approach is preferred. Updates to the summary table are incremental and immediate, reducing the need for costly full-table scans or materialized view refreshes. Using triggers ensures real-time updates without periodic refreshes. Indexing the `route_number` column in the summary table optimizes read performance, crucial for real-time data applications. The response time is independent of the data size since the summary table stores pre-aggregated data for each route.\n\n### Conclusion\nThis approach ensures efficient data ingestion, storage, and retrieval while maintaining performance. The design choices, particularly regarding indexing and data handling, optimize the application's ability to manage large datasets and provide real-time statistics effectively.\n\nThe correctness of the average speed per route endpoint was verified by cross-checking results with the following PostgreSQL query:\n```sql\nSELECT route_number, AVG(speed) AS average_speed\nFROM vehicles\nGROUP BY route_number;\n```\n\n### Metro Max Speeds Near Office Queries\n\nTo retrieve maximum speeds achieved by metro vehicles (routes `'M1'` and `'M2'`) near the office, two separate endpoints are implemented:\n\n1. **GET** `/api/vehicles/metro-speeds-timestamp`\n2. **GET** `/api/vehicles/unique-metro-speeds-timestamp`\n\nThe first endpoint returns all metro vehicle records near the office, including multiple entries per vehicle when it reached similar speeds at different timestamps or locations. It groups data by `vehicle_number`, `timestamp`, and coordinates, which can lead to duplicates in the result set because of the same metro vehicle having the same speed and distance at a slightly different timestamp (aka `milliseconds_ago`).\n\nTo address this, the second endpoint ensures uniqueness by using PostgreSQL’s `DISTINCT ON (vehicle_number)` with `ORDER BY speed DESC`. This guarantees only one row per metro vehicle—the one with its highest recorded speed—allowing for cleaner and more concise results.\n\nThis design choice offers flexibility:\n- The first query is suited for **granular time-based analysis** or **trend monitoring**.\n- The second is optimized for **real-time dashboards**, focusing on the **latest top performance per vehicle**.\n\nThe difference between them emphasizes the trade-off between **data completeness** and **result clarity**, especially when dealing with high-frequency, real-time geospatial data.\n\n## Application Installation\n1. **First make sure to install PostGIS**\n```\nsudo apt update\nsudo apt install postgresql-15-postgis-3\n```\n\n2. **Clone the repository:**\n```bash\ngit clone https://github.com/Costopoulos/RealTime-Transport-Statistics.git\ncd RealTime-Transport-Statistics\n```\n\n3. **Install requirements**\n```\nnpm install\n```\n\n4. **Configure the environment variables:**\n\nCreate a `.env` file in the root of the project and add the following environment variables:\n```env\nDB_USER=postgres\nDB_PASSWORD=yourpassword\nDB_HOST=localhost\nDB_PORT=5432\nDB_NAME=transport_db\n```\n\n5. **Set up the database:**\n\nMake sure PostgreSQL is running and create the database:\n```bash\npsql -U postgres -c \"CREATE DATABASE transport_db;\"\n```\n\nRun the SQL script to set up the tables and indexes:\n```bash\npsql -U postgres -d transport_db -f db/setup.sql\n```\n\nAlternatively, if you're not working in a Unix environment, you can manually copy paste the contents of `db/setup.sql` in the `transport_db` after having created the latter.\n\n## Running the App\n```\nnpm start\n```\n\n## Test\nTesting is performed using `Jest`, being configured in `jest.config.js`. `Mocking` as well as actual endpoint testing is included. The following command runs the tests while making sure to detect open handles.\n```\nnpm run test\n```\n\nThis is the result of `npm run test`:\n```\n\u003e realtime-transport-statistics@1.0.0 test\n\u003e jest --detectOpenHandles\n\n PASS  tests/controllers/vehicleController.test.ts\n  POST /api/vehicles/ingest\n    ✓ should fail with non-numeric duration (55 ms)\n  GET /api/vehicles/closest\n    ✓ should fail when latitude and longitude are missing (11 ms)\n    ✓ should return closest vehicles when lat and long are provided (10 ms)\n  GET /api/vehicles/average-speed\n    ✓ should return average speed per route (17 ms)\n  GET /api/vehicles/metro-max-speeds\n    ✓ should return max speeds near office (11 ms)\n\nTest Suites: 1 passed, 1 total\nTests:       5 passed, 5 total\nSnapshots:   0 total\nTime:        1.607 s, estimated 3 s\nRan all test suites.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcostopoulos%2Frealtime-transport-statistics","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcostopoulos%2Frealtime-transport-statistics","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcostopoulos%2Frealtime-transport-statistics/lists"}