{"id":26982697,"url":"https://github.com/meakbiyik/routeformer","last_synced_at":"2026-01-19T14:01:51.968Z","repository":{"id":284257913,"uuid":"954302062","full_name":"meakbiyik/routeformer","owner":"meakbiyik","description":"Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction","archived":false,"fork":false,"pushed_at":"2025-07-07T21:17:19.000Z","size":69725,"stargazers_count":7,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-09-05T07:43:53.495Z","etag":null,"topics":["autonomous-driving","computer-vision"],"latest_commit_sha":null,"homepage":"https://meakbiyik.com/routeformer","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/meakbiyik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-24T21:51:38.000Z","updated_at":"2025-08-01T11:02:09.000Z","dependencies_parsed_at":"2025-09-05T07:39:11.343Z","dependency_job_id":"fb3f2121-dc00-4872-9a3e-cdf05ede3d34","html_url":"https://github.com/meakbiyik/routeformer","commit_stats":null,"previous_names":["meakbiyik/routeformer"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/meakbiyik/routeformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meakbiyik%2Frouteformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meakbiyik%2Frouteformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meakbiyik%2Frouteformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meakbiyik%2Frouteformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/meakbiyik","download_url":"https://codeload.github.com/meakbiyik/routeformer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/meakbiyik%2Frouteformer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28571556,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-19T12:50:50.164Z","status":"ssl_error","status_checked_at":"2026-01-19T12:50:42.704Z","response_time":67,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","computer-vision"],"created_at":"2025-04-03T16:34:34.050Z","updated_at":"2026-01-19T14:01:51.956Z","avatar_url":"https://github.com/meakbiyik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Routeformer: Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction\n\n\u003cdiv align=\"center\"\u003e\n\n[![Website](docs/imgs/badges/badge_project_page.svg)](https://meakbiyik.com/routeformer/)\n[![Paper](docs/imgs/badges/badge_pdf.svg)](https://arxiv.org/abs/2312.08558)\n[![Dataset](docs/imgs/badges/badge_dataset.svg)](https://huggingface.co/datasets/meakbiyik/GEM_gaze-assisted-ego-motion-in-driving)\n\n\u003c/div\u003e\n\nThis repository will host the code and supplementary materials for our paper **\"Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction\"** accepted at ICLR 2025. It includes the implementation of our novel multimodal ego-trajectory prediction network, **Routeformer**, and the GEM dataset.\n\n\u003c!-- limit to max-width of 400px --\u003e\n\n\u003cdiv align=\"center\"\u003e\n\u003cimg src=\"docs/imgs/model.png\" alt=\"Figure\" style=\"max-width:auto; max-height:300px;\" /\u003e\n\u003c/div\u003e\n\n## Overview\n\nUnderstanding drivers' decision-making is crucial for road safety. While predicting an ego-vehicle’s path is important for driver-assistance systems, most existing methods focus primarily on external factors like other vehicles' motions. Our work addresses this limitation by integrating the driver's attention with the surrounding scene, combining GPS data, environmental context, and driver field-of-view information (first-person video and gaze fixations).\n\nIn this repository, you will eventually find:\n\n- **Code:** The implementation of Routeformer and associated tools.\n- **GEM Dataset:** A comprehensive dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data. The link to the GEM dataset will be provided once available.\n\n## Getting Started\n\n### Installation\n\n1. Clone the repository:\n\n    ```bash\n    git clone https://github.com/meakbiyik/routeformer.git\n    cd routeformer\n    ```\n\n2. Install the dependencies using [Poetry](https://python-poetry.org/):\n\n    ```bash\n    poetry install\n    ```\n\n    **Note on the `av` dependency:** This project uses the `av` library for video processing, which has `ffmpeg` as a dependency. If you have `ffmpeg` already installed on your system, you might encounter issues with the default installation. In that case, it is recommended to install `av` with the following command to avoid building it from source:\n\n    ```bash\n    pip install av --no-binary av\n    ```\n\n## Repository Structure\n\nHere's a brief overview of the most important files and directories:\n\n- `routeformer/models/routeformer.py`: This file contains the core implementation of the **Routeformer** model.\n- `experiments/full_comparison.py`: This is the main script to run the experiments and reproduce the results from the paper.\n- `routeformer/io/dataset.py`: Contains the dataset loading and processing logic.\n- `docs/`: Contains additional documentation, including details on the dataset and data extraction.\n\n## Abstract\n\nUnderstanding drivers' decision-making is crucial for road safety. Although predicting the ego-vehicle's path is valuable for driver-assistance systems, existing methods mainly focus on external factors like other vehicles' motions, often neglecting the driver's attention and intent. To address this gap, we infer the ego-trajectory by integrating the driver's attention and the surrounding scene. We introduce Routeformer, a novel multimodal ego-trajectory prediction network combining GPS data, environmental context, and driver field-of-view—comprising first-person video and gaze fixations. We also present the Path Complexity Index (PCI), a new metric for trajectory complexity that enables a more nuanced evaluation of challenging scenarios. To tackle data scarcity and enhance diversity, we introduce GEM, a comprehensive dataset of urban driving scenarios enriched with synchronized driver field-of-view and gaze data. Extensive evaluations on GEM and DR(eye)VE demonstrate that Routeformer significantly outperforms state-of-the-art methods, achieving notable improvements in prediction accuracy across diverse conditions. Ablation studies reveal that incorporating driver field-of-view data yields significantly better average displacement error, especially in challenging scenarios with high PCI scores, underscoring the importance of modeling driver attention. All data, code, and models will be made publicly available.\n\n## Citation\n\nIf you use our work, please consider citing our paper:\n\n```bibtex\n@inproceedings{akbiyik2023routeformer,\n    title={Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction},\n    author={M. Eren Akbiyik, Nedko Savov, Danda Pani Paudel, Nikola Popovic, Christian Vater, Otmar Hilliges, Luc Van Gool, Xi Wang},\n    booktitle={International Conference on Learning Representations},\n    year={2025}\n}\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeakbiyik%2Frouteformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmeakbiyik%2Frouteformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmeakbiyik%2Frouteformer/lists"}