{"id":37637463,"url":"https://github.com/vita-epfl/social-transmotion","last_synced_at":"2026-01-16T11:11:16.124Z","repository":{"id":214232031,"uuid":"735619221","full_name":"vita-epfl/social-transmotion","owner":"vita-epfl","description":"[ICLR 2024] Official implementation of \"Social-Transmotion: Promptable Human Trajectory Prediction\" in PyTorch.","archived":false,"fork":false,"pushed_at":"2024-04-13T16:24:46.000Z","size":226,"stargazers_count":16,"open_issues_count":1,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-14T06:53:15.434Z","etag":null,"topics":["attention-mechanism","deep-learning","human-trajectory-prediction","prompt"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vita-epfl.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-12-25T15:12:40.000Z","updated_at":"2024-04-15T11:14:15.276Z","dependencies_parsed_at":null,"dependency_job_id":"8dab079c-5c75-423b-818e-8c18abd6b624","html_url":"https://github.com/vita-epfl/social-transmotion","commit_stats":null,"previous_names":["vita-epfl/social-transmotion"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/vita-epfl/social-transmotion","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vita-epfl%2Fsocial-transmotion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vita-epfl%2Fsocial-transmotion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vita-epfl%2Fsocial-transmotion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vita-epfl%2Fsocial-transmotion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vita-epfl","download_url":"https://codeload.github.com/vita-epfl/social-transmotion/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vita-epfl%2Fsocial-transmotion/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28478134,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-16T06:30:42.265Z","status":"ssl_error","status_checked_at":"2026-01-16T06:30:16.248Z","response_time":107,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","deep-learning","human-trajectory-prediction","prompt"],"created_at":"2026-01-16T11:11:16.049Z","updated_at":"2026-01-16T11:11:16.106Z","avatar_url":"https://github.com/vita-epfl.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\u003ch1\u003e Social-Transmotion:\u003cbr\u003e  Promptable Human Trajectory Prediction \u003c/h1\u003e\n\u003ch3\u003eSaeed Saadatnejad*, Yang Gao*, Kaouther Messaoud, Alexandre Alahi\n\u003c/h3\u003e\n\u003ch4\u003e \u003ci\u003e International Conference on Learning Representations (ICLR), Austria, May 2024 \u003c/i\u003e\u003c/h4\u003e\n\n[[Paper](https://arxiv.org/abs/2312.16168)] [[ICLR page](https://iclr.cc/virtual/2024/poster/18604)] [[Poster](docs/Poster.pdf)] [[Slides](docs/iclr_slides.pdf)]\n\n\n\u003cimage src=\"docs/social-transmotion.png\" width=\"500\"\u003e\n\n\u003c/div\u003e\n\n\u003cdiv align=\"center\"\u003e \u003ch3\u003e Abstract \u003c/h3\u003e  \u003c/div\u003e\n\u003cdiv align=\"justify\"\u003e\n\nAccurate human trajectory prediction is crucial for applications such as autonomous vehicles, robotics, and surveillance systems. Yet, existing models often fail to fully leverage the non-verbal social cues human subconsciously communicate when navigating the space. To address this, we introduce Social-Transmotion, a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. We translate the idea of a prompt from Natural Language Processing (NLP) to the task of human trajectory prediction, where a prompt can be a sequence of x-y coordinates on the ground, bounding boxes in the image plane, or body pose keypoints in either 2D or 3D. This, in turn, augments trajectory data, leading to enhanced human trajectory prediction. Using masking technique, our model exhibits flexibility and adaptability by capturing spatiotemporal interactions between agents based on the available visual cues. We delve into the merits of using 2D versus 3D poses, and a limited set of poses. Additionally, we investigate the spatial and temporal attention map to identify which keypoints and time-steps in the sequence are vital for optimizing human trajectory prediction. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.\n\u003c/br\u003e\n\n\n# Getting Started\n\nInstall the requirements using `pip`:\n```\npip install -r requirements.txt\n```\n\nWe have conveniently added the preprocessed data to the release section of the repository (for license details, please refer to the original papers).\nPlace the data subdirectory of JTA under `data/jta_all_visual_cues` and the data subdirectory of JRDB under `data/jrdb_2dbox` of the repository.\n\n# Training and Testing\n\n## JTA dataset\nYou can train the Social-Transmotion model on this dataset using the following command:\n```\npython train_jta.py --cfg configs/jta_all_visual_cues.yaml --exp_name jta\n```\n\n\nTo evaluate the trained model, use the following command:\n```\npython evaluate_jta.py --ckpt ./experiments/jta/checkpoints/checkpoint.pth.tar --metric ade_fde --modality traj+all\n```\nPlease note that the evaluation modality can be any of `[traj, traj+2dbox, traj+3dpose, traj+2dpose, traj+3dpose+3dbox, traj+all]`.\nFor the ease of use, we have also provided the trained model in the release section of this repo. In order to use that, you should pass the address of the saved checkpoint via `--ckpt`.\n\n## JRDB dataset\nYou can train the Social-Transmotion model on this dataset using the following command:\n```\npython train_jrdb.py --cfg configs/jrdb_2dbox.yaml --exp_name jrdb\n```\n\nTo evaluate the trained model, use the following command:\n```\npython evaluate_jrdb.py --ckpt ./experiments/jrdb/checkpoints/checkpoint.pth.tar --metric ade_fde --modality traj+2dbox\n```\nPlease note that the evaluation modality can be one any of `[traj, traj+2dbox]`.\nFor the ease of use, we have also provided the trained model in the release section of this repo. In order to use that, you should pass the address of the saved checkpoint via `--ckpt`.\n\n# Work in Progress\n\nThis repository is work-in-progress and will continue to get updated and improved over the coming months.\n\n```\n@InProceedings{saadatnejad2024socialtransmotion,\n      title={Social-Transmotion: Promptable Human Trajectory Prediction}, \n      author={Saadatnejad, Saeed and Gao, Yang and Messaoud, Kaouther and Alahi, Alexandre},\n      booktitle={International Conference on Learning Representations (ICLR)},\n      year={2024},\n}\n```\n\nFeel free to check the follow-up work, Multi-Transmotion, a pre-trained model for multimodal multitask human motion prediction.\n```\n@inproceedings{gao2024multi,\n    title={Multi-Transmotion: Pre-trained Model for Human Motion Prediction},\n    author={Gao, Yang and Luan, Po-Chien and Alahi, Alexandre},\n    booktitle={8th Annual Conference on Robot Learning},\n    year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-epfl%2Fsocial-transmotion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvita-epfl%2Fsocial-transmotion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvita-epfl%2Fsocial-transmotion/lists"}