{"id":31944406,"url":"https://github.com/ehsanik/touchtorch","last_synced_at":"2025-10-14T10:24:20.147Z","repository":{"id":44193946,"uuid":"254711282","full_name":"ehsanik/touchTorch","owner":"ehsanik","description":"Use the Force Luke! Learning to Predict Physical Forces by Simulating Effects [CVPR2020] (https://arxiv.org/pdf/2003.12045.pdf)","archived":false,"fork":false,"pushed_at":"2023-07-23T11:33:41.000Z","size":426,"stargazers_count":64,"open_issues_count":7,"forks_count":12,"subscribers_count":2,"default_branch":"master","last_synced_at":"2023-08-12T17:07:49.139Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ehsanik.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-10T18:49:43.000Z","updated_at":"2023-06-06T07:30:49.000Z","dependencies_parsed_at":"2023-01-21T19:46:23.416Z","dependency_job_id":null,"html_url":"https://github.com/ehsanik/touchTorch","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/ehsanik/touchTorch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FtouchTorch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FtouchTorch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FtouchTorch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FtouchTorch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ehsanik","download_url":"https://codeload.github.com/ehsanik/touchTorch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ehsanik%2FtouchTorch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018782,"owners_count":26086452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T10:24:14.736Z","updated_at":"2025-10-14T10:24:20.141Z","avatar_url":"https://github.com/ehsanik.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# [Use the Force Luke! Learning to Predict Physical Forces by Simulating Effects](https://arxiv.org/pdf/2003.12045.pdf)\n## K Ehsani, S Tulsiani, S Gupta, A Farhadi, A Gupta\nThis project is an oral presentation at CVPR2020.\n\n[(Project Page)](https://ehsanik.github.io/forcecvpr2020/) [(PDF)](https://arxiv.org/pdf/2003.12045.pdf) [(Slides)](https://github.com/ehsanik/forcecvpr2020/blob/master/img/3095-talk.pdf) [(Video)](https://ehsanik.github.io/forcecvpr2020/#slide_video)\n\n\u003ccenter\u003e\u003cimg src=\"figs/teaser_force.jpg\" height=\"300px\" \u003e\u003c/center\u003e\n\n### Abstract\n\nWhen we humans look at a video of human-object interaction, we can not only infer what is happening but we can even extract actionable information and imitate those interactions. On the other hand, current recognition or geometric approaches lack the physicality of action representation. In this paper, we take a step towards a more physical understanding of actions. We address the problem of inferring contact points and the physical forces from videos of humans interacting with objects. One of the main challenges in tackling this problem is obtaining ground-truth labels for forces. We sidestep this problem by instead using a physics simulator for supervision. Specifically, we use a simulator to predict effects and enforce that estimated forces must lead to the same effect as depicted in the video.\n\nOur quantitative and qualitative results show that:\n\u003col\u003e\n    \u003cli\u003eWe can predict meaningful forces from videos whose effects lead to accurate imitation of the motions observe.\u003c/li\u003e\n    \u003cli\u003eBy jointly optimizing for contact point and force prediction, we can improve the performance on both tasks in comparison to independent training. \u003c/li\u003e\n    \u003cli\u003eWe can learn a representation from this model that generalizes to novel objects using few shot examples. \u003c/li\u003e\n\u003c/ol\u003e\n\n\n\n### Installation\n1. Install requirements:\n\n```\npip3 install -r requirements.txt\n```\n\n2. Clone the repository using the command:\n\n```\ngit clone https://github.com/ehsanik/touchTorch\ncd touchTorch\n```\n\n3. Download the dataset from [here](https://drive.google.com/file/d/1fs-L0PSxHhY22Q-SBHG-bnmdUWTR90-c/view?usp=sharing) and extract it to DatasetForce.\n4. Download pretrained weights from [here](https://drive.google.com/file/d/1JNTkWJX0EmdzXx6A18A7lFq1tiDuLlgE/view?usp=sharing) and extract it to \"DatasetForce/trained_weights\". \n\n\n### Dataset\n\nDataset statistics is provided in the paper. The structure of the dataset is as follows:\n\n```\nDatasetForce\n└── images\n│   └── OBJECTNAME\n│       └── image_*.jpeg\n└── annotations\n│   ├── [test/train/val]_time_to_clip_ind.json\n│   ├── [test/train/val]_cleaned_start_states.json\n│   ├── time_to_keypoint_fps_{FPS}.json\n│   ├── time_to_obj_state_fps_{FPS}.json\n│   └── clean_clip_to_contact_point.json\n└── trained_weights\n│   └── *.pytar\n└── objects_16k  -----  YCB objects modified for PyBullet\n    └── OBJECTNAME\n        └── google_16k\n            ├── textured.urdf  -----  URDF to load in PyBullet\n            ├── textured_big.obj  -----  Object mesh\n            ├── textured_big.obj.mtl  -----  Corresponding texture\n            ├── textured_bigconvex.obj  -----  Simplified mesh for collision \n            │                                   (Calculated using [v-hacd])\n            └── *_keypoints.json  -----  Keypoints on the object mesh\n```\n### Train\n\nTo train your own model:\n\n```\npython3 main.py --title joint_training --sequence_length 10 --gpu-ids 0 --number_of_cp 5 \\\n   --model SeparateTowerModel --dataset DatasetWAugmentation --loss KPProjectionCPPredictionLoss \\ \n   --object_list ALL --data DatasetForce\n```\n\nSee `scripts/train_scripts.sh` for additional training scripts.\n\n### Test\n\nTo test using the pretrained model and reproduce the results in the paper:\n\n```\npython3 main.py --title test_all_joint_training --sequence_length 10 --gpu-ids 0 \\ \n   --number_of_cp 5 --model SeparateTowerModel --dataset DatasetWAugmentation \\\n   --loss KPProjectionCPPredictionLoss --object_list ALL --data DatasetForce \\\n   --reload DatasetForce/trained_weights/all_obj_end2end.pytar test\n```\n\nSee `scripts/test_w_weights.sh` for additional scripts.\n\n\n\n### Citation\n\nIf you find this project useful in your research, please consider citing:\n\n```\n   @inproceedings{ehsani2020force,\n     title={Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects},\n      author={Ehsani, Kiana and Tulsiani, Shubham and Gupta, Saurabh and Farhadi, Ali and Gupta, Abhinav},\n     booktitle={CVPR},\n     year={2020}\n   }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fehsanik%2Ftouchtorch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fehsanik%2Ftouchtorch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fehsanik%2Ftouchtorch/lists"}