{"id":16589844,"url":"https://github.com/kabouzeid/jointtransformer","last_synced_at":"2025-03-23T14:31:13.707Z","repository":{"id":225689437,"uuid":"672863980","full_name":"kabouzeid/JointTransformer","owner":"kabouzeid","description":"JointTransformer: Winner of the HANDS'2023 ARCTIC Challenge @ ICCV","archived":false,"fork":false,"pushed_at":"2024-08-06T12:45:55.000Z","size":15881,"stargazers_count":11,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-16T00:27:16.710Z","etag":null,"topics":["handpose","transformer"],"latest_commit_sha":null,"homepage":"https://arctic-leaderboard.is.tuebingen.mpg.de/leaderboard","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kabouzeid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-07-31T10:41:14.000Z","updated_at":"2025-01-07T02:48:13.000Z","dependencies_parsed_at":"2024-08-06T14:47:33.630Z","dependency_job_id":"63dac7bd-ebda-49eb-bd5d-89fcbc1a538e","html_url":"https://github.com/kabouzeid/JointTransformer","commit_stats":null,"previous_names":["kabouzeid/jointtransformer"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2FJointTransformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2FJointTransformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2FJointTransformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kabouzeid%2FJointTransformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kabouzeid","download_url":"https://codeload.github.com/kabouzeid/JointTransformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245115729,"owners_count":20563220,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["handpose","transformer"],"created_at":"2024-10-11T23:10:04.442Z","updated_at":"2025-03-23T14:31:08.681Z","avatar_url":"https://github.com/kabouzeid.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JointTransformer\n\nWinner of the HANDS'2023 ARCTIC Challenge @ ICCV (1st place in both p1 and p2 tasks).\n\n[Arctic Leaderboard](https://arctic-leaderboard.is.tuebingen.mpg.de/leaderboard).\n\n\n## Installation\n\n### 1. Dependencies\n\n- Python 3.10.4\n- CUDA 11.8\n\n```bash\npip install -U pip wheel\npip install torch torchvision -c requirements_frozen.txt --extra-index-url https://download.pytorch.org/whl/cu118\npip install -r requirements_frozen.txt\n```\n\n### 2. Patch `smplx`\n\nModify `smplx` package to return 21 joints for instead of 16:\n\n```bash\nvim .venv/lib/python3.10/site-packages/smplx/body_models.py\n```\n\nUncomment line 1681: `joints = self.vertex_joint_selector(vertices, joints)`.\n\nIf you are unsure about where `body_models.py` is, run these on a terminal:\n\n```bash\npython\n\u003e\u003e\u003e import smplx\n\u003e\u003e\u003e print(smplx.__file__)\n```\n\n### 3. Dataset\n\nRefer to the original [README](#original-readme) to setup the dataset.\n\n## Training\n\nFor the allocentric task (p1) we used this command:\n\n```bash\npython scripts_method/train.py --setup p1 --method transformer_sf --trainsplit train --valsplit smallval --optimizer adamw --lr_scheduler cosine --lr 1e-4 --backbone vit-g --freeze_backbone --decoder_depth=12\n```\n[[`p1 checkpoint (504802f70)`](https://omnomnom.vision.rwth-aachen.de/data/jointtransformer/final_p1_504802f70.ckpt)]\n\nFor the egocentric task (p2) we used this command (where `504802f70` is the checkpoint from the p1 run):\n\n```bash\npython scripts_method/train.py --setup p2 --method transformer_sf --trainsplit train --valsplit smallval --optimizer adamw --lr_scheduler cosine --lr 1e-4 --backbone vit-g --freeze_backbone --decoder_depth=12 --load_ckpt logs/504802f70/checkpoints/last.ckpt --precision 32\n```\n[[`p2 checkpoint (462fc683e)`](https://omnomnom.vision.rwth-aachen.de/data/jointtransformer/final_p2_462fc683e.ckpt)]\n\n---\n\n# Original README\n\n## ARCTIC 🥶: A Dataset for Dexterous Bimanual Hand-Object Manipulation\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"docs/static/arctic-logo.svg\" alt=\"Image\" width=\"600\" height=\"100\" /\u003e\n\u003c/p\u003e\n\n[ [Project Page](https://arctic.is.tue.mpg.de) ]\n[ [Paper](https://download.is.tue.mpg.de/arctic/arctic_april_24.pdf) ]\n[ [Video](https://www.youtube.com/watch?v=bvMm8gfFbZ8) ]\n[ [Register ARCTIC Account](https://arctic.is.tue.mpg.de/register.php) ]\n[ [ICCV Competition](https://sites.google.com/view/hands2023/home) ]\n[ [Leaderboard](docs/leaderboard.md) ]\n\n\n\u003cp align=\"center\"\u003e    \n    \u003cimg src=\"docs/static/teaser.jpeg\" alt=\"Image\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\n\nThis is a repository for preprocessing, splitting, visualizing, and rendering (RGB, depth, segmentation masks) the ARCTIC dataset.\nFurther, here, we provide code to reproduce our baseline models in our CVPR 2023 paper (Vancouver, British Columbia 🇨🇦) and developing custom models.\n\nOur dataset contains heavily dexterous motion:\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./docs/static/dexterous.gif\" alt=\"Image\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\n### Why use ARCTIC?\n\nSummary on dataset:\n- It contains 2.1M high-resolution images paired with annotated frames, enabling large-scale machine learning.\n- Images are from 8x 3rd-person views and 1x egocentric view (for mixed-reality setting).\n- It includes 3D groundtruth for SMPL-X, MANO, articulated objects.\n- It is captured in a MoCap setup using 54 high-end Vicon cameras.\n- It features highly dexterous bimanual manipulation motion (beyond quasi-static grasping).\n\nPotential tasks with ARCTIC:\n- Generating [hand grasp](https://korrawe.github.io/HALO/HALO.html) or [motion](https://github.com/cghezhang/ManipNet) with articulated objects\n- Generating [full-body grasp](https://grab.is.tue.mpg.de/) or [motion](https://goal.is.tue.mpg.de/) with articulated objects\n- Benchmarking performance of articulated object pose estimators from [depth images](https://articulated-pose.github.io/) with human in the scene\n- Studying our [NEW tasks](https://download.is.tue.mpg.de/arctic/arctic_april_24.pdf) of consistent motion reconstruction and interaction field estimation\n- Studying egocentric hand-object reconstruction\n- Reconstructing [full-body with hands and articulated objects](https://3dlg-hcvc.github.io/3dhoi/) from RGB images\n\n\nCheck out our [project page](https://arctic.is.tue.mpg.de) for more details.\n\n### News\n\n- 2023.09.11: [ARCTIC leaderboard](https://arctic-leaderboard.is.tuebingen.mpg.de/) online!\n- 2023.06.16: ICCV ARCTIC [challenge](https://sites.google.com/view/hands2023/home) starts!\n- 2023.05.04: ARCTIC dataset with code for dataloaders, visualizers, models is officially announced (version 1.0)! \n- 2023.03.25: ARCTIC ☃️ dataset (version 0.1) is available! 🎉\n\nInvited talks/posters at CVPR2023:\n- [4D-HOI workshop: Keynote](https://4dhoi.github.io/)\n- [Ego4D + EPIC workshop: Oral presentation](https://ego4d-data.org/workshops/cvpr23)\n- [Rhobin workshop: Poster](https://rhobin-challenge.github.io/schedule.html)\n- [3D scene understanding: Oral presentation](https://scene-understanding.com)\n\n### Features\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"./docs/static/viewer_demo.gif\" alt=\"Image\" width=\"80%\"/\u003e\n\u003c/p\u003e\n\n- Instructions to download the ARCTIC dataset.\n- Scripts to process our dataset and to build data splits.\n- Rendering scripts to render our 3D data into RGB, depth, and segmentation masks.\n- A viewer to interact with our dataset.\n- Instructions to setup data, code, and environment to train our baselines.\n- A generalized codebase to train, visualize and evaluate the results of ArcticNet and InterField for the ARCTIC benchmark.\n- A viewer to interact with the prediction.\n\n### Getting started\n\nGet a copy of the code:\n```bash\ngit clone https://github.com/zc-alexfan/arctic.git\n```\n\n- Setup environment: see [`docs/setup.md`](docs/setup.md)\n- Download and visualize ARCTIC dataset: see [`docs/data/README.md`](docs/data/README.md)\n- Training, evaluating for our ARCTIC baselines: see [`docs/model/README.md`](docs/model/README.md).\n- Evaluation on test set: see [`docs/leaderboard.md`](docs/leaderboard.md)\n- FAQ: see [`docs/faq.md`](docs/faq.md)\n\n### License\n\nSee [LICENSE](LICENSE).\n\n### Citation\n\n```bibtex\n@inproceedings{fan2023arctic,\n  title = {{ARCTIC}: A Dataset for Dexterous Bimanual Hand-Object Manipulation},\n  author = {Fan, Zicong and Taheri, Omid and Tzionas, Dimitrios and Kocabas, Muhammed and Kaufmann, Manuel and Black, Michael J. and Hilliges, Otmar},\n  booktitle = {Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},\n  year = {2023}\n}\n```\n\nOur paper benefits a lot from [aitviewer](https://github.com/eth-ait/aitviewer). If you find our viewer useful, to appreciate their hard work, consider citing:\n\n```bibtex\n@software{kaufmann_vechev_aitviewer_2022,\n  author = {Kaufmann, Manuel and Vechev, Velko and Mylonopoulos, Dario},\n  doi = {10.5281/zenodo.1234},\n  month = {7},\n  title = {{aitviewer}},\n  url = {https://github.com/eth-ait/aitviewer},\n  year = {2022}\n}\n```\n\n### Acknowledgments\n\nConstructing the ARCTIC dataset is a huge effort. The authors deeply thank: [Tsvetelina Alexiadis (TA)](https://ps.is.mpg.de/person/talexiadis) for trial coordination; [Markus Höschle (MH)](https://ps.is.mpg.de/person/mhoeschle), [Senya Polikovsky](https://is.mpg.de/person/senya), [Matvey Safroshkin](https://is.mpg.de/person/msafroshkin), [Tobias Bauch (TB)](https://www.linkedin.com/in/tobiasbauch/?originalSubdomain=de) for the capture setup; MH, TA and [Galina Henz](https://ps.is.mpg.de/person/ghenz) for data capture; [Priyanka Patel](https://ps.is.mpg.de/person/ppatel) for alignment; [Giorgio Becherini](https://ps.is.mpg.de/person/gbecherini) and [Nima Ghorbani](https://nghorbani.github.io/) for MoSh++; [Leyre Sánchez Vinuela](https://is.mpg.de/person/lsanchez), [Andres Camilo Mendoza Patino](https://ps.is.mpg.de/person/acmendoza), [Mustafa Alperen Ekinci](https://ps.is.mpg.de/person/mekinci) for data cleaning; TB for Vicon support; MH and [Jakob Reinhardt](https://ps.is.mpg.de/person/jreinhardt) for object scanning; [Taylor McConnell](https://ps.is.mpg.de/person/tmcconnell) for Vicon support, and data cleaning coordination; [Benjamin Pellkofer](https://ps.is.mpg.de/person/bpellkofer) for IT/web support; [Neelay Shah](https://ps.is.mpg.de/person/nshah), [Jean-Claude Passy](https://is.mpg.de/person/jpassy), [Valkyrie Felso](https://is.mpg.de/person/vfelso) for evaluation server. We also thank [Adrian Spurr](https://ait.ethz.ch/people/spurra/) and [Xu Chen](https://ait.ethz.ch/people/xu/) for insightful discussion. OT and DT were supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B\".\n\n\n### Contact\n\nFor technical questions, please create an issue. For other questions, please contact `arctic@tue.mpg.de`.\n\nFor commercial licensing, please contact `ps-licensing@tue.mpg.de`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkabouzeid%2Fjointtransformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkabouzeid%2Fjointtransformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkabouzeid%2Fjointtransformer/lists"}