https://github.com/ai4ce/egopat3dv2
[ICRA 2024] Official Implementation of EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interaction
https://github.com/ai4ce/egopat3dv2
computer-vision dataset egocentric-vision human-robot-collaboration human-robot-interaction robotics
Last synced: 5 months ago
JSON representation
[ICRA 2024] Official Implementation of EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interaction
- Host: GitHub
- URL: https://github.com/ai4ce/egopat3dv2
- Owner: ai4ce
- Created: 2023-07-14T02:59:41.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2024-10-10T18:33:53.000Z (over 1 year ago)
- Last Synced: 2025-04-22T20:43:39.633Z (about 1 year ago)
- Topics: computer-vision, dataset, egocentric-vision, human-robot-collaboration, human-robot-interaction, robotics
- Language: Python
- Homepage: https://ai4ce.github.io/EgoPAT3Dv2/
- Size: 252 MB
- Stars: 3
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# [EgoPAT3Dv2: Predicting 3D Action Target from 2D Egocentric Vision for Human-Robot Interaction](https://ai4ce.github.io/EgoPAT3Dv2/).
[Irving Fang](https://irvingf7.github.io/)\*, [Yuzhong Chen](https://github.com/yuzhongchen/)\*, [Yifan Wang]()\*, [Jianghan Zhang]()\+, [Qiushi Zhang]()\+, [Jiali Xu]()\+, [Xibo He](), [Weibo Gao](), [Hao Su](), [Yiming Li](https://yimingli-page.github.io/), [Chen Feng](https://scholar.google.com/citations?user=YeG8ZM0AAAAJ)
## Project Website
Please visit [our project website](https://ai4ce.github.io/EgoPAT3Dv2/) for more information, such as a **video presentation**.
## Environment Setup
The project was developed on `Python 3.11.5` and `PyTorch 2.1.1` with `CUDA 11.8.0` binaries. For more details about the required packages, please take a look at `requirements.txt`
## Dataset
Please visit our [Hugging Face repo](https://huggingface.co/datasets/ai4ce/EgoPAT3Dv2/tree/main) to access and prepare the dataset.
## Training
We used the configuration files in `configs` to control the hyperparameters during our experiments. For more details on the hyperparameters, please refer to the [README](./configs/README.md) for configuration files.
To train a model with specified hyperparameters, please run
```
python train_DDP.py --config_file [your configuration file]
```
**Note:** Our training code is written for distributed training on NYU HPC with `PyTorch DDP` and `singularity` container. Please modify the `train_DDP.py` code to suit your needs. You can refer to the `train.SBATCH` file for more info on our distributed training setup.
## Testing
We used the configuration files in `configs` to control the hyperparameters during our experiments. For more details on the hyperparameters, please refer to the [README](./configs/README.md) for configuration files.
To test a model with specified hyperparameters, please run
```
python test_DDP.py --config_file [your configuration file]
```
**Note:** Slightly different from the training code, our testing code is written for single-card inference. Similarly, you can check out the `test.SBATCH` file for more info on our HPC testing setup.
### Testing Results Visualization
To produce aggregated results similar to what's presented in Table I of the paper, please run
```
python eval.py --model_name [model name in the config file]
```
You can also use the `--mode` parameter to dictate whether the test results are created for the seen or unseen test set.
**Note:** We are currently updating this file.