{"id":17948970,"url":"https://github.com/hysts/pl_gaze_estimation","last_synced_at":"2025-03-24T22:35:45.625Z","repository":{"id":61158157,"uuid":"431838502","full_name":"hysts/pl_gaze_estimation","owner":"hysts","description":null,"archived":true,"fork":false,"pushed_at":"2021-11-27T14:24:36.000Z","size":46,"stargazers_count":40,"open_issues_count":0,"forks_count":14,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-02-14T06:26:43.095Z","etag":null,"topics":["computer-vision","gaze","gaze-estimation","pytorch","pytorch-lightning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hysts.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-25T12:32:42.000Z","updated_at":"2024-12-06T12:40:39.000Z","dependencies_parsed_at":"2022-10-11T23:29:55.458Z","dependency_job_id":null,"html_url":"https://github.com/hysts/pl_gaze_estimation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hysts%2Fpl_gaze_estimation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hysts%2Fpl_gaze_estimation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hysts%2Fpl_gaze_estimation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hysts%2Fpl_gaze_estimation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hysts","download_url":"https://codeload.github.com/hysts/pl_gaze_estimation/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245366205,"owners_count":20603438,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","gaze","gaze-estimation","pytorch","pytorch-lightning"],"created_at":"2024-10-29T09:10:51.971Z","updated_at":"2025-03-24T22:35:40.592Z","avatar_url":"https://github.com/hysts.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# pl_gaze_estimation\n\n[![MIT License](https://img.shields.io/badge/license-MIT-green)](https://opensource.org/licenses/MIT)\n[![GitHub stars](https://img.shields.io/github/stars/hysts/pl_gaze_estimation.svg?style=flat-square\u0026logo=github\u0026label=Stars\u0026logoColor=white)](https://github.com/hysts/pl_gaze_estimation)\n\nTraining code for gaze estimation models using MPIIGaze, MPIIFaceGaze, and ETH-XGaze.\n\nhttps://user-images.githubusercontent.com/25161192/143683988-06ab6495-ae97-4201-b3f1-39dacf4bc69c.mp4\n\nA demo is available in [this repo](https://github.com/hysts/pytorch_mpiigaze_demo).\n\n## Installation\n\n- Linux (Tested on Ubuntu only)\n- Python \u003e= 3.9\n\n```bash\npip install -r requirements.txt\n```\n\nFor docker environment, see [here](#docker-environment).\n\n## Usage\n\nThe basic usage is as follows:\n```bash\npython train.py --configs /path/to/your/config.yaml\n```\n\nMultiple config files can be specified to set new variables or overwrite previously\ndefined values.\nYou can also overwrite previously defined values by listing the key/value\npairs after the `--options` flags.\n\n```\nusage: train.py [-h] --configs [CONFIGS ...] [--options ...]\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --configs [CONFIGS ...]\n                        Paths to config files.\n  --options ...         Variables to overwrite. (optional)\n```\n\n### MPIIGaze\n\n```bash\ntest_id=0\npython train.py --configs configs/examples/mpiigaze_lenet.yaml \\\n                --options EXPERIMENT.TEST_ID ${test_id} \\\n                          EXPERIMENT.OUTPUT_DIR exp0000/$(printf %02d ${test_id})\n```\n\nYou need to download and preprocess the dataset before training by running the following commands:\n```bash\nbash scripts/download_mpiigaze_dataset.sh\npython scripts/preprocess_mpiigaze.py --dataset datasets/MPIIGaze -o datasets/\n```\n(Additional packages are required to preprocess the dataset. See the [code](scripts/preprocess_mpiigaze.py).)\n\n### MPIIFaceGaze\n\n```bash\ntest_id=0\npython train.py --configs configs/examples/mpiifacegaze.yaml \\\n                --options SCHEDULER.WARMUP.EPOCHS 3 \\\n                          EXPERIMENT.TEST_ID ${test_id} \\\n                          EXPERIMENT.OUTPUT_DIR exp0000/$(printf %02d ${test_id})\n```\n\nYou need to download and preprocess the dataset before training by running the following commands:\n```bash\nbash scripts/download_mpiifacegaze_dataset.sh\npython scripts/preprocess_mpiifacegaze.py --dataset datasets/MPIIFaceGaze_normalized -o datasets/\n```\n(Additional packages are required to preprocess the dataset. See the [code](scripts/preprocess_mpiifacegaze.py).)\n\n### ETH-XGaze\n\n```bash\npython train.py \\\n    --config configs/examples/eth_xgaze.yaml \\\n    --options \\\n        VAL.VAL_INDICES \"[1, 23, 24, 35, 38, 46, 58, 63, 70, 78]\" \\\n        SCHEDULER.EPOCHS 15 \\\n        SCHEDULER.MULTISTEP.MILESTONES \"[10, 13, 14]\" \\\n        DATASET.TRANSFORM.TRAIN.HORIZONTAL_FLIP true \\\n        EXPERIMENT.OUTPUT_DIR exp0000\n```\n\n## Docker Environment\n\ndocker-compose 1.29.2 is required.\n\n### Build\n```bash\ndocker-compose build train\n```\n\n### Train\n```bash\ndocker-compose run --rm -u $(id -u):$(id -g) -v /path/to/datasets:/datasets train python train.py --configs /path/to/your/config.yaml\n```\n\n## Results on ETH-XGaze dataset\n\nAll the results in the table below are of a single run.\nIn these experiments, the data of subjects with id 1, 23, 24, 35, 38, 46, 58, 63, 70, and 78\nwere used as the validation data.\nThe models were trained for 15 epochs with a multi-step learning rate schedule.\nThe learning rate was multiplied by 0.1 at epochs 10, 13, and 14.\n\n|  Model             | hflip | GPU    | precision | batch size | optimizer | lr     | weight decay | training time | val angle error | val loss |\n|--------------------|-------|--------|-----------|------------|-----------|--------|--------------|---------------|-----------------|----------|\n| EfficientNet-lite0 | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |      4h42m    | 5.330           | 0.06970  |\n| EfficientNet-b0    | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |      5h58m    | 5.139           | 0.06672  |\n| ResNet18           | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |      4h04m    | 4.878           | 0.06427  |\n| ResNet50           | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |      8h42m    | 4.720           | 0.06087  |\n| HRNet-18           | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |     21h56m    | 4.657           | 0.05937  |\n| ResNeSt26d         | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |      8h02m    | 4.409           | 0.05678  |\n| RegNetY160         | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |   1d05h30m    | 4.377           | 0.05638  |\n| Swin-S             | yes   | V100   | 32        | 64         | Adam      | 0.0001 | 0            |   1d00h08m    | 4.318           | 0.05629  |\n| HRNet-64           | yes   | V100x8 | 16        | 64         | AdamW     | 0.0008 | 0.05         |      3h11m    | 4.302           | 0.05523  |\n| ResNeSt269e        | yes   | V100x8 | 16        | 56         | AdamW     | 0.0008 | 0.05         |      5h31m    | 4.045           | 0.05200  |\n\n## Related repos\n\n- https://github.com/hysts/pytorch_mpiigaze\n- https://github.com/hysts/pytorch_mpiigaze_demo\n\n## References\n\n- Cai, Xin, Boyu Chen, Jiabei Zeng, Jiajun Zhang, Yunjia Sun, Xiao Wang, Zhilong Ji, Xiao Liu, Xilin Chen, and Shiguang Shan. \"Gaze Estimation with an Ensemble of Four Architectures.\" arXiv preprint arXiv:2107.01980 (2021). [arXiv:2107.01980](https://arxiv.org/abs/2107.01980), [GitHub](https://github.com/VIPL-TAL-GAZE/GAZE2021)\n- Zhang, Xucong, Seonwook Park, Thabo Beeler, Derek Bradley, Siyu Tang, and Otmar Hilliges. \"ETH-XGaze: A Large Scale Dataset for Gaze Estimation under Extreme Head Pose and Gaze Variation.\" In European Conference on Computer Vision (ECCV), 2020. [arXiv:2007.15837](https://arxiv.org/abs/2007.15837), [Project Page](https://ait.ethz.ch/projects/2020/ETH-XGaze/), [GitHub](https://github.com/xucong-zhang/ETH-XGaze), [Leaderboard](https://competitions.codalab.org/competitions/28930)\n- Zhang, Xucong, Yusuke Sugano, Mario Fritz, and Andreas Bulling. \"Appearance-based Gaze Estimation in the Wild.\" Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. [arXiv:1504.02863](https://arxiv.org/abs/1504.02863), [Project Page](https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/research/gaze-based-human-computer-interaction/appearance-based-gaze-estimation-in-the-wild/)\n- Zhang, Xucong, Yusuke Sugano, Mario Fritz, and Andreas Bulling. \"It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation.\" Proc. of the IEEE Conference on Computer Vision and Pattern Recognition Workshops(CVPRW), 2017. [arXiv:1611.08860](https://arxiv.org/abs/1611.08860), [Project Page](https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/gaze-based-human-computer-interaction/its-written-all-over-your-face-full-face-appearance-based-gaze-estimation/)\n- Zhang, Xucong, Yusuke Sugano, Mario Fritz, and Andreas Bulling. \"MPIIGaze: Real-World Dataset and Deep Appearance-Based Gaze Estimation.\" IEEE transactions on pattern analysis and machine intelligence 41 (2017). [arXiv:1711.09017](https://arxiv.org/abs/1711.09017)\n- Zhang, Xucong, Yusuke Sugano, and Andreas Bulling. \"Evaluation of Appearance-Based Methods and Implications for Gaze-Based Applications.\" Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2019. [arXiv](https://arxiv.org/abs/1901.10906), [code](https://git.hcics.simtech.uni-stuttgart.de/public-projects/opengaze)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhysts%2Fpl_gaze_estimation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhysts%2Fpl_gaze_estimation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhysts%2Fpl_gaze_estimation/lists"}