{"id":15036746,"url":"https://github.com/apple/ml-hugs","last_synced_at":"2025-04-07T15:05:59.737Z","repository":{"id":235279500,"uuid":"783979393","full_name":"apple/ml-hugs","owner":"apple","description":"Official repository of HUGS: Human Gaussian Splats (CVPR 2024)","archived":false,"fork":false,"pushed_at":"2024-05-01T18:03:17.000Z","size":7127,"stargazers_count":274,"open_issues_count":21,"forks_count":35,"subscribers_count":16,"default_branch":"main","last_synced_at":"2025-03-31T14:09:14.400Z","etag":null,"topics":["gaussian-splatting","neural-rendering"],"latest_commit_sha":null,"homepage":"https://machinelearning.apple.com/research/hugs","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/apple.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-09T00:22:59.000Z","updated_at":"2025-03-30T03:28:17.000Z","dependencies_parsed_at":"2024-05-01T19:38:04.225Z","dependency_job_id":null,"html_url":"https://github.com/apple/ml-hugs","commit_stats":null,"previous_names":["apple/ml-hugs"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hugs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hugs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hugs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/apple%2Fml-hugs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/apple","download_url":"https://codeload.github.com/apple/ml-hugs/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247675596,"owners_count":20977376,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gaussian-splatting","neural-rendering"],"created_at":"2024-09-24T20:32:11.545Z","updated_at":"2025-04-07T15:05:59.717Z","avatar_url":"https://github.com/apple.png","language":"Python","funding_links":[],"categories":["Paper List"],"sub_categories":["Follow-up Papers"],"readme":"# HUGS: Human Gaussian Splats\n\nThis repository is a reference implementation for HUGS. HUGS reconstructs both the background scene and an animatable human from a single video using neural radiance fields.\n\n[[Paper](https://arxiv.org/abs/2311.17910)] | [[Project Page](https://machinelearning.apple.com/research/hugs)]\n\n\u003e [**HUGS: Human Gaussian Splats**](https://arxiv.org/abs/2311.17910),            \n\u003e [Muhammed Kocabas](https://ps.is.tuebingen.mpg.de/person/mkocabas), \n\u003e [Jen-Hao Rick Chang](https://rick-chang.github.io/), \n\u003e [James Gabriel](https://www.linkedin.com/in/jamescgabriel/), \n\u003e [Oncel Tuzel](https://www.onceltuzel.net/), \n\u003e [Anurag Ranjan](https://anuragranj.github.io/)       \n\u003e *IEEE Computer Vision and Pattern Recognition (CVPR) 2024* \n\n\u003cp float=\"center\"\u003e\n  \u003cimg src=\"assets/hugs_teaser.png\" width=\"100%\" /\u003e\n\u003c/p\u003e\n\n# Getting Started\n\nWe tested our system with Ubuntu 22.04.3 using a CUDA 11.7 compatible GPU.\n\n- Clone our repo:\n```\ngit clone --recursive git@github.com:apple/ml-hugs.git\n```\n\n- Run the setup script to create a conda environment and install the required packages.\n```\nsource scripts/conda_setup.sh\n```\n\n# Preparing the datasets and models\n\n## Datasets\n- Download the SMPL neutral body model\n    - Register to [SMPL](https://smpl.is.tue.mpg.de/index.html) website.\n    - Download v1.1.0 and SMPL UV obj file from the [download](https://smpl.is.tue.mpg.de/download.php) page.\n    - Extract the files and rename `basicModel_neutral_lbs_10_207_0_v1.0.0.pkl` to `SMPL_NEUTRAL.pkl`.\n    - Put the files into `./data/smpl/` folder with the following structure:\n\n        ```\n        data/smpl/\n        ├── SMPL_NEUTRAL.pkl\n        └── smpl_uv.obj\n        ```\n\n- Download NeuMan dataset and pretrained models:\n    - Data ([download](https://docs-assets.developer.apple.com/ml-research/models/hugs/neuman_data.zip))\n    - Pretrained models ([download](https://docs-assets.developer.apple.com/ml-research/models/hugs/hugs_pretrained_models.zip))\n\n    Alternately, run the following script to set up data and pretrained models.\n    ```\n    source scripts/prepare_data_models.sh\n    ```\n\n- Download AMASS dataset for novel animation rendering:\n  - AMASS dataset is used for rendering novel poses.\n  - We used SFU mocap(SMPL+H G) and MPI_mosh (SMPL+H G) subsets, please download from [AMASS](https://amass.is.tue.mpg.de/download.php).\n  - Put the downloaded mocap data in to `./data/` folder.\n\nAfter following the above steps, you should obtain a folder structure similar to this:\n\n```\ndata/\n├── smpl\n│   ├── SMPL_FEMALE.pkl\n│   ├── SMPL_MALE.pkl\n│   ├── SMPL_NEUTRAL.pkl\n│   ├── smpl_uv.obj\n├── neuman\n│   └── dataset\n│       ├── bike\n│       ├── citron\n│       ├── jogging\n│       ├── lab\n│       ├── parkinglot\n│       └── seattle\n├── MPI_mosh\n│   ├── 00008\n│   ├── 00031\n│   ├── ...\n│   └── 50027\n└── SFU\n    ├── 0005\n    ├── 0007\n    ├── ...\n    └── 0018\n```\n\n\n# Training\n\nTo train HUGS on NeuMan dataset, there are three different modes you can choose from: 1. joint human and scene 2. human only, 3. scene only. \n\n1. Joint human and scene training\n\n    This is the original HUGS setup where jointly optimize human Gaussians and scene Gaussians. \n    ```\n    python main.py --cfg_file cfg_files/release/neuman/hugs_human_scene.yaml dataset.seq=lab\n    ```\n\n2. Human only training\n\n    This mode only optimizes the Triplane+MLP model introduced in HUGS.\n\n    ```\n    python main.py --cfg_file cfg_files/release/neuman/hugs_human.yaml dataset.seq=lab\n    ```\n\n\n3. Scene only training\n\n    This setup is identical to original 3DGS paper. Here we provide the script to run it on the NeuMan dataset\n\n    ```\n    python main.py --cfg_file cfg_files/release/neuman/hugs_scene.yaml dataset.seq=lab\n    ```\n\n`cfg_files/release` directory contains the final configuration files we used to train HUGS. Please refer to the [config.py](hugs/cfg/config.py) file to see different config parameters and their meanings.\n\n**Note**: Expect to see slight differences compared to the pretrained models. This is due to the inherent randomness in the rendering process, which makes achieving deterministic results across multiple runs challenging, even when proper seeding is applied. So it is expected to obtain results slightly different than what is reported in the paper.\n\n# Evaluation and Animation\n\nHere we show how to perform evaluation with the pretrained models on the NeuMan dataset.\n\n```\npython scripts/evaluate.py -o \u003c\u003cpath to the output directory\u003e\u003e\n```\n\nThis command will print out the PSNR, SSIM, and LPIPS metrics for a given pretrained model.\n\n# Citation\n```\n@inproceedings{\n    kocabas2024hugs,\n    title={{HUGS}: Human Gaussian Splatting},\n    author={Kocabas, Muhammed and Chang, Jen-Hao Rick and Gabriel, James and Tuzel, Oncel and Ranjan, Anurag},\n    booktitle = {2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    year = {2024},\n    url={https://arxiv.org/abs/2311.17910}\n}\n```\n\n# License\nThe code is released under the [LICENSE](LICENSE) terms.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-hugs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapple%2Fml-hugs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapple%2Fml-hugs/lists"}