{"id":13584979,"url":"https://github.com/synthesiaresearch/humanrf","last_synced_at":"2025-04-07T06:32:10.243Z","repository":{"id":163611202,"uuid":"633752971","full_name":"synthesiaresearch/humanrf","owner":"synthesiaresearch","description":"Official code for \"HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion\"","archived":false,"fork":false,"pushed_at":"2024-09-17T10:45:58.000Z","size":110702,"stargazers_count":467,"open_issues_count":13,"forks_count":28,"subscribers_count":39,"default_branch":"main","last_synced_at":"2025-03-26T21:38:39.808Z","etag":null,"topics":["3d-reconstruction","computer-graphics","computer-vision","machine-learning","nerf","siggraph2023"],"latest_commit_sha":null,"homepage":"http://actors-hq.com","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/synthesiaresearch.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-28T07:46:47.000Z","updated_at":"2025-03-18T15:24:14.000Z","dependencies_parsed_at":"2024-01-13T18:24:24.625Z","dependency_job_id":"5d35568f-15bd-48a3-b3a2-22a8d924876f","html_url":"https://github.com/synthesiaresearch/humanrf","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synthesiaresearch%2Fhumanrf","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synthesiaresearch%2Fhumanrf/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synthesiaresearch%2Fhumanrf/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/synthesiaresearch%2Fhumanrf/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/synthesiaresearch","download_url":"https://codeload.github.com/synthesiaresearch/humanrf/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247607230,"owners_count":20965942,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-reconstruction","computer-graphics","computer-vision","machine-learning","nerf","siggraph2023"],"created_at":"2024-08-01T15:04:38.409Z","updated_at":"2025-04-07T06:32:10.237Z","avatar_url":"https://github.com/synthesiaresearch.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Official repository of HumanRF and ActorsHQ\n\n![teaser image](./media/teaser.jpg)\n\n**HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion**\u003cbr\u003e\nMustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, Matthias Nießner\n\n## Installation\n\n```bash\n\ngit clone --depth=1 --recursive https://github.com/synthesiaresearch/humanrf\n\n# Install GLM\nsudo apt-get install libglm-dev\n\n# Install required packages and Tiny CUDA NN.\npip install -r requirements.txt\npip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch\n\n# Install ActorsHQ package (dataset and data loader)\ncd actorshq\npip3 install .\n\n# Install HumanRF package (method)\ncd ../humanrf\npip3 install .\n\n# Add the installation folder to the PYTHONPATH\nexport PYTHONPATH=$PYTHONPATH:/path/to/repo\n```\n\n## Usage\n\nTo get started quickly you can download a part of ActorsHQ and run HumanRF with the following commands:\n\n```bash\n./actorshq/dataset/download_manager.py \\\n    actorshq_access_4x.yaml \\\n    /tmp/actorshq \\\n    --actor Actor01 \\\n    --sequence Sequence1 \\\n    --scale 4 \\\n    --frame_start 15 \\\n    --frame_stop 65\n\n./humanrf/run.py \\\n    --config example_humanrf \\\n    --workspace /tmp/example_workspace \\\n    --dataset.path /tmp/actorshq\n```\n\nFor that you will need an access file `actorshq_access_4x.yaml`, which you can request [here](https://www.actors-hq.com/), see section \"Data access and download\" for more details.\n\n## Data overview\n\nThe following datastructure is used to store the data:\n\n```\n└── Actor01\n    └── Sequence1\n        ├── 1x\n        │   ├── calibration.csv  # Camera calibration data as described below.\n        │   ├── light_annotations.csv  # 2D annotations for light sources.\n        │   ├── masks  # Per-frame mask for each camera.\n        │   │   ├── Cam001/Cam001_mask000000.png\n        │   │   ...\n        │   └── rgbs  # Per-frame rgb for each camera, with background removed.\n        │       ├── Cam001/Cam001_rgb000000.jpg\n        │       ...\n        ├── 2x\n        │   ...\n        ├── 4x\n        │   ...\n        ├── aabbs.csv  # Per-frame axis-aligned bounding boxes of the meshes.\n        ├── occupancy_grids  # Per-frame occupancy grids.\n        │   ├── occupancy_grid000000.npz\n        │   ...\n        ├── meshes.abc  # Per-frame meshes in the Alembic format.\n        ├── scene.blend  # Blender scene file that visualizes meshes, cameras and rgb images.\n        └── scene.json  # Scene description file that stores the number of frames.\n```\n\n## Data access and download\n\nIn order to be able to download the ActorsHQ dataset you need to request an access yaml file from [here](https://www.actors-hq.com/). This file contains the credentials to download the data from our cloud storage. Given that file you can use the `download_manager.py` program to download a subset of the data. For example, to download the first 50 frames of the first sequence of the first actor in 4x scale you can use the following command, which stores the data to `/tmp/actorshq`:\n\n```bash\n./actorshq/dataset/download_manager.py \\\n    actorshq_access_4x.yaml \\\n    /tmp/actorshq \\\n    --actor Actor01 \\\n    --sequence Sequence1 \\\n    --scale 4 \\\n    --frame_start 0 \\\n    --frame_stop 50\n```\n\n## Calibration format\n\nCalibration data is provided an `calibration.csv` files that have the following format:\n\n```\nname, w, h, rx, ry, rz, tx, ty, tz, fx, fy, px, py\nCam001, 4112, 3008, 3.14159265359, 0.0, 0.0, 1.0, 0.0, 230.0, 1.773863, 1.773863, 0.5, 0.5\n...\n```\n\nHere, the rotation vector `rx, ry, rz` is in axis-angle format and focal length and principal point are normalized by image width and height.\n\nTo quickly visualize the cameras in 3D the following snippet can be used:\n```python\n#!/usr/bin/env python3\nimport numpy as np\nfrom actorshq.dataset.camera_data import CameraData, read_calibration_csv\nfrom matplotlib import pyplot as plt\nfrom scipy.spatial.transform import Rotation\n\ncameras = read_calibration_csv(\"/path/to/calibration.csv\")\n\nfig = plt.figure()\nax = fig.add_subplot(111, projection='3d')\nax.set_xlim(-2.5, 2.5)\nax.set_ylim(-1.0, 4.0)\nax.set_zlim(-2.5, 2.5)\n\ndef draw_axis(c,v,color):\n    p = c+v\n    ax.plot([c[0], p[0]], [c[1], p[1]], [c[2], p[2]], color=color, lw=1)\n\ns = 0.1\ndraw_axis(np.array([0,0,0]), np.array([s,0,0]), \"red\")\ndraw_axis(np.array([0,0,0]), np.array([0,s,0]), \"green\")\ndraw_axis(np.array([0,0,0]), np.array([0,0,s]), \"blue\")\nfor camera in cameras:\n    rotation = Rotation.from_rotvec(camera.rotation_axisangle)\n    draw_axis(camera.translation, rotation.apply(np.array([s,0,0])), \"red\")\n    draw_axis(camera.translation, rotation.apply(np.array([0,s,0])), \"green\")\n    draw_axis(camera.translation, rotation.apply(np.array([0,0,s])), \"blue\")\nplt.show()\n```\n\nWhich will result in a visualization similar to this one:\n\n![](media/camera_setup_matplotlib.jpg)\n\nA depthmap, as rendered with the provided [renderer](actorshq/toolbox/mesh_tools/renderer/main.cpp), can be unprojected to a pointcloud as follows (output `.xzy` can be opened with [meshlab](https://www.meshlab.net/)):\n\n```python\n#!/usr/bin/env python3\nimport numpy as np\nfrom actorshq.dataset.camera_data import CameraData, read_calibration_csv\nimport os\nos.environ[\"OPENCV_IO_ENABLE_OPENEXR\"]=\"1\"\nimport cv2\n\ncamera = read_calibration_csv(\"/path/to/calibration.csv\")[0]\ndepth = cv2.imread(\"/path/to/depth/Cam001_depth000000.exr\", cv2.IMREAD_UNCHANGED)\npath_xyz = \"/path/to/output/cloud.xyz\"\n\ndef unproject(cam: CameraData, depth: np.array):\n    xx, yy = np.meshgrid(np.arange(cam.width), np.arange(cam.height))\n    coords = np.stack([xx, yy, np.ones_like(xx)], axis=-1) * depth[:,:,np.newaxis]\n    coords = coords[depth \u003e 0].reshape(-1, 3)\n    Kinv = np.linalg.inv(cam.intrinsic_matrix())\n    return coords @ Kinv.T\n\npoints = unproject(camera, depth)\n\nwith open(path_xyz, \"w\") as f:\n    for p in points:\n        f.write(f\"{p[0]} {p[1]} {p[2]} \\n\")\n```\n\n## Docker\n\n```bash\nnvidia-docker build --build-arg TCNN_CUDA_ARCHITECTURES=86 .\n```\n\n## Toolbox\n\nBesides the source code of HumanRF and the raw data we provide additional tools and examples to ease the work with the ActorsHQ dataset.\nThese tools are located in [actorshq/toolbox](actorshq/toolbox) and are described in a [separate readme](actorshq/toolbox/README.md) file.\n\nThe following functionality is provided:\n* Export ActorsHQ to a blend file (note that these can be downloaded via [download_manager.py](./actorshq/dataset/download_manager.py))\n* Export ActorsHQ to colmap format\n* Export ActorsHQ to NGP format\n* Import DFA to ActorsHQ format\n* Occupancy grid generator\n* Alembic (`.abc`) to obj converter\n* Alembic renderer (masks / depthmaps)\n\n## Citation\n\nIf you use ActorsHQ or HumanRF in your work, please consider citing via\n\n```bibtex\n@article{isik2023humanrf,\n  title = {HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion},\n  author = {I\\c{s}{\\i}k, Mustafa and Rünz, Martin and Georgopoulos, Markos and Khakhulin, Taras\n   and Starck, Jonathan and Agapito, Lourdes and Nießner, Matthias},\n  journal = {ACM Transactions on Graphics (TOG)},\n  volume = {42},\n  number = {4},\n  pages = {1--12},\n  year = {2023},\n  publisher = {ACM New York, NY, USA},\n  doi = {10.1145/3592415},\n  url = {https://doi.org/10.1145/3592415},\n}\n```\n\n## License\n\nThis work is made available under the [Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/) license, see [LICENSE.txt](LICENSE.txt) for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsynthesiaresearch%2Fhumanrf","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsynthesiaresearch%2Fhumanrf","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsynthesiaresearch%2Fhumanrf/lists"}