{"id":13445088,"url":"https://github.com/nogu-atsu/NARF","last_synced_at":"2025-03-20T19:31:29.805Z","repository":{"id":94955545,"uuid":"352954907","full_name":"nogu-atsu/NARF","owner":"nogu-atsu","description":"Neural Articulated Radiance Field","archived":false,"fork":false,"pushed_at":"2022-04-12T04:15:26.000Z","size":466,"stargazers_count":151,"open_issues_count":2,"forks_count":10,"subscribers_count":18,"default_branch":"master","last_synced_at":"2024-08-01T04:02:39.426Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nogu-atsu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-30T10:06:36.000Z","updated_at":"2024-06-28T14:22:15.000Z","dependencies_parsed_at":"2023-04-04T16:31:48.132Z","dependency_job_id":null,"html_url":"https://github.com/nogu-atsu/NARF","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogu-atsu%2FNARF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogu-atsu%2FNARF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogu-atsu%2FNARF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nogu-atsu%2FNARF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nogu-atsu","download_url":"https://codeload.github.com/nogu-atsu/NARF/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244676455,"owners_count":20491829,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T04:01:49.289Z","updated_at":"2025-03-20T19:31:29.794Z","avatar_url":"https://github.com/nogu-atsu.png","language":"Python","funding_links":[],"categories":["Papers"],"sub_categories":["NeRF Related Tasks"],"readme":"# Neural Articulated Radiance Field\n# NARF\n\n\n\u003e Neural Articulated Radiance Field  \n\u003e Atsuhiro Noguchi, Xiao Sun, Stephen Lin, Tatsuya Harada  \n\u003e ICCV 2021  \n\n[[Paper]](https://arxiv.org/abs/2104.03110) [[Code]](https://github.com/nogu-atsu/NARF#code)\n\n## Abstract\nWe present Neural Articulated Radiance Field (NARF), a novel deformable 3D representation for articulated objects learned from images. While recent advances in 3D implicit representation have made it possible to learn models of complex objects, learning pose-controllable representations of articulated objects remains a challenge, as current methods require 3D shape supervision and are unable to render appearance. In formulating an implicit representation of 3D articulated objects, our method considers only the rigid transformation of the most relevant object part in solving for the radiance field at each 3D location. In this way, the proposed method represents pose-dependent changes without significantly increasing the computational complexity. NARF is fully differentiable and can be trained from images with pose annotations. Moreover, through the use of an autoencoder, it can learn appearance variations over multiple instances of an object class. Experiments show that the proposed method is efficient and can generalize well to novel poses.\n\n## Method\nWe extend Neural Radiance Fields (NeRF) to articulated objects.\nNARF is a NeRF conditioned on skeletal parameters and skeletal posture, and is an MLP that outputs the density and color of a point with 3D position and 2D viewing direction as input.\nSince articulated objects can be regarded as multiple rigid bodies connected by joints, the following two assumptions can be made\n\n- The density of each part does not change in the coordinate system fixed to the part.\n- A point on the surface of the object belongs to only one of the parts.\n\nTherefore, we transform the input 3D coordinates into local coordinates of each part and use them as input for the model. From the second hypothesis, we use selector MLP to select only one necessary coordinate and mask the others.\n\nAn overview of the model is shown in the figure.\n\n![overview](https://github.com/nogu-atsu/NARF/wiki/images/overview.jpg)\n\nThe model is trained with the L2 loss between the generated image and the ground truth image.\n\n## Results\nThe proposed NARF is capable of rendering images with explicit control of the viewpoint, bone pose, and bone parameters. These representations are disentangled and can be controlled independently.\n\nViewpoint change (seen in training)\n\n\u003cimg src=\"https://github.com/nogu-atsu/NARF/wiki/images/concat_inter_camera_arf.mp4.mp4.gif\" width=\"640px\"\u003e\n\nPose change (unseen in training)\n\n\u003cimg src=\"https://github.com/nogu-atsu/NARF/wiki/images/concat_inter_pose_arf.mp4.gif\" width=\"640px\"\u003e\n\nBone length change (unseen in training)\n\n\u003cimg src=\"https://github.com/nogu-atsu/NARF/wiki/images/concat_inter_bone_arf.mp4.mp4.gif\" width=\"640px\"\u003e\n\nNARF generalizes well to unseen viewpoints during training.\n\n\u003cimg src=\"https://github.com/nogu-atsu/NARF/wiki/images/concat_inter_camera_ood_arf.mp4.mp4.gif\" width=\"640px\"\u003e\n\nFurthermore, NARF can render segmentation for each part by visualizing the output values of the selector.\n\n\u003cimg src=\"https://github.com/nogu-atsu/NARF/wiki/images/concat_inter_camera_arf_segmentation.mp4.mp4.gif\" width=\"640px\"\u003e\n\nNARF can learn appearance variations by combining it with an autoencoder. \nThe video below visualizes the disentangled representations and segmentation masks learned by NARF autoencoder.\n\n\u003cimg src=\"https://github.com/nogu-atsu/NARF/wiki/images/ae_results.mp4.gif\" width=\"1024px\"\u003e\n\n\n# Code\n## Envirionment\npython 3.7.*\\\npytorch \u003e= 1.7.1\\\ntorchvision \u003e= 0.8.2\n```\npip install tensorboardx pyyaml opencv-python pandas ninja easydict tqdm scipy scikit-image\n```\n\n## Dataset preparation\n### THUman\nPlease refer to https://github.com/nogu-atsu/NARF/tree/master/data/THUman\n\n### Your own dataset\nComing soon.\n\n## Training\n- Write config file like `NARF/configs/THUman/results_wxl_20181008_wlz_3_M/NARF_D.yml`. Do not change `default.yml`\n    - `out_root`: root directory to save models\n    - `out`: experiment name\n    - `data_root`: directory the `dataset` is in\n- Run training specifying a config file\n  \n    ```CUDA_VISIBLE_DEVICES=0 python train.py --config NARF/configs/[your_config.yml] --num_workers 1```\n- Distributed data parallel\n  \n    ```python train_ddp.py --config NARF/configs/[your_config.yml] --gpus 4 --num_workers 1```\n\n## Validation\n- Single gpu\n\n    ```python train.py --config NARF/configs/[your_config.yml] --num_workers 1 --validation --resume_latest```\n- Multiple gpus\n\n    ```python train_ddp.py --config NARF/configs/[your_config.yml] --gpus 4 --num_workers 1 --validation --resume_latest```\n- The results are saved to `val_metrics.json` in the same directory as the snapshots.\n\n## Computational cost\n```\npython computational_cost.py --config NARF/configs/[your_config.yml]\n```\n\n## Visualize results\n\n- Generate interpolation videos\n  ```\n  cd visualize\n  python NARF_interpolation.py --config ../NARF/configs/[your_config.yml]\n  ```\n\n  The results are saved to the same directory as the snapshots.\n  With the default settings, it takes 30 minutes on a V100 gpu to generate a 30-frame video\n\n# Acknowledgement\nhttps://github.com/rosinality/stylegan2-pytorch \\\nhttps://github.com/ZhengZerong/DeepHuman \\\nhttps://smpl.is.tue.mpg.de/\n\n# BibTex\n```\n@inproceedings{2021narf,\n  author    = {Noguchi, Atsuhiro and Sun, Xiao and Lin, Stephen and Harada, Tatsuya},\n  title     = {Neural Articulated Radiance Field},\n  booktitle = {International Conference on Computer Vision},\n  year      = {2021},\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnogu-atsu%2FNARF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnogu-atsu%2FNARF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnogu-atsu%2FNARF/lists"}