{"id":19544124,"url":"https://github.com/sunset1995/horizonnet","last_synced_at":"2025-04-05T12:07:49.689Z","repository":{"id":46605570,"uuid":"171268612","full_name":"sunset1995/HorizonNet","owner":"sunset1995","description":"Pytorch implementation of HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation.","archived":false,"fork":false,"pushed_at":"2024-04-14T13:30:40.000Z","size":10497,"stargazers_count":330,"open_issues_count":33,"forks_count":91,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-03-29T11:08:58.429Z","etag":null,"topics":["360-photo","computer-vision","cvpr2019","horizonnet","pano-stretch-augmentation","room-layout"],"latest_commit_sha":null,"homepage":"https://sunset1995.github.io/HorizonNet/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sunset1995.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-18T11:05:44.000Z","updated_at":"2025-03-25T16:10:59.000Z","dependencies_parsed_at":"2024-11-11T03:35:16.327Z","dependency_job_id":null,"html_url":"https://github.com/sunset1995/HorizonNet","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunset1995%2FHorizonNet","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunset1995%2FHorizonNet/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunset1995%2FHorizonNet/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sunset1995%2FHorizonNet/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sunset1995","download_url":"https://codeload.github.com/sunset1995/HorizonNet/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247332609,"owners_count":20921853,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["360-photo","computer-vision","cvpr2019","horizonnet","pano-stretch-augmentation","room-layout"],"created_at":"2024-11-11T03:24:40.146Z","updated_at":"2025-04-05T12:07:49.661Z","avatar_url":"https://github.com/sunset1995.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HorizonNet\n\nThis is the implementation of our CVPR'19 \"[\nHorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation](https://arxiv.org/abs/1901.03861)\" ([project page](https://sunset1995.github.io/HorizonNet/)).\n\n![](assets/teaser.jpg)\n\n### Update\n- **2021.11.04: Report results on [Zillow Indoor dataset](https://github.com/zillow/zind). (See [the report :clipboard: on ZInD](README_ZInD.md)).**\n- 2021.04.03: Check out our new project [HoHoNet](https://github.com/sunset1995/HoHoNet) on this task and more!\n- 2021.03.14: (1) Use mesh instead of point cloud as layout viewer. (2) Update lsd detector dependency.\n- 2019.08.19: Report results on [Structured3D dataset](https://structured3d-dataset.org/). (See [the report :clipboard: on ST3D](README_ST3D.md)).\n- 2019.06.15: Bug fix for general layout (`dataset.py`, `inference.py` and `misc/post_proc.py`)\n\n### Feature\nThis repo is a **pure python** implementation that you can:\n- **Inference on your images** to get cuboid or general shaped room layout\n- **3D layout viewer**\n- **Correct rotation pose** to ensure manhattan alignment\n- **Pano stretch augmentation** copy and paste to apply on your own task\n- **Quantitative evaluatation** of 2D IoU, 3D IoU, Corner Error, Pixel Error of cuboid/general shape\n- **Your own dataset** preparation and training\n\n### Method overview\n![](assets/pipeline.jpg)\n\n### Installation\nPytorch installation is machine dependent, please install the correct version for your machine. The tested version is pytorch 1.8.1 with python 3.7.6.\n\n\u003cdetails\u003e\n  \u003csummary\u003e Dependencies (click to expand) \u003c/summary\u003e\n\n   - numpy\n   - scipy\n   - sklearn\n   - Pillow\n   - tqdm\n   - tensorboardX\n   - opencv-python\u003e=3.1 (for pre-processing)\n   - pylsd-nova\n   - open3d\u003e=0.7 (for layout 3D viewer)\n   - shapely\n\u003c/details\u003e\n\n\n### Download\n#### Dataset\n- PanoContext/Stanford2D3D Dataset\n    - [Download preprocessed pano/s2d3d](https://drive.google.com/open?id=1e-MuWRx3T4LJ8Bu4Dc0tKcSHF9Lk_66C) for training/validation/testing\n        - Put all of them under `data` directory so you should get:\n            ```\n            HorizonNet/\n            ├──data/\n            |  ├──layoutnet_dataset/\n            |  |  |--finetune_general/\n            |  |  |--test/\n            |  |  |--train/\n            |  |  |--valid/\n            ```\n        - `test`, `train`, `valid` are processed from [LayoutNet's cuboid dataset](https://github.com/zouchuhang/LayoutNet).\n        - `finetune_general` is re-annotated by us from `train` and `valid`. It contains  65 general shaped rooms.\n- Structured3D Dataset\n    - See [the tutorial](https://github.com/sunset1995/HorizonNet/blob/master/README_ST3D.md#dataset-preparation) to prepare training/validation/testing for HorizonNet.\n- Zillow Indoor Dataset\n    - See [the tutorial](https://github.com/sunset1995/HorizonNet/blob/master/README_ZInD.md#dataset-preparation) to prepare training/validation/testing for HorizonNet.\n\n\n#### Pretrained Models\nPlase download the pre-trained model [here](https://drive.google.com/drive/folders/1bgJspDogOHGdwXxCB8o3irU3_Gz9rTpK?usp=drive_link)\n- `resnet50_rnn__panos2d3d.pth`\n    - Trained on PanoContext/Stanford2d3d 817 pano images.\n    - Trained for 300 epoch\n- `resnet50_rnn__st3d.pth`\n    - Trained on Structured3D 18362 pano images\n    - Data setup: original furniture and lighting.\n    - Trained for 50 epoch.\n- `resnet50_rnn__zind.pth`\n    - Trained on Zillow Indoor 20077 pano images.\n    - Data setup: `layout_visible`, `is_primary`, `is_inside`, `is_ceiling_flat`.\n    - Trained for 50 epoch.\n\n\n## Inference on your images\n\nIn below explaination, I will use `assets/demo.png` for example.\n- ![](assets/demo.png) (modified from PanoContext dataset)\n\n\n### 1. Pre-processing (Align camera rotation pose)\n- **Execution**: Pre-process the above `assets/demo.png` by firing below command.\n    ```bash\n    python preprocess.py --img_glob assets/demo.png --output_dir assets/preprocessed/\n    ```\n    - `--img_glob` telling the path to your 360 room image(s).\n        - support shell-style wildcards with quote (e.g. `\"my_fasinated_img_dir/*png\"`).\n    - `--output_dir` telling the path to the directory for dumping the results.\n    - See `python preprocess.py -h` for more detailed script usage help.\n- **Outputs**: Under the given `--output_dir`, you will get results like below and prefix with source image basename.\n    - The aligned rgb images `[SOURCE BASENAME]_aligned_rgb.png` and line segments images `[SOURCE BASENAME]_aligned_line.png`\n        - `demo_aligned_rgb.png` | `demo_aligned_line.png`\n          :--------------------: | :---------------------:\n          ![](assets/preprocessed/demo_aligned_rgb.png) | ![](assets/preprocessed/demo_aligned_line.png)\n    - The detected vanishing points `[SOURCE BASENAME]_VP.txt` (Here `demo_VP.txt`)\n        ```\n        -0.002278 -0.500449 0.865763\n        0.000895 0.865764 0.500452\n        0.999999 -0.001137 0.000178\n        ```\n\n\n### 2. Estimating layout with HorizonNet\n- **Execution**: Predict the layout from above aligned image and line segments by firing below command.\n    ```bash\n    python inference.py --pth ckpt/resnet50_rnn__mp3d.pth --img_glob assets/preprocessed/demo_aligned_rgb.png --output_dir assets/inferenced --visualize\n    ```\n    - `--pth` path to the trained model.\n    - `--img_glob` path to the preprocessed image.\n    - `--output_dir` path to the directory to dump results.\n    - `--visualize` optinoal for visualizing model raw outputs.\n    - `--force_cuboid` add this option if you want to estimate cuboid layout (4 walls).\n- **Outputs**: You will get results like below and prefix with source image basename.\n    - The 1d representation are visualized under file name `[SOURCE BASENAME].raw.png`\n    - The extracted corners of the layout `[SOURCE BASENAME].json`\n        ```\n        {\"z0\": 50.0, \"z1\": -59.03114700317383, \"uv\": [[0.029913906008005142, 0.2996523082256317], [0.029913906008005142, 0.7240479588508606], [0.015625, 0.3819984495639801], [0.015625, 0.6348703503608704], [0.056027885526418686, 0.3881891965866089], [0.056027885526418686, 0.6278984546661377], [0.4480381906032562, 0.3970482349395752], [0.4480381906032562, 0.6178648471832275], [0.5995567440986633, 0.41122356057167053], [0.5995567440986633, 0.601679801940918], [0.8094607591629028, 0.36505699157714844], [0.8094607591629028, 0.6537724137306213], [0.8815288543701172, 0.2661873996257782], [0.8815288543701172, 0.7582473754882812], [0.9189453125, 0.31678876280784607], [0.9189453125, 0.7060701847076416]]}\n        ```\n\n\n### 3. Layout 3D Viewer\n- **Execution**: Visualizing the predicted layout in 3D using points cloud.\n    ```bash\n    python layout_viewer.py --img assets/preprocessed/demo_aligned_rgb.png --layout assets/inferenced/demo_aligned_rgb.json --ignore_ceiling\n    ```\n    - `--img` path to preprocessed image\n    - `--layout` path to the json output from `inference.py`\n    - `--ignore_ceiling` prevent showing ceiling\n    - See `python layout_viewer.py -h` for usage help.\n- **Outputs**: In the window, you can use mouse and scroll wheel to change the viewport\n    - ![](assets/demo_3d_layout.jpg)\n\n\n## Your own dataset\nSee [tutorial](README_PREPARE_DATASET.md) on how to prepare it.\n\n\n## Training\nTo train on a dataset, see `python train.py -h` for detailed options explaination.\\\nExample:\n```bash\npython train.py --id resnet50_rnn\n```\n- Important arguments:\n    - `--id` required. experiment id to name checkpoints and logs\n    - `--ckpt` folder to output checkpoints (default: ./ckpt)\n    - `--logs` folder to logging (default: ./logs)\n    - `--pth` finetune mode if given. path to load saved checkpoint.\n    - `--backbone` backbone of the network (default: resnet50)\n        - other options: `{resnet18,resnet34,resnet50,resnet101,resnet152,resnext50_32x4d,resnext101_32x8d,densenet121,densenet169,densenet161,densenet201}`\n    - `--no_rnn` whether to remove rnn (default: False)\n    - `--train_root_dir` root directory to training dataset. (default: `data/layoutnet_dataset/train`)\n    - `--valid_root_dir` root directory to validation dataset. (default: `data/layoutnet_dataset/valid/`)\n        - If giveng, the epoch with best 3DIoU on validation set will be saved as `{ckpt}/{id}/best_valid.pth`\n    - `--batch_size_train` training mini-batch size (default: 4)\n    - `--epochs` epochs to train (default: 300)\n    - `--lr` learning rate (default: 0.0001)\n    - `--device` set CUDA enabled device using device id (not to be used if multi_gpu is used)\n    - `--multi_gpu` enable parallel computing on all available GPUs\n\n\n## Quantitative Evaluation - Cuboid Layout\nTo evaluate on PanoContext/Stanford2d3d dataset, first running the cuboid trained model for all testing images:\n```bash\npython inference.py --pth ckpt/resnet50_rnn__panos2d3d.pth --img_glob \"data/layoutnet_dataset/test/img/*\" --output_dir output/panos2d3d/resnet50_rnn/ --force_cuboid\n```\n- `--img_glob` shell-style wildcards for all testing images.\n- `--output_dir` path to the directory to dump results.\n- `--force_cuboid` enfoce output cuboid layout (4 walls) or the PE and CE can't be evaluated.\n\nTo get the quantitative result:\n```bash\npython eval_cuboid.py --dt_glob \"output/panos2d3d/resnet50_rnn/*json\" --gt_glob \"data/layoutnet_dataset/test/label_cor/*txt\"\n```\n- `--dt_glob` shell-style wildcards for all the model estimation.\n- `--gt_glob` shell-style wildcards for all the ground truth.\n\nIf you want to:\n- just evaluate PanoContext only `python eval_cuboid.py --dt_glob \"output/panos2d3d/resnet50_rnn/*json\" --gt_glob \"data/layoutnet_dataset/test/label_cor/pano*txt\"`\n- just evaluate Stanford2d3d only `python eval_cuboid.py --dt_glob \"output/panos2d3d/resnet50_rnn/*json\" --gt_glob \"data/layoutnet_dataset/test/label_cor/camera*txt\"`\n\n:clipboard: The quantitative result for the released `resnet50_rnn__panos2d3d.pth` is shown below:\n\n| Testing Dataset | 3D IoU(%) | Corner error(%) | Pixel error(%) |\n| :-------------: | :-------: | :------: | :--------------: |\n| PanoContext     | `83.39` | `0.76` | `2.13` |\n| Stanford2D3D    | `84.09` | `0.63` | `2.06` |\n| All             | `83.87` | `0.67` | `2.08` |\n\n\n## Quantitative Evaluation - General Layout\n- See [the report :clipboard: on ST3D](README_ST3D.md) for more detail.\n- See [the report :clipboard: on MP3D](README_MP3D.md) for more detail.\n\n\n## TODO\n- Faster pre-processing script (top-fron alignment) (maybe cython implementation or [fernandez2018layouts](https://github.com/cfernandezlab/Lines-and-Vanishing-Points-directly-on-Panoramas))\n\n\n## Acknowledgement\n- Credit of this repo is shared with [ChiWeiHsiao](https://github.com/ChiWeiHsiao).\n- Thanks [limchaos](https://github.com/limchaos) for the suggestion about the potential boost by fixing the non-expected behaviour of Pytorch dataloader. (See [Issue#4](https://github.com/sunset1995/HorizonNet/issues/4))\n\n\n## Citation\n```\n@inproceedings{SunHSC19,\n  author    = {Cheng Sun and\n               Chi{-}Wei Hsiao and\n               Min Sun and\n               Hwann{-}Tzong Chen},\n  title     = {HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch\n               Data Augmentation},\n  booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition, {CVPR}\n               2019, Long Beach, CA, USA, June 16-20, 2019},\n  pages     = {1047--1056},\n  year      = {2019},\n}\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunset1995%2Fhorizonnet","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsunset1995%2Fhorizonnet","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsunset1995%2Fhorizonnet/lists"}