{"id":18009135,"url":"https://github.com/kwea123/miner_pl","last_synced_at":"2025-07-26T20:05:51.156Z","repository":{"id":50671255,"uuid":"488063077","full_name":"kwea123/MINER_pl","owner":"kwea123","description":"Unofficial implementation (replicates paper results!) of MINER: Multiscale Implicit Neural Representations in pytorch-lightning","archived":false,"fork":false,"pushed_at":"2022-10-30T00:11:32.000Z","size":15819,"stargazers_count":61,"open_issues_count":0,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-21T22:22:03.510Z","etag":null,"topics":["coordinate-mlp","gaussian-activation","image-reconstruction","miner","neural-fields","pytorch-lightning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kwea123.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-05-03T03:12:41.000Z","updated_at":"2025-03-18T11:16:06.000Z","dependencies_parsed_at":"2023-01-20T11:18:00.313Z","dependency_job_id":null,"html_url":"https://github.com/kwea123/MINER_pl","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2FMINER_pl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2FMINER_pl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2FMINER_pl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kwea123%2FMINER_pl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kwea123","download_url":"https://codeload.github.com/kwea123/MINER_pl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245662929,"owners_count":20652100,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["coordinate-mlp","gaussian-activation","image-reconstruction","miner","neural-fields","pytorch-lightning"],"created_at":"2024-10-30T02:08:48.441Z","updated_at":"2025-03-26T13:31:46.990Z","avatar_url":"https://github.com/kwea123.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MINER_pl\nUnofficial implementation of [MINER: Multiscale Implicit Neural Representations](https://arxiv.org/pdf/2202.03532.pdf) in pytorch-lightning.\n\n**Official implementation** : https://github.com/vishwa91/miner\n\n![image](https://user-images.githubusercontent.com/11364490/168208863-656a0a7d-35d9-4b9b-86f9-d52da4182e35.png)\n\n# :open_book: Ref readings\n\n*  [Laplacian pyramid explanation](https://paperswithcode.com/method/laplacian-pyramid)\n\n*  My explanatory videos\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://youtu.be/cXZtbfjnJtA\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/11364490/168209075-330d879e-2bff-467f-bf31-4e0ad2809777.png\", width=\"45%\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://youtu.be/MSVEhq67Ca4\"\u003e\n    \u003cimg src=\"https://user-images.githubusercontent.com/11364490/168209233-4bde51ba-df6d-4fdb-87d6-9704986c1248.png\", width=\"45%\"\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n# :warning: Main differences w.r.t. the original paper before continue:\n*  In the pseudo code on page 8, where the author states **Weight sharing for images**, it means finer level networks are initialized with coarser level network weights. However, I did not find the correct way to implement this. Therefore, I initialize the network weights from scratch for all levels.\n*  The paper says it uses sinusoidal activation (does he mean SIREN? I don't know), but I use gaussian activation (in hidden layers) with trainable parameters (per block) like my experiments in [the other repo](https://github.com/kwea123/Coordinate-MLPs). In finer levels where the model predicts laplacian pyramids, I use sinusoidal activation `x |-\u003e sin(ax)` with trainable parameters `a` (per block) as output layer (btw, this performs *significantly* better than simple `tanh`). Moreover, I precompute the maximum amplitude for laplacian residuals, and use it to scale the output, and I find it to be better than without scaling.\n*  I experimented with a common trick in coordinate mlp: *[positional encoding](https://github.com/tancik/fourier-feature-networks)* and find that using it can increase training time/accuracy with the same number of parameters (by reducing 1 layer). This can be turned on/off by specifying the argument `--use_pe`. The optimal number of frequencies depends on the patch size, the larger patch sizes, the more number of frequencies you need and vice versa.\n*  Some difference in the hyperparameters: the default learning rate is `3e-2` instead of `5e-4`. Optimizer is `RAdam` instead of `Adam`. Block pruning happens when the loss is lower than `1e-4` (i.e. when PSNR\u003e=40) for image and `5e-3` for occupancy rather than `2e-7`.\n\n# :computer: Installation\n\n*  Run `pip install -r requirements.txt`.\n*  Download the images from [Acknowledgement](#gift_heart-acknowledgement) or prepare your own images into a folder called `images`.\n*  Download the meshes from [Acknowledgement](#gift_heart-acknowledgement) or prepare your own meshes into a folder called `meshes`.\n\n# :key: Training\n\n\u003cdetails\u003e\n  \u003csummary\u003e\u003ch2\u003eimage\u003c/summary\u003e\n\nPluto example:\n```python3\npython train.py \\\n    --task image --path images/pluto.png \\\n    --input_size 4096 4096 --patch_size 32 32 --batch_size 256 --n_scales 4 \\\n    --use_pe --n_layers 3 \\\n    --num_epochs 50 50 50 200 \\\n    --exp_name pluto4k_4scale\n```\n\nTokyo station example:\n```python3\npython train.py \\\n    --task image --path images/tokyo-station.jpg \\\n    --input_size 6000 4000 --patch_size 25 25 --batch_size 192 --n_scales 5 \\\n    --use_pe --n_layers 3 \\\n    --num_epochs 50 50 50 50 150 \\\n    --exp_name tokyo6k_5scale\n```\n\n| Image (size) | Train time (s) | GPU mem (MiB) | #Params (M) | PSNR |\n|:---:|:---:|:---:|:---:|:---:|\n| Pluto (4096x4096) | 53 | 3171 | 9.16 | 42.14 |\n| Pluto (8192x8192) | 106 | 6099 | 28.05 | 45.09 |\n| Tokyo station (6000x4000) | 68 | 6819 | 35.4 | 42.48 |\n| Shibuya (7168x2560) | 101 | 8967 | 17.73 | 37.78 |\n| Shibuya (14336x5120) | 372 | 8847 | 75.42 | 39.32 |\n| Shibuya (28672x10240) | 890 | 10255 | 277.37 | 41.93 |\n| Shibuya (28672x10240)* | 1244 | 6277 | 98.7 | 37.59 |\n\n*paper settings (6 scales, each network has 4 layer with 9 hidden units)\n\nThe original image will be resized to `img_wh` for reconstruction. You need to make sure `img_wh` divided by `2^(n_scales-1)` (the resolution at the coarsest level) is still a multiple of `patch_wh`.\n\n------------------------------------------\n\n\u003c/details\u003e\n  \n\u003cdetails\u003e\n  \u003csummary\u003e\u003ch2\u003emesh\u003c/summary\u003e\n\nFirst, convert the mesh to N^3 occupancy grid by\n```python3\npython preprocess_mesh.py --N 512 --M 1 --T 1 --path \u003cpath/to/mesh\u003e \n```\nThis will create N^3 occupancy to be regressed by the neural network.\nFor detailed options, please see [preprocess_mesh.py](preprocess_mesh.py). Typically, increase `M` or `T` if you find the resulting occupancy bad.\n\nNext, start training (bunny example):\n```python3\npython train.py \\\n    --task mesh --path occupancy/bunny_512.npy \\\n    --input_size 512 --patch_size 16 --batch_size 512 --n_scales 4 \\\n    --use_pe --n_freq 5 --n_layers 2 --n_hidden 8 \\\n    --loss_thr 5e-3 --b_chunks 512 \\\n    --num_epochs 50 50 50 150 \\\n    --exp_name bunny512_4scale\n```\n\n------------------------------------------\n\n\u003c/details\u003e\n\nFor full options, please see [here](opt.py). Some important options:\n\n*  If your GPU memory is not enough, try reducing `batch_size`.\n*  By default it will not log intermediate images to tensorboard to save time. To visualize image reconstruction and active blocks, add `--log_image` argument.\n\nYou are recommended to monitor the training progress by\n```\ntensorboard --logdir logs\n```\n\nwhere you can see training curves and images.\n\n# :red_square::green_square::blue_square: Block decomposition\n\nTo reconstruct the image using trained model and to visualize block decomposition per scale like Fig. 4 in the paper, see [image_test.ipynb](image_test.ipynb) or [mesh_test.ipynb](mesh_test.ipynb)\n\n\u003c!-- Pretrained models can be downloaded from [releases](https://github.com/kwea123/MINER_pl/releases). --\u003e\n\nExamples:\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/168275200-e625d828-61df-4ff2-a658-7dd10e123847.jpg\", width=\"45%\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/168275208-a35e828d-0ca0-408f-90c3-89dd97d108ba.jpg\", width=\"45%\"\u003e\n\u003c/p\u003e\n    \n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/169640414-9da542dc-2df5-4a46-b80a-9591e86f98b3.jpg\", width=\"30%\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/169640416-59e4391f-7377-4b7e-b103-715b25d7253c.jpg\", width=\"30%\"\u003e\n  \u003cimg src=\"https://user-images.githubusercontent.com/11364490/169640742-49f4a43e-4705-4463-bbe4-822839220ddd.jpg\", width=\"30%\"\u003e\n\u003c/p\u003e\n\n# :bulb: Implementation tricks\n\n*  Setting `num_workers=0` in dataloader increased the speed a lot.\n*  As suggested in *training details* on page 4, I implement parallel block inference by defining parameters of shape `(n_blocks, n_in, n_out)` and use `@` operator (same as `torch.bmm`) for faster inference.\n*  To perform block pruning efficiently, I create two copies of the same network, and continually train and prune one of them while copying the trained parameters to the target network (somehow like in reinforcement learning, e.g. DDPG). This allows the network as well as the optimizer to shrink, therefore achieve higher memory and speed performance.\n*  In validation, I perform inference in chunks like NeRF, and pass each chunk to cpu to reduce GPU memory usage.\n\n# :gift_heart: Acknowledgement\n\n*  Pluto image: [NASA](https://solarsystem.nasa.gov/resources/933/true-colors-of-pluto/?category=planets/dwarf-planets_pluto)\n\n*  Shibuya image: [Trevor Dobson](https://www.flickr.com/photos/trevor_dobson_inefekt69/29314390837)\n\n*  Tokyo station image: [baroparo](https://pixabay.com/photos/tokyo-station-tokyo-station-japan-641769/?download)\n\n*  [Stanford scanning](http://graphics.stanford.edu/data/3Dscanrep/)\n\n*  [Turbosquid](https://www.turbosquid.com/)\n\n# :question: Further readings\n\nDuring a stream, my audience suggested me to test on this image with random pixels:\n\n![random](https://user-images.githubusercontent.com/11364490/168567099-56c226a8-9f79-4710-9291-cc7ecef26f6d.png)\n\nThe default `32x32` patch size doesn't work well, since the texture varies too quickly inside a patch. Decreasing to `16x16` and increasing network hidden units make the network converge right away to `43.91 dB` under a minute. Surprisingly, with the other image reconstruction SOTA [instant-ngp](https://github.com/NVlabs/instant-ngp#image-of-einstein), the network is stuck at `17 dB` no matter how long I train.\n\n![ngp-random](https://user-images.githubusercontent.com/11364490/168567783-40a9c123-01af-472c-9885-e52a778df18e.png)\n\nIs this a possible weakness of instant-ngp? What effect could it bring to real application? You are welcome to test other methods to reconstruct this image!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwea123%2Fminer_pl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkwea123%2Fminer_pl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkwea123%2Fminer_pl/lists"}