{"id":14260809,"url":"https://github.com/anish-bhattacharya/EvDNeRF","last_synced_at":"2025-08-13T09:32:12.183Z","repository":{"id":197876152,"uuid":"699397511","full_name":"anish-bhattacharya/EvDNeRF","owner":"anish-bhattacharya","description":"Event-based dynamic neural radiance fields for generating eventstreams from novel viewpoints and time windows. WACV 2024.","archived":false,"fork":false,"pushed_at":"2024-08-08T22:07:03.000Z","size":2029,"stargazers_count":14,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-08-22T13:14:53.353Z","etag":null,"topics":["3d-vision","event-based-vision","event-cameras","events","neural-implicit-representations","neural-radiance-fields","wacv2024"],"latest_commit_sha":null,"homepage":"https://www.anishbhattacharya.com/research/evdnerf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anish-bhattacharya.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-02T14:54:07.000Z","updated_at":"2024-08-08T22:07:07.000Z","dependencies_parsed_at":"2024-04-06T00:28:29.368Z","dependency_job_id":"8aa976f6-bc19-41fc-8c22-63846b8c4983","html_url":"https://github.com/anish-bhattacharya/EvDNeRF","commit_stats":null,"previous_names":["anish-bhattacharya/evdnerf"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anish-bhattacharya%2FEvDNeRF","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anish-bhattacharya%2FEvDNeRF/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anish-bhattacharya%2FEvDNeRF/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anish-bhattacharya%2FEvDNeRF/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anish-bhattacharya","download_url":"https://codeload.github.com/anish-bhattacharya/EvDNeRF/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":229754465,"owners_count":18119124,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d-vision","event-based-vision","event-cameras","events","neural-implicit-representations","neural-radiance-fields","wacv2024"],"created_at":"2024-08-22T13:00:33.825Z","updated_at":"2024-12-14T20:30:33.500Z","avatar_url":"https://github.com/anish-bhattacharya.png","language":"Python","funding_links":[],"categories":["2024"],"sub_categories":["Real"],"readme":"# EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields\n\n\u003cfont size=\"3\"\u003e\u003cu\u003e[Project page](https://www.anishbhattacharya.com/research/evdnerf)\u003c/u\u003e\u003c/font\u003e\n\nWe train dynamic neural radiance fields to predict eventstreams from novel viewpoints and timestamps of a given scene. We share the code for data generation of simulated and real data, as well as code to train and test EvDNeRF models. This codebase was used for the paper EvDNeRF: Reconstructing Event Data with Dynamic Neural Radiance Fields.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"media/jd.gif\" alt=\"Jet-Down render\" width=\"200\" height=\"200\" /\u003e\n  \u003cimg src=\"media/multi.gif\" alt=\"Multi render\" width=\"200\" height=\"200\" /\u003e\n  \u003cimg src=\"media/real-fork.gif\" alt=\"Real-Fork render\" width=\"271\" height=\"200\" /\u003e\n\u003c/p\u003e\n\n## Installation\n\nFirst, clone the package and make a couple necessary directories:\n```\ngit clone git@github.com:anish-bhattacharya/EvDNeRF.git\ncd EvDNeRF\nmkdir data logs\n```\n\n### Preferred method (docker)\n\nWe provide a docker container that can be pulled from dockerhub.\n```\ndocker pull evdnerf/evdnerf:dev-user\n```\n\n```\ndocker run -d --rm -it --gpus all -v /path/to/EvDNeRF:/home/user/EvDNeRF evdnerf/evdnerf:dev-user\n```\n\nThis will run a docker container image, the name of which can be seen in the output of `docker ps`. Start a shell in the docker container via:\n```\ndocker exec -it -u root \u003crunning_docker_image_name\u003e bash\n```\n\nNow you should be able to run an example train or test configuration within this docker container (see following sections).\n\n### Alternate method (conda env, borrowed from D-NeRF)\n\n(Similar to [D-NeRF installation](https://github.com/albertpumarola/D-NeRF)).\nFrom inside the EvDNeRF directory:\n```\nconda create -n evdnerf python=3.6\nconda activate evdnerf\npip install -r requirements.txt\ncd torchsearchsorted\npip install .\ncd ..\n```\n\nYou will have to install additional dependencies manually; we have not provided an updated requirements.txt yet.\n\n## Train\n\n### Datasets\n\nWe provide datasets `public_datasets` at the following link to train EvDNeRF models from simulated and real datasets. After extracting the datasets tar, please read the contained, short README.\n\n[Data Link](https://upenn.box.com/v/EvDNeRF-datashare)\n\n### Example training experiment\n\nTo train from the `jet-down-32` dataset, we will use the `configs/jet_train.txt` config file. Edit the `root_path` and `datadir` to point to your `EvDNeRF/` directory and inside the `public_datasets` directory, respectively.\n\nYou may add a `ft_file` line (refer to the test config file for an example) if you wish to continue training from a weights checkpoint. You may also change the lines `i_print` (prints training stats to console every `i_print` iterations), `i_img` (runs validation and saves event images ang grayscale images every `i_img` iterations), `testskip` (one of how many frames to render when doing validation), `i_weights` (saves weights every `i_weights` iterations), `N_iter` (how many iterations to train). We present the best results on simulated data, for example, by training for 100k iterations on 32 event frames (e.g., `jet-down-32` dataset), then 50k on 64 (e.g., `jet-down-64` dataset), and 50k on 128 (e.g., `jet-down-128` dataset). 100k on 32 and 100k on 128 also yields similar results.\n\nRun the following command to start training:\n```\npython run_evdnerf.py --config configs/jet_train.txt\n```\nThis command will run training on the `jet-*` dataset. Progressive weights and validation are saved to a log folder: `./logs/\u003cexp_datetime\u003e`. Any weights file absolute path can then be inputted as the `ft_file` argument to test on.\n\nLoss and other training statistics are logged with tensorboard. You can view all logged runs in tensorboard by running the following from inside `EvDNeRF/`, then opening `http://localhost:8000/` in your browser.\n```\ntensorboard --logdir logs --port 8000\n```\n\n## Test\n\n### Pretrained weights\n\nFirst download pretrained weights. They should placed in the directory `EvDNeRF/pretrained_weights`. This is the same link as the one above.\n\n[Data Link](https://upenn.box.com/v/EvDNeRF-datashare)\n\n### Example test experiment\n\nThere are various testing configurations, including event frame generation directly from `json` files specifying poses and timestamps (via setting `render_test_path`), or calculating and comparing metrics against an established test set (via setting `render_test`).\n\nFor an example, we set `render_test_path` to render predicted events for any defined `json` file. Various such files are found in `test_configs/`. `test_configs/transforms_test_validation.json` contains validation viewpoints each with 32 timesteps. To test a model trained on the `jet-down` dataset, we will use the `configs/jet_test.txt` config file, where `ft_file` points to the corresponding pretrained weights file. Edit the `root_path` and `datadir` in the config file to point to your `EvDNeRF/` directory and inside the `public_datasets` directory, respectively.\n\nTo test the model, run:\n```\npython run_evdnerf.py --config configs/jet_test.txt\n```\nWhen finished, resulting event batches in the form of images are found in `./logs/\u003cexp_datetime\u003e/\u003cexp_datetime\u003e/evim/*.png`.\n\n## Data generation\n\nComing soon for simulated and real data generation!\n\n## Acknowledgements\n\nThis codebase is based on the [implementation of D-NeRF](https://github.com/albertpumarola/D-NeRF).\n\n## Citation\n```\n@inproceedings{bhattacharya2024evdnerf,\n  title={Evdnerf: Reconstructing event data with dynamic neural radiance fields},\n  author={Bhattacharya, Anish and Madaan, Ratnesh and Cladera, Fernando and Vemprala, Sai and Bonatti, Rogerio and Daniilidis, Kostas and Kapoor, Ashish and Kumar, Vijay and Matni, Nikolai and Gupta, Jayesh K},\n  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},\n  pages={5846--5855},\n  year={2024}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanish-bhattacharya%2FEvDNeRF","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanish-bhattacharya%2FEvDNeRF","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanish-bhattacharya%2FEvDNeRF/lists"}