{"id":13844398,"url":"https://github.com/jingweiz/pytorch-dnc","last_synced_at":"2025-07-11T22:31:59.719Z","repository":{"id":118625789,"uuid":"90908029","full_name":"jingweiz/pytorch-dnc","owner":"jingweiz","description":"Neural Turing Machine (NTM) \u0026 Differentiable Neural Computer (DNC) with pytorch \u0026 visdom","archived":false,"fork":false,"pushed_at":"2018-02-20T10:38:57.000Z","size":3376,"stargazers_count":278,"open_issues_count":1,"forks_count":52,"subscribers_count":23,"default_branch":"master","last_synced_at":"2024-08-05T17:41:44.116Z","etag":null,"topics":["deep-learning","dnc","external-memory","ntm","pytorch","visdom"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jingweiz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-05-10T21:01:44.000Z","updated_at":"2024-05-10T11:50:14.000Z","dependencies_parsed_at":null,"dependency_job_id":"e05ee506-d9ca-4615-8d08-c9cdffc8dac9","html_url":"https://github.com/jingweiz/pytorch-dnc","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-dnc","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-dnc/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-dnc/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jingweiz%2Fpytorch-dnc/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jingweiz","download_url":"https://codeload.github.com/jingweiz/pytorch-dnc/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225763397,"owners_count":17520454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","dnc","external-memory","ntm","pytorch","visdom"],"created_at":"2024-08-04T17:02:41.664Z","updated_at":"2024-11-21T16:31:26.199Z","avatar_url":"https://github.com/jingweiz.png","language":"Python","funding_links":[],"categories":["Python (1887)","Python","Paper implementations｜论文实现","Paper implementations"],"sub_categories":["Other libraries｜其他库:","Other libraries:"],"readme":"# **Neural Turing Machine** (NTM) \u0026\n# **Differentiable Neural Computer** (DNC) with\n# **pytorch** \u0026 **visdom**\n*******\n\n\n* Sample on-line plotting while training(avg loss)/testing(write/read weights \u0026 memory)\n  NTM on the copy task (top 2 rows, 1st row converges to sequentially write to lower locations, 2nd row converges to sequentially write to upper locations) and\n  DNC on the repeat-copy task (3rd row) (the write/read weights here are after location focus so are no longer necessarily normalized within each head by design):\n\n\u003cimg src=\"/assets/ntm_copy_train_revised_16_0.png\" width=\"205\"/\u003e \u003cimg src=\"/assets/ntm_copy_test_revised_16_0.gif\" width=\"600\"/\u003e\n\u003cimg src=\"/assets/ntm_copy_train_revised_16_1.png\" width=\"205\"/\u003e \u003cimg src=\"/assets/ntm_copy_test_revised_16_1.gif\" width=\"600\"/\u003e\n\u003cimg src=\"/assets/dnc_repeat_copy_train_revised_tanh.png\" width=\"205\"/\u003e \u003cimg src=\"/assets/dnc_repeat_copy_test_revised.gif\" width=\"600\"/\u003e\n\n\n* Sample loggings while training DNC on the repeat-copy task (we use ```WARNING``` as the logging level currently to get rid of the ```INFO``` printouts from visdom):\n```bash\n[WARNING ] (MainProcess) \u003c===================================\u003e\n[WARNING ] (MainProcess) bash$: python -m visdom.server\n[WARNING ] (MainProcess) http://localhost:8097/env/daim_17051000\n[WARNING ] (MainProcess) \u003c===================================\u003e Agent:\n[WARNING ] (MainProcess) \u003c-----------------------------======\u003e Env:\n[WARNING ] (MainProcess) Creating {repeat-copy | } w/ Seed: 123\n[WARNING ] (MainProcess) Word     {length}:   {4}\n[WARNING ] (MainProcess) Words #  {min, max}: {1, 2}\n[WARNING ] (MainProcess) Repeats  {min, max}: {1, 2}\n[WARNING ] (MainProcess) \u003c-----------------------------======\u003e Circuit:    {Controller, Accessor}\n[WARNING ] (MainProcess) \u003c--------------------------------===\u003e Controller:\n[WARNING ] (MainProcess) LSTMController (\n  (in_2_hid): LSTMCell(70, 64, bias=1)\n)\n[WARNING ] (MainProcess) \u003c--------------------------------===\u003e Accessor:   {WriteHead, ReadHead, Memory}\n[WARNING ] (MainProcess) \u003c-----------------------------------\u003e WriteHeads: {1 heads}\n[WARNING ] (MainProcess) DynamicWriteHead (\n  (hid_2_key): Linear (64 -\u003e 16)\n  (hid_2_beta): Linear (64 -\u003e 1)\n  (hid_2_alloc_gate): Linear (64 -\u003e 1)\n  (hid_2_write_gate): Linear (64 -\u003e 1)\n  (hid_2_erase): Linear (64 -\u003e 16)\n  (hid_2_add): Linear (64 -\u003e 16)\n)\n[WARNING ] (MainProcess) \u003c-----------------------------------\u003e ReadHeads:  {4 heads}\n[WARNING ] (MainProcess) DynamicReadHead (\n  (hid_2_key): Linear (64 -\u003e 64)\n  (hid_2_beta): Linear (64 -\u003e 4)\n  (hid_2_free_gate): Linear (64 -\u003e 4)\n  (hid_2_read_mode): Linear (64 -\u003e 12)\n)\n[WARNING ] (MainProcess) \u003c-----------------------------------\u003e Memory:     {16(batch_size) x 16(mem_hei) x 16(mem_wid)}\n[WARNING ] (MainProcess) \u003c-----------------------------======\u003e Circuit:    {Overall Architecture}\n[WARNING ] (MainProcess) DNCCircuit (\n  (controller): LSTMController (\n    (in_2_hid): LSTMCell(70, 64, bias=1)\n  )\n  (accessor): DynamicAccessor (\n    (write_heads): DynamicWriteHead (\n      (hid_2_key): Linear (64 -\u003e 16)\n      (hid_2_beta): Linear (64 -\u003e 1)\n      (hid_2_alloc_gate): Linear (64 -\u003e 1)\n      (hid_2_write_gate): Linear (64 -\u003e 1)\n      (hid_2_erase): Linear (64 -\u003e 16)\n      (hid_2_add): Linear (64 -\u003e 16)\n    )\n    (read_heads): DynamicReadHead (\n      (hid_2_key): Linear (64 -\u003e 64)\n      (hid_2_beta): Linear (64 -\u003e 4)\n      (hid_2_free_gate): Linear (64 -\u003e 4)\n      (hid_2_read_mode): Linear (64 -\u003e 12)\n    )\n  )\n  (hid_to_out): Linear (128 -\u003e 5)\n)\n[WARNING ] (MainProcess) No Pretrained Model. Will Train From Scratch.\n[WARNING ] (MainProcess) \u003c===================================\u003e Training ...\n[WARNING ] (MainProcess) Reporting       @ Step: 500 | Elapsed Time: 30.609361887\n[WARNING ] (MainProcess) Training Stats:   avg_loss:         0.014866309287\n[WARNING ] (MainProcess) Evaluating      @ Step: 500\n[WARNING ] (MainProcess) Evaluation        Took: 1.6457400322\n[WARNING ] (MainProcess) Iteration: 500; loss_avg: 0.0140423600748\n[WARNING ] (MainProcess) Saving Model    @ Step: 500: /home/zhang/ws/17_ws/pytorch-dnc/models/daim_17051000.pth ...\n[WARNING ] (MainProcess) Saved  Model    @ Step: 500: /home/zhang/ws/17_ws/pytorch-dnc/models/daim_17051000.pth.\n[WARNING ] (MainProcess) Resume Training @ Step: 500\n...\n```\n*******\n\n\n## What is included?\nThis repo currently contains the following algorithms:\n\n- Neural Turing Machines (NTM) [[1]](https://arxiv.org/abs/1410.5401)\n- Differentiable Neural Computers (DNC) [[2]](http://www.nature.com/nature/journal/v538/n7626/full/nature20101.html)\n\nTasks:\n- copy\n- repeat-copy\n\n## Code structure \u0026 Naming conventions\nNOTE: we follow the exact code structure as [pytorch-rl](https://github.com/jingweiz/pytorch-rl) so as to make the code easily transplantable.\n* ```./utils/factory.py```\n\u003e We suggest the users refer to ```./utils/factory.py```,\n where we list all the integrated ```Env```, ```Circuit```, ```Agent``` into ```Dict```'s.\n All of the core classes are implemented in ```./core/```.\n The factory pattern in ```./utils/factory.py``` makes the code super clean,\n as no matter what type of ```Circuit``` you want to train,\n or which type of ```Env``` you want to train on,\n all you need to do is to simply modify some parameters in ```./utils/options.py```,\n then the ```./main.py``` will do it all (NOTE: this ```./main.py``` file never needs to be modified).\n* namings\n\u003e To make the code more clean and readable, we name the variables using the following pattern:\n\u003e * ```*_vb```: ```torch.autograd.Variable```'s or a list of such objects\n\u003e * ```*_ts```: ```torch.Tensor```'s or a list of such objects\n\u003e * otherwise: normal python datatypes\n\n\n## Dependencies\n- Python 2.7\n- [PyTorch \u003e=v0.2.0](http://pytorch.org/)\n- [Visdom](https://github.com/facebookresearch/visdom)\n*******\n\n\n## How to run:\nYou only need to modify some parameters in ```./utils/options.py``` to train a new configuration.\n\n* Configure your training in ```./utils/options.py```:\n\u003e * ```line 12```: add an entry into ```CONFIGS``` to define your training (```agent_type```, ```env_type```, ```game```, ```circuit_type```)\n\u003e * ```line 28```: choose the entry you just added\n\u003e * ```line 24-25```: fill in your machine/cluster ID (```MACHINE```) and timestamp (```TIMESTAMP```) to define your training signature (```MACHINE_TIMESTAMP```),\n the corresponding model file and the log file of this training will be saved under this signature (```./models/MACHINE_TIMESTAMP.pth``` \u0026 ```./logs/MACHINE_TIMESTAMP.log``` respectively).\n Also the visdom visualization will be displayed under this signature (first activate the visdom server by type in bash: ```python -m visdom.server \u0026```, then open this address in your browser: ```http://localhost:8097/env/MACHINE_TIMESTAMP```)\n\u003e * ```line 28```: to train a model, set ```mode=1``` (training visualization will be under ```http://localhost:8097/env/MACHINE_TIMESTAMP```); to test the model of this current training, all you need to do is to set ```mode=2``` (testing visualization will be under ```http://localhost:8097/env/MACHINE_TIMESTAMP_test```).\n\n* Run:\n\u003e ```python main.py```\n*******\n\n\n## Implementation Notes:\nThe difference between ```NTM``` \u0026 ```DNC``` is stated as follows in the\n```DNC```[2] paper:\n\u003e Comparison with the neural Turing machine. The neural Turing machine (NTM) was\nthe predecessor to the DNC described in this work. It used a similar\narchitecture of neural network controller with read–write access to a memory\nmatrix, but differed in the access mechanism used to interface with the memory.\nIn the NTM, content-based addressing was combined with location-based addressing\nto allow the network to iterate through memory locations in order of their\nindices (for example, location n followed by n+1 and so on). This allowed the\nnetwork to store and retrieve temporal sequences in contiguous blocks of memory.\nHowever, there were several drawbacks. First, the NTM has no mechanism to ensure\nthat blocks of allocated memory do not overlap and interfere—a basic problem of\ncomputer memory management. Interference is not an issue for the dynamic memory\nallocation used by DNCs, which provides single free locations at a time,\nirrespective of index, and therefore does not require contiguous blocks. Second,\nthe NTM has no way of freeing locations that have already been written to and,\nhence, no way of reusing memory when processing long sequences. This problem is\naddressed in DNCs by the free gates used for de-allocation. Third, sequential\ninformation is preserved only as long as the NTM continues to iterate through\nconsecutive locations; as soon as the write head jumps to a different part of\nthe memory (using content-based addressing) the order of writes before and after\nthe jump cannot be recovered by the read head. The temporal link matrix used by\nDNCs does not suffer from this problem because it tracks the order in which\nwrites were made.\n\nWe thus make some effort to put those two together in a combined codebase.\nThe classes implemented have the following hierarchy:\n* Agent\n  * Env\n  * Circuit\n    * Controller\n    * Accessor\n      * WriteHead\n      * ReadHead\n      * Memory\n\nThe part where ```NTM``` \u0026 ```DNC``` differs is the ```Accessor```, where in the\ncode ```NTM``` uses the ```StaticAccessor```(may not be an appropriate name but\nwe use this to make the code more consistent) and ```DNC``` uses the\n```DynamicAccessor```. Both ```Accessor``` classes use ```_content_focus()```\nand ```_location_focus()```(may not be an appropriate name for ```DNC``` but we\nuse this to make the code more consistent). The ```_content_focus()``` is the\nsame for both classes, but the ```_location_focus()``` for ```DNC``` is much\nmore complicated as it uses ```dynamic allocation``` additionally for write and\n```temporal link``` additionally for read. Those focus (or attention) mechanisms\nare implemented in ```Head``` classes, and those focuses output a ```weight```\nvector for each ```head``` (write/read). Those ```weight``` vectors are then used in\n```_access()``` to interact with the ```external memory```.\n\n## A side note:\nThe sturcture for ```Env``` might look strange as this class was originally\ndesigned for ```reinforcement learning``` settings as in\n[pytorch-rl](https://github.com/jingweiz/pytorch-rl); here we use it for\nproviding datasets for ```supervised learning```, so the ```reward```,\n```action``` and ```terminal``` are always left blank in this repo.\n*******\n\n\n## Repos we referred to during the development of this repo:\n* [deepmind/dnc](https://github.com/deepmind/dnc)\n* [ypxie/pytorch-NeuCom](https://github.com/ypxie/pytorch-NeuCom)\n* [bzcheeseman/pytorch-EMM](https://github.com/bzcheeseman/pytorch-EMM)\n* [DoctorTeeth/diffmem](https://github.com/DoctorTeeth/diffmem)\n* [kaishengtai/torch-ntm](https://github.com/kaishengtai/torch-ntm)\n* [Mostafa-Samir/DNC-tensorflow](https://github.com/Mostafa-Samir/DNC-tensorflow)\n*******\n\n## The following paper might be interesting to take a look:)\n\u003e [Neural SLAM](https://arxiv.org/abs/1706.09520): We present an approach for agents to learn representations of a global map from sensor data, to aid their exploration in new environments. To achieve this, we embed procedures mimicking that of traditional Simultaneous Localization and Mapping (SLAM) into the soft attention based addressing of external memory architectures, in which the external memory acts as an internal representation of the environment. This structure encourages the evolution of SLAM-like behaviors inside a completely differentiable deep neural network. We show that this approach can help reinforcement learning agents to successfully explore new environments where long-term memory is essential. We validate our approach in both challenging grid-world environments and preliminary Gazebo experiments. A video of our experiments can be found at: \\url{https://goo.gl/RfiSxo}.\n\n```\n@article{zhang2017neural,\n  title={Neural SLAM},\n  author={Zhang, Jingwei and Tai, Lei and Boedecker, Joschka and Burgard, Wolfram and Liu, Ming},\n  journal={arXiv preprint arXiv:1706.09520},\n  year={2017}\n}\n\n```\n*******\n\n\n## Citation\nIf you find this library useful and would like to cite it, the following would be appropriate:\n```\n@misc{pytorch-dnc,\n  author = {Zhang, Jingwei},\n  title = {jingweiz/pytorch-dnc},\n  url = {https://github.com/jingweiz/pytorch-dnc},\n  year = {2017}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjingweiz%2Fpytorch-dnc","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjingweiz%2Fpytorch-dnc","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjingweiz%2Fpytorch-dnc/lists"}