{"id":13474293,"url":"https://github.com/kevinzakka/recurrent-visual-attention","last_synced_at":"2025-03-26T21:31:13.218Z","repository":{"id":106954036,"uuid":"111703103","full_name":"kevinzakka/recurrent-visual-attention","owner":"kevinzakka","description":"A PyTorch Implementation of \"Recurrent Models of Visual Attention\"","archived":true,"fork":false,"pushed_at":"2023-02-24T04:25:37.000Z","size":21505,"stargazers_count":469,"open_issues_count":18,"forks_count":124,"subscribers_count":14,"default_branch":"master","last_synced_at":"2024-10-30T07:47:37.143Z","etag":null,"topics":["attention","pytorch","ram","recurrent-attention-model","recurrent-models"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kevinzakka.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2017-11-22T15:44:04.000Z","updated_at":"2024-10-21T08:31:15.000Z","dependencies_parsed_at":"2024-01-13T18:24:18.727Z","dependency_job_id":"1a648836-5e38-4f01-9a3c-208c3054f50f","html_url":"https://github.com/kevinzakka/recurrent-visual-attention","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinzakka%2Frecurrent-visual-attention","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinzakka%2Frecurrent-visual-attention/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinzakka%2Frecurrent-visual-attention/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kevinzakka%2Frecurrent-visual-attention/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kevinzakka","download_url":"https://codeload.github.com/kevinzakka/recurrent-visual-attention/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245738586,"owners_count":20664306,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention","pytorch","ram","recurrent-attention-model","recurrent-models"],"created_at":"2024-07-31T16:01:11.164Z","updated_at":"2025-03-26T21:31:12.520Z","avatar_url":"https://github.com/kevinzakka.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Recurrent Visual Attention\n\nThis is a **PyTorch** implementation of [Recurrent Models of Visual Attention](https://arxiv.org/abs/1406.6247) by *Volodymyr Mnih, Nicolas Heess, Alex Graves and Koray Kavukcuoglu*.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"./plots/bbox.png\" alt=\"Drawing\", width=60%\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"./plots/glimpses.png\" alt=\"Drawing\", width=23%\u003e\n\u003c/p\u003e\n\nThe *Recurrent Attention Model* (RAM) is a neural network that processes inputs sequentially, attending to different locations within the image one at a time, and incrementally combining information from these fixations to build up a dynamic internal representation of the image.\n\n## Model Description\n\nIn this paper, the attention problem is modeled as the sequential decision process of a goal-directed agent interacting with a visual environment. The agent is built around a recurrent neural network: at each time step, it processes the sensor data, integrates information over time, and chooses how to act and how to deploy its sensor at the next time step.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"./plots/model.png\" alt=\"Drawing\", width=70%\u003e\n\u003c/p\u003e\n\n- **glimpse sensor**: a retina that extracts a foveated glimpse `phi` around location `l` from an image `x`. It encodes the region around `l` at a high-resolution but uses a progressively lower resolution for pixels further from `l`, resulting in a compressed representation of the original image `x`.\n- **glimpse network**: a network that combines the \"what\" (`phi`) and the \"where\" (`l`) into a glimpse feature vector w`g_t`.\n- **core network**: an RNN that maintains an internal state that integrates information extracted from the history of past observations. It encodes the agent's knowledge of the environment through a state vector `h_t` that gets updated at every time step `t`.\n- **location network**: uses the internal state `h_t` of the core network to produce the location coordinates `l_t` for the next time step.\n- **action network**: after a fixed number of time steps, uses the internal state `h_t` of the core network to produce the final output classification `y`.\n\n## Results\n\nI decided to tackle the `28x28` MNIST task with the RAM model containing 6 glimpses, of size `8x8`, with a scale factor of `1`.\n\n| Model | Validation Error | Test Error |\n|-------|------------------|------------|\n| 6 8x8 | 1.1              | 1.21       |\n\nI haven't done random search on the policy standard deviation to tune it, so I expect the test error can be reduced to sub `1%` error. I'll be updating the table above with results for the `60x60` Translated MNIST, `60x60` Cluttered Translated MNIST and the new Fashion MNIST dataset when I get the time.\n\nFinally, here's an animation showing the glimpses extracted by the network on a random batch at epoch 23.\n\n\u003cp align=\"center\"\u003e\n \u003cimg src=\"./plots/example.gif\" alt=\"Drawing\", width=70%\u003e\n\u003c/p\u003e\n\nWith the Adam optimizer, paper accuracy can be reached in ~160 epochs.\n\n## Usage\n\nThe easiest way to start training your RAM variant is to edit the parameters in `config.py` and run the following command:\n\n```\npython main.py\n```\n\nTo resume training, run:\n\n```\npython main.py --resume=True\n```\n\nFinally, to test a checkpoint of your model that has achieved the best validation accuracy, run the following command:\n\n```\npython main.py --is_train=False\n```\n\n## References\n\n- [Torch Blog Post on RAM](http://torch.ch/blog/2015/09/21/rmva.html)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinzakka%2Frecurrent-visual-attention","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkevinzakka%2Frecurrent-visual-attention","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkevinzakka%2Frecurrent-visual-attention/lists"}