{"id":13936709,"url":"https://github.com/jiamings/fast-weights","last_synced_at":"2025-04-10T20:10:37.360Z","repository":{"id":84351020,"uuid":"72268559","full_name":"jiamings/fast-weights","owner":"jiamings","description":"Implementation of the paper [Using Fast Weights to Attend to the Recent Past](https://arxiv.org/abs/1610.06258)","archived":false,"fork":false,"pushed_at":"2016-11-03T15:49:22.000Z","size":232,"stargazers_count":172,"open_issues_count":1,"forks_count":22,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-24T17:52:45.707Z","etag":null,"topics":["accuracy","deep-learning","fast-weights","tensorflow","tensorflow-models"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jiamings.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-29T06:19:37.000Z","updated_at":"2024-05-29T08:24:48.000Z","dependencies_parsed_at":"2023-03-04T10:00:14.745Z","dependency_job_id":null,"html_url":"https://github.com/jiamings/fast-weights","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiamings%2Ffast-weights","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiamings%2Ffast-weights/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiamings%2Ffast-weights/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jiamings%2Ffast-weights/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jiamings","download_url":"https://codeload.github.com/jiamings/fast-weights/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248288352,"owners_count":21078903,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["accuracy","deep-learning","fast-weights","tensorflow","tensorflow-models"],"created_at":"2024-08-07T23:02:55.704Z","updated_at":"2025-04-10T20:10:37.338Z","avatar_url":"https://github.com/jiamings.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"## Using Fast Weights to Attend to the Recent Past\n\nReproducing the associative model experiment on the paper\n\n[Using Fast Weights to Attend to the Recent Past](https://arxiv.org/abs/1610.06258) by Jimmy Ba et al. (Incomplete)\n\n\n\n### Prerequisites\n\nTensorflow (version \u003e= 0.8)\n\n\n\n### How to Run the Experiments\n\nGenerate a dataset\n\n```\n$ python generator.py\n```\n\nThis script generates a file called `associative-retrieval.pkl`, which can be used for training.\n\n\n\nRun the model\n\n```\n$ python fw.py\n```\n\n\n\n### Findings\n\nThe following is the accuracy and loss graph for R=20. **The experiments are barely tuned.**\n\n![](fig/acc.png)\n\n\n\n![](fig/loss.png)\n\n**Layer Normalization is extremely crucial for the success of training.** \n\n- Otherwise, training will not converge when the inner step is larger than 1. \n- Even when inner step of 1, the performance without layer normalization is much worse. For R=20, only 0.4 accuracy can be achieved (which is same as the level of other models.)\n- Even with Layer Normalization, using slow weights (ie. vanilla RNN) is much worse than using fast weights.\n\n\n\nFurther improvements:\n\n- Complete fine-tuning\n- Work on other tasks\n\n\n\n\n### References\n\n[Using Fast Weights to Attend to the Recent Past](https://arxiv.org/abs/1610.06258). Jimmy Ba,  Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, Catalin Ionescu.\n\n[Layer Normalization](https://arxiv.org/abs/1607.06450). Jimmy Ba, Ryan Kiros, Geoffery Hinton.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiamings%2Ffast-weights","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjiamings%2Ffast-weights","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjiamings%2Ffast-weights/lists"}