{"id":18647246,"url":"https://github.com/ttitcombe/tf_nalu","last_synced_at":"2026-05-01T19:31:27.397Z","repository":{"id":115040849,"uuid":"149994351","full_name":"TTitcombe/tf_NALU","owner":"TTitcombe","description":"Tensorflow implementation of the paper \"Neural Arithmetic Logic Units\"","archived":false,"fork":false,"pushed_at":"2019-04-07T19:40:32.000Z","size":217,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-17T19:08:18.171Z","etag":null,"topics":["deep-learning","deepmind","machine-learning","nalu","neural-arithmetic-logic-unit","neural-network","tensorflow"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/TTitcombe.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-09-23T14:53:59.000Z","updated_at":"2019-03-20T18:47:21.000Z","dependencies_parsed_at":null,"dependency_job_id":"079cc5cc-1071-47df-b6e8-c05a78dec090","html_url":"https://github.com/TTitcombe/tf_NALU","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/TTitcombe/tf_NALU","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TTitcombe%2Ftf_NALU","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TTitcombe%2Ftf_NALU/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TTitcombe%2Ftf_NALU/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TTitcombe%2Ftf_NALU/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/TTitcombe","download_url":"https://codeload.github.com/TTitcombe/tf_NALU/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/TTitcombe%2Ftf_NALU/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32510593,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"online","status_checked_at":"2026-05-01T02:00:05.856Z","response_time":64,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","deepmind","machine-learning","nalu","neural-arithmetic-logic-unit","neural-network","tensorflow"],"created_at":"2024-11-07T06:25:21.457Z","updated_at":"2026-05-01T19:31:27.376Z","avatar_url":"https://github.com/TTitcombe.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# TF Neural Arithmetic Logic Unit\n\n![Extrapolation test](https://github.com/TTitcombe/tf_NALU/blob/master/figures/extrapolation_test.png)\n\nA TensorFlow implementation of the paper [\"Neural Arithmetic Logic Units\"](https://arxiv.org/pdf/1808.00508.pdf) by Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer, Phil Blunsom.\n\nExplanation of the paper, and a write-up of this code, can be found [here](https://medium.com/@t.j.titcombe/understanding-neural-arithmetic-logic-units-5ca9d0041473).\n\nThis code currently provides a NAC module, a NALU module, and a MLP.\n\nTo perform the static numerical tests, run **static_arithmetic_task.py**.\n\nThe proof-of-necessity experiment in the paper, trying to learn the identity mapping with a neural network, can be carried out in **test_nn_extrapolation.py**. The results of the quick test done with this code can be seen above.\n\n\n## Results\n\n### Sanity Check\nI completed a quick sanity check by training a 1-layer NALU on 10 2-dimensional data points until loss became unchanged (a smaller version of the static arithmetic task).\nLooking at the weight values (W) and g in results/Weights_Sanity_Test.txt, we can see that the NALU learns almost perfectly how to model basic arithmetic functions in this toy example. We're good to continue with the more complex tasks!\n\n### Static Arithmetic\nCurrently have only performed the experiments of section 4.1 of the paper, the static numerical tests (addition, subtraction, multiplication, division, square, square root).\n\nThe results can be seen in results/static_arithmetic_test.txt\nWe can see that, for all operations except for divide, NAC or NALU achieved the best results. The results for divide are not surprising, as having a/b will be close to 1 unless a \u003e\u003e b or b \u003c\u003c a, thus making it quite easy for a MLP to find some local minimum near its initialised state. To combat this one could draw a and b from a greater number of data dimensions, and make the difference between them more distinct i.e. a is the sum of 20 dimensions, b is the sum of 5.\n\nFor addition and subtraction, NAC outperformed NALU. While NALU can learn to act as a NAC, these results are to be expected due to the extra complexity of a NALU - its more difficult to learn to become a high-performing NAC than for the NAC to learn to be high-performing.\n\nNote that I did very minimal hyper-perameter tuning, and number of data points and epochs were kept to a minimum, because I am not made of compute. Compared to the paper, it's clear that I've found suboptimal solutions. However, these results are enough to convince me of the power of the NALU.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fttitcombe%2Ftf_nalu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fttitcombe%2Ftf_nalu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fttitcombe%2Ftf_nalu/lists"}