{"id":14977263,"url":"https://github.com/iacolippo/direct-feedback-alignment","last_synced_at":"2025-10-28T03:30:48.441Z","repository":{"id":94984795,"uuid":"81566555","full_name":"iacolippo/Direct-Feedback-Alignment","owner":"iacolippo","description":"Experiments with Direct Feedback Alignment training scheme for DNNs","archived":false,"fork":false,"pushed_at":"2017-02-23T14:34:16.000Z","size":332,"stargazers_count":31,"open_issues_count":1,"forks_count":7,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-01T10:41:37.555Z","etag":null,"topics":["dfa","ipynb","neural-network"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iacolippo.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-02-10T13:13:07.000Z","updated_at":"2024-10-27T05:22:02.000Z","dependencies_parsed_at":"2023-06-11T10:30:52.935Z","dependency_job_id":null,"html_url":"https://github.com/iacolippo/Direct-Feedback-Alignment","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iacolippo%2FDirect-Feedback-Alignment","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iacolippo%2FDirect-Feedback-Alignment/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iacolippo%2FDirect-Feedback-Alignment/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iacolippo%2FDirect-Feedback-Alignment/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iacolippo","download_url":"https://codeload.github.com/iacolippo/Direct-Feedback-Alignment/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":238590593,"owners_count":19497351,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dfa","ipynb","neural-network"],"created_at":"2024-09-24T13:55:22.655Z","updated_at":"2025-10-28T03:30:43.110Z","avatar_url":"https://github.com/iacolippo.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Direct-Feedback-Alignment\n\n## Understanding the general framework\n\nIn [dfa-linear-net.ipynb](https://github.com/iacolippo/Direct-Feedback-Alignment/blob/master/dfa-linear-net.ipynb), I show how a neural network without activation function can learn a linear function (multiplication by a matrix) using direct feedback alignment (DFA), as in [Nøkland, 2016](https://arxiv.org/pdf/1609.01596.pdf). There is also some theory about it.\n\nIn [dfa-mnist.ipynb](https://github.com/iacolippo/Direct-Feedback-Alignment/blob/master/dfa-mnist.ipynb), I show how a neural network trained with DFA achieves very similar results to one trained with backpropagation. The architecture is very simple: one hidden layer of 800 Tanh units, sigmoid in the last layer and binary crossentropy loss.\n\nGo to the last lines of [mlp-torch-results.txt](https://github.com/iacolippo/Direct-Feedback-Alignment/blob/master/mlp-torch-results.txt) if you want to see the results of the same architecture using Torch code provided by Nøkland.\n\n## Stacking neural networks\n\nDo networks with different feedback matrices learn different features at least in the first few steps? Apparently yes. **Stacking** works training a lot of weak learners on recognizing different features and using their outputs as inputs for a new model, which will learn how to combine these weak learners and give a performance boost.\n\nIn [Stacking-dfa-nets](https://github.com/iacolippo/Direct-Feedback-Alignment/tree/master/Stacking-dfa-nets) folder, you have the following files. They must be executed in the following order:\n\n1. *create_dataset.py*: preprocess MNIST data loaded from Keras and save them to a Numpy file *mnist.npz*, ready to be used.\n2. *weak-learners.py* or *diff-weak-learners.py*: train as many weak learners as you want (NNs with one hidden layer 800 Tanh units). The difference between the first and the second is that the first trains all of them starting from the same initialization, while the second initializes each one of them in a different state. They generate respectively files called: *train_linouts.npz* \u0026 *test_linouts.npz*, *diff-train_linouts.npz* \u0026 *diff-test_linouts.npz*\n3. *stacked-model.py* or *RD-stacked-model.py*: train respectively a dense or an RD layer on top of the features extracted by each weak learner. The program takes as input the names of the files generated by the previous steps and the number of weak learners given in the previous step.\n\nExample call to train 50 weak learners:\n```bash\npython weak_learners.py 50\n```\n\nExample call to train a stacked model on top of 50 weak learners:\n```bash\npython weak_learners.py 50 train_linouts.npz test_linouts.npz\n```\n\n## RD Layers\n\nLayers with a linear number of parameters vaguely inspired by [ACDC](https://arxiv.org/abs/1511.05946). Basically they do these operations:\n\n![Equation](http://latex.codecogs.com/gif.latex?a_1%20%3D%20D_1%20x%5Cmbox%7B%2C%20%7D%20a_2%20%3D%20R%20a_1%5Cmbox%7B%2C%20%7D%20a_3%20%3D%20D_2%20a_2)\n\nwhere D1 and D2 are diagonal matrices and R is a random matrix.\n\n# Requirements\n\n- numpy\n- matplotlib\n- scipy\n- keras\n- scikit-learn\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiacolippo%2Fdirect-feedback-alignment","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiacolippo%2Fdirect-feedback-alignment","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiacolippo%2Fdirect-feedback-alignment/lists"}