{"id":13689345,"url":"https://github.com/loeweX/Greedy_InfoMax","last_synced_at":"2025-05-01T23:34:05.778Z","repository":{"id":53022125,"uuid":"188848433","full_name":"loeweX/Greedy_InfoMax","owner":"loeweX","description":"Code for the paper:   Putting An End to End-to-End: Gradient-Isolated Learning of Representations","archived":false,"fork":false,"pushed_at":"2023-03-28T14:52:19.000Z","size":14175,"stargazers_count":283,"open_issues_count":0,"forks_count":36,"subscribers_count":16,"default_branch":"master","last_synced_at":"2024-08-03T15:17:28.457Z","etag":null,"topics":["deep-learning","local-learning","pytorch"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1905.11786","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/loeweX.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2019-05-27T13:27:34.000Z","updated_at":"2024-06-26T03:03:59.000Z","dependencies_parsed_at":"2022-09-08T15:51:02.600Z","dependency_job_id":"bf505ca3-07af-441a-a0d1-444e2d29f53b","html_url":"https://github.com/loeweX/Greedy_InfoMax","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loeweX%2FGreedy_InfoMax","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loeweX%2FGreedy_InfoMax/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loeweX%2FGreedy_InfoMax/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/loeweX%2FGreedy_InfoMax/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/loeweX","download_url":"https://codeload.github.com/loeweX/Greedy_InfoMax/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224282239,"owners_count":17285793,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","local-learning","pytorch"],"created_at":"2024-08-02T15:01:44.302Z","updated_at":"2025-05-01T23:34:05.767Z","avatar_url":"https://github.com/loeweX.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"# Greedy InfoMax\n\nWe can train a neural network **without end-to-end backpropagation** and achieve competitive performance.\n\nThis repo provides the code for the experiments in our paper:\n\nSindy Löwe*, Peter O'Connor, Bastiaan S. Veeling* - [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://arxiv.org/abs/1905.11786)\n\n\u0026ast;equal contribution\n\n\n## What is Greedy InfoMax?\n\nWe simply divide existing architectures into gradient-isolated modules and optimize the mutual information between cross-patch intermediate representations.\n\n\n![The Greedy InfoMax Learning Approach](media/architecture.png)\n\n\nWhat we found exciting is that despite each module being trained greedily, it improves upon the representation of the previous module. This enables you to keep stacking modules until downstream performance saturates.\n\n\u003cp align=\"center\"\u003e \n    \u003cimg src=\"./media/LatentClassification.png\" width=\"700\"\u003e\n\u003c/p\u003e\n\n\n## How to run the code\n\n### Dependencies\n\n- [Python and Conda](https://www.anaconda.com/)\n- Setup the conda environment `infomax` by running:\n\n    ```bash\n    bash setup_dependencies.sh\n    ```\n\nAdditionally, for the audio experiments:\n- Install [torchaudio](https://github.com/pytorch/audio) in the `infomax` environment\n- Download audio datasets \n    ```bash \n    bash download_audio_data.sh\n    ```\n\n### Usage\n\n#### Vision Experiments\n- To replicate the vision results from our paper, run\n\n    ``` bash\n    source activate infomax\n    bash vision_traineval.sh\n    ```\n    This will train the Greedy InfoMax model as well as evaluate it by training a linear image classifiers on top of it\n    \n    \n\n- View all possible command-line options by running\n\n    ``` bash\n    python -m GreedyInfoMax.vision.main_vision --help\n    ```    \n    \n    Some of the more important options are:\n    \n    * in order to train the baseline CPC model with end-to-end backpropagation instead of the Greedy InfoMax model set: \n    ```bash\n    --model_splits 1\n    ```\n\n    * If you want to save GPU memory, you can train layers sequentially, one at a time, by setting the module to be trained (0-2), e.g.\n    \n    ```bash \n    --train_module 0\n    ```\n    \n\n#### Audio Experiments\n- To replicate the audio results from our paper, run\n\n    ``` bash\n    source activate infomax\n    bash audio_traineval.sh\n    ```\n    This will train the Greedy InfoMax model as well as evaluate it by training two linear classifiers on top of it - one for speaker and one for phone classification.\n    \n    \n\n- View all possible command-line options by running\n\n    ``` bash\n    python -m GreedyInfoMax.audio.main_audio --help\n    ```    \n    \n    Some of the more important options are:\n    \n    * in order to train the baseline CPC model with end-to-end backpropagation instead of the Greedy InfoMax model set: \n    ```bash\n    --model_splits 1\n    ```\n\n    * If you want to save GPU memory, you can train layers sequentially, one at a time, by setting the layer to be trained (0-5), e.g.\n    \n    ```bash \n    --train_layer 0\n    ```\n    \n## Want to learn more about Greedy InfoMax?\nCheck out my [blog post](https://loewex.github.io/GreedyInfoMax.html) for an intuitive explanation of Greedy InfoMax. \n\nAdditionally, you can watch my [presentation at NeurIPS 2019](https://slideslive.com/38923276). My slides for this talk are available [here](media/Presentation_GreedyInfoMax_NeurIPS.pdf).\n\n\n## Cite\n\nPlease cite our paper if you use this code in your own work:\n\n```\n@inproceedings{lowe2019putting,\n  title={Putting an End to End-to-End: Gradient-Isolated Learning of Representations},\n  author={L{\\\"o}we, Sindy and O'Connor, Peter and Veeling, Bastiaan},\n  booktitle={Advances in Neural Information Processing Systems},\n  pages={3039--3051},\n  year={2019}\n}\n```\n\n\n## References \n- [Representation Learning with Contrastive Predictive Coding - Oord et al.](https://arxiv.org/abs/1807.03748)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FloeweX%2FGreedy_InfoMax","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FloeweX%2FGreedy_InfoMax","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FloeweX%2FGreedy_InfoMax/lists"}