{"id":15640281,"url":"https://github.com/reiinakano/invariant-risk-minimization","last_synced_at":"2025-04-30T07:45:11.139Z","repository":{"id":113779273,"uuid":"204282353","full_name":"reiinakano/invariant-risk-minimization","owner":"reiinakano","description":"Implementation of Invariant Risk Minimization https://arxiv.org/abs/1907.02893","archived":false,"fork":false,"pushed_at":"2020-02-13T18:42:43.000Z","size":383,"stargazers_count":85,"open_issues_count":1,"forks_count":9,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-02-25T05:51:06.933Z","etag":null,"topics":["causality","deep-learning","invariant-risk-minimization","machine-learning","neural-network"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/reiinakano.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-25T11:01:24.000Z","updated_at":"2024-12-17T21:14:56.000Z","dependencies_parsed_at":"2023-03-13T13:18:15.819Z","dependency_job_id":null,"html_url":"https://github.com/reiinakano/invariant-risk-minimization","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reiinakano%2Finvariant-risk-minimization","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reiinakano%2Finvariant-risk-minimization/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reiinakano%2Finvariant-risk-minimization/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/reiinakano%2Finvariant-risk-minimization/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/reiinakano","download_url":"https://codeload.github.com/reiinakano/invariant-risk-minimization/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":242619122,"owners_count":20159001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["causality","deep-learning","invariant-risk-minimization","machine-learning","neural-network"],"created_at":"2024-10-03T11:33:32.531Z","updated_at":"2025-03-08T22:30:55.490Z","avatar_url":"https://github.com/reiinakano.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Implementation of Invariant Risk Minimization (https://arxiv.org/abs/1907.02893)\n\nThis is an attempt to reproduce the \"Colored MNIST\" experiments from the\npaper [Invariant Risk Minimization](https://arxiv.org/abs/1907.02893)\nby Arjovsky, et. al.\n\nAfter trying lots of hyperparameters and various tricks, this implementation \nachieves close to the paper-reported values (train accuracy \u003e 70%, \ntest accuracy \u003e 60%), though training can be quite unstable depending on\nthe random seed.\n\nThe most common failure case is when the gradient norm penalty term is weighted\ntoo highly relative to the ERM term. In this case, Φ converges to a function that \nreturns the same value for all inputs. The classifier cannot recover from this point\nand the accuracy is stuck at 50% for all environments. This makes sense mathematically.\nIf the intermediate representation is the same regardless of input, then *any*\nclassifier is the ideal classifier, resulting in the penalty gradient being 0.\n\nAnother failure case is when the gradient norm penalty is too low and the\noptimization essentially acts as in ERM (train accuracy \u003e 80%, test accuracy ~10%).\n\nThe most important trick I used to get this to work is through scheduled \nincrease of the gradient norm penalty weight.\nWe start at 0 for the gradient norm penalty weight, essentially beginning as ERM,\nthen slowly increase it per epoch.\n\nI use early stopping to stop training once the accuracy on all environments, \nincluding the test set, reach an acceptable value. Yes, stopping training based on \nperformance on the test set is not good practice, but I could not\nfind a principled way of stopping training by only observing performance on the\ntraining environments. One thing that might be needed when applying IRM to\nreal-world datasets is to leave out a separate environment as a validation set,\nwhich we can use for early stopping. The downside is we'll need a minimum of 4\nenvironments to perform IRM (2 train, 1 validation, 1 test).\n\nFeel free to leave an issue if you find a bug or a set of hyperparameters \nthat makes this training stable. ~~Otherwise, let's all just wait for the authors'\ncode, which they say will be available soon.~~ The authors' original code is here: https://github.com/facebookresearch/InvariantRiskMinimization, \nand apparently posted two months before I started this. \nFor some reason, I wasn't able to find this when I searched the first time.\nLooks like instead of a gradual increase of the gradient norm\npenalty, what they do is start at 0 for a few iterations then jump straight up\nto the higher value for the rest of training.\nI think the important thing is to make sure the training effectively starts as ERM (0 penalty) \nbefore adding in the IRM penalty term.\n\n## How to run\n\nYou can run [the provided notebook](https://colab.research.google.com/github/reiinakano/invariant-risk-minimization/blob/master/invariant_risk_minimization_colored_mnist.ipynb) in Colaboratory.\n\nAlternatively, you can run `main.py` locally. There is also an \nimplementation of ERM in `main.py` if you want to run a baseline.\nCode depends on Pytorch.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freiinakano%2Finvariant-risk-minimization","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Freiinakano%2Finvariant-risk-minimization","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Freiinakano%2Finvariant-risk-minimization/lists"}