{"id":19682772,"url":"https://github.com/ml-jku/mc-lstm","last_synced_at":"2025-04-29T05:30:39.703Z","repository":{"id":109348111,"uuid":"329609341","full_name":"ml-jku/mc-lstm","owner":"ml-jku","description":"Experiments with Mass Conserving LSTMs","archived":false,"fork":false,"pushed_at":"2021-07-20T14:47:30.000Z","size":137,"stargazers_count":38,"open_issues_count":0,"forks_count":14,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-05T13:23:36.333Z","etag":null,"topics":["deep-learning","lstm","machine-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ml-jku.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-01-14T12:30:18.000Z","updated_at":"2025-03-21T16:09:57.000Z","dependencies_parsed_at":"2023-04-26T15:15:59.384Z","dependency_job_id":null,"html_url":"https://github.com/ml-jku/mc-lstm","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-jku%2Fmc-lstm","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-jku%2Fmc-lstm/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-jku%2Fmc-lstm/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ml-jku%2Fmc-lstm/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ml-jku","download_url":"https://codeload.github.com/ml-jku/mc-lstm/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":251444121,"owners_count":21590423,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","lstm","machine-learning"],"created_at":"2024-11-11T18:12:09.725Z","updated_at":"2025-04-29T05:30:38.033Z","avatar_url":"https://github.com/ml-jku.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MC-LSTM: Mass-Conserving LSTM\n\nPieter-Jan Hoedt, Frederik Kratzert, \nDaniel Klotz, Christina Halmich, \nMarkus Holzleitner, Grey Nearing, \nSepp Hochreiter, Günter Klambauer\n\nMC-LSTM is an adaptation of LSTM [(Hochreiter \u0026 Schmidhuber, 1997)](#lstm) that allows to enforce conservation laws in regression problems. To test the benefits of this inductive bias in practice, we conducted experiments on a) arithmetic tasks (cf. [Madsen et al., 2020](#nau)), b) Traffic forecasting, c) energy prediction for pendulum (cf. [Greydanus et al., 2019](#hamiltonian)), and d) Rainfall-runoff modeling ([Kratzert et al., 2019](#hydrolstm)).\n\n### Sub-repositories\n\nThis repository contains code for the LSTM addition, traffic forecasting and pendulum experiments.\nThe neural arithmetic experiments were conducted on a [fork](https://github.com/hoedt/stable-nalu) of the repository from [Madsen et al. (2020)](#nau).\nThe experiments in hydrology were conducted using the [neuralhydrology](https://github.com/neuralhydrology/neuralhydrology) framework.\n\n## Paper\n\n[Openreview](https://openreview.net/forum?id=Rld-9OxQ6HU),\n[Pre-print](https://arxiv.org/abs/2101.05186),\n[ICML](http://proceedings.mlr.press/v139/hoedt21a.html)\n\n### Abstract\n\nThe success of Convolutional Neural Networks (CNNs) in computer vision is mainly driven by their strong inductive bias, which is strong enough to allow CNNs to solve vision-related tasks with random weights, meaning without learning. Similarly, Long Short-Term Memory (LSTM) has a strong inductive bias towards storing information over time. However, many real-world systems are governed by conservation laws, which lead to the redistribution of particular quantities — e.g. in physical and economical systems. Our novel Mass-Conserving LSTM (MC-LSTM) adheres to these conservation laws by extending the inductive bias of LSTM to model the redistribution of those stored quantities. MC-LSTMs set a new state-of-the-art for neural arithmetic units at learning arithmetic operations, such as addition tasks, which have a strong conservation law, as the sum is constant overtime. Further, MC-LSTM is applied to traffic forecasting, modeling a pendulum, and a large benchmark dataset in hydrology, where it sets a new state-of-the-art for predicting peak flows. In the hydrology example, we show that MC-LSTM states correlate with real world processes and are therefore interpretable.\n\n### Citation\n\nTo cite this work, you can use the following bibtex entry:\n ```bib\n@inproceedings{pmlr-v139-hoedt21a,\n  title     = {MC-LSTM: Mass-Conserving LSTM},\n  author    = {Hoedt, Pieter-Jan and Kratzert, Frederik and Klotz, Daniel and Halmich, Christina and Holzleitner, Markus and Nearing, Grey S and Hochreiter, Sepp and Klambauer, Guenter},\n  booktitle = {Proceedings of the 38th International Conference on Machine Learning},\n  pages     = {4275--4286},\n  year      = {2021},\n  editor    = {Meila, Marina and Zhang, Tong},\n  volume    = {139},\n  series    = {Proceedings of Machine Learning Research},\n  month     = {18--24 Jul},\n  publisher = {PMLR},\n  pdf       = {http://proceedings.mlr.press/v139/hoedt21a/hoedt21a.pdf},\n  url       = {http://proceedings.mlr.press/v139/hoedt21a.html},\n}\n\n```\n\n## Environment\n\nThe code in this repository (excluding the sub-repositories) should run as-is in an environment as specified by `requirements.txt`.\nWhen using `conda`, such an environment can be set up using\n```\nconda create -n mclstm --file requirements.txt -c pytorch\n```\n**if** you remove the `autograd` dependency!\n`autograd` must be installed with `pip` individually. \nAlternatively, you can use `-c conda-forge`, which does provide `autograd`.\n\n## References\n\n - \u003cspan id=\"hamiltonian\"\u003eGreydanus, S., Dzamba, M., \u0026 Yosinski, J. (2019).\u003c/span\u003e [Hamiltonian neural networks](https://proceedings.neurips.cc/paper/2019/hash/26cd8ecadce0d4efd6cc8a8725cbd1f8-Abstract.html). Advances in Neural Information Processing Systems, 32, 15379-15389.\n - \u003cspan id=\"lstm\"\u003eHochreiter, S., \u0026 Schmidhuber, J. (1997).\u003c/span\u003e [Long short-term memory](https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735). Neural computation, 9(8), 1735-1780. ([pdf](https://www.bioinf.jku.at/publications/older/2604.pdf))\n - \u003cspan id=\"nau\"\u003eMadsen, A., \u0026 Johansen, A. R. (2020).\u003c/span\u003e [Neural Arithmetic Units](https://openreview.net/forum?id=H1gNOeHKPS). In International Conference on Learning Representations.\n - \u003cspan id=\"hydrologylstm\"\u003eKratzert, F., Klotz, D., Shalev, G., Klambauer, G., Hochreiter, S., \u0026 Nearing, G. (2019).\u003c/span\u003e [Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets](https://hess.copernicus.org/articles/23/5089/2019/hess-23-5089-2019.html). Hydrology and Earth System Sciences, 23(12), 5089-5110.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fml-jku%2Fmc-lstm","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fml-jku%2Fmc-lstm","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fml-jku%2Fmc-lstm/lists"}