{"id":11563764,"url":"https://github.com/thuml/Nonstationary_Transformers","last_synced_at":"2025-10-03T14:31:04.912Z","repository":{"id":62776113,"uuid":"543619814","full_name":"thuml/Nonstationary_Transformers","owner":"thuml","description":"Code release for \"Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting\" (NeurIPS 2022), https://arxiv.org/abs/2205.14415","archived":false,"fork":false,"pushed_at":"2024-08-19T03:35:19.000Z","size":4346,"stargazers_count":467,"open_issues_count":2,"forks_count":73,"subscribers_count":9,"default_branch":"main","last_synced_at":"2024-09-29T14:31:50.694Z","etag":null,"topics":["deep-learning","forecasting","non-stationary","time-series"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thuml.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-09-30T13:41:48.000Z","updated_at":"2024-09-29T07:30:03.000Z","dependencies_parsed_at":"2023-09-25T03:52:40.086Z","dependency_job_id":"2704d267-c0d9-4c57-bad3-faf9eb2d8ae9","html_url":"https://github.com/thuml/Nonstationary_Transformers","commit_stats":{"total_commits":29,"total_committers":1,"mean_commits":29.0,"dds":0.0,"last_synced_commit":"d99c426719f66a317967b06cc78c154dd023074c"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FNonstationary_Transformers","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FNonstationary_Transformers/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FNonstationary_Transformers/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thuml%2FNonstationary_Transformers/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thuml","download_url":"https://codeload.github.com/thuml/Nonstationary_Transformers/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235146404,"owners_count":18943256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","forecasting","non-stationary","time-series"],"created_at":"2024-06-23T05:59:42.536Z","updated_at":"2025-10-03T14:31:04.304Z","avatar_url":"https://github.com/thuml.png","language":"Python","funding_links":[],"categories":["时间序列"],"sub_categories":["网络服务_其他"],"readme":"# Non-stationary Transformers\n\nThis is the codebase for the paper:\n[Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://arxiv.org/abs/2205.14415), NeurIPS 2022. [[Slides]](https://cloud.tsinghua.edu.cn/f/8d6ce7b18d3c468190e7/), [[Poster]](https://cloud.tsinghua.edu.cn/f/6eea66909aa7465ca9a4/).\n\n:triangular_flag_on_post: **News** (2023.02) Non-stationary Transformer has been included in [[Time-Series-Library]](https://github.com/thuml/Time-Series-Library), which covers long- and short-term forecasting, imputation, anomaly detection, and classification.\n\n## Discussions\n\nThere are already several discussions about our paper, we appreciate a lot for their valuable comments and efforts: [[Official]](https://mp.weixin.qq.com/s/LkpkTiNBVBYA-FqzAdy4dw), [[OpenReview]](https://openreview.net/forum?id=ucNDIDRNjjv), [[Zhihu]](https://zhuanlan.zhihu.com/p/535931701).\n\n## Architecture\n\n![arch](./figures/arch.png)\n\n### Series Stationarization\n\nSeries Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability. \n\n![arch](./figures/ss.png)\n\n### De-stationary Attention\n\nDe-stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from unstationarized series. \n\n![arch](./figures/da.png)\n\n\n## Showcases\n\n![arch](./figures/showcases.png)\n\n## Preparation\n\n1. Install Python 3.7 and neccessary dependencies.\n```\npip install -r requirements.txt\n```\n2. All the six benchmark datasets can be obtained from [Google Drive](https://drive.google.com/file/d/1CC4ZrUD4EKncndzgy5PSTzOPSqcuyqqj/view?usp=sharing) or [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b8f4a78a39874ac9893e/?dl=1).\n\n## Training scripts\n\n### Non-stationary Transformer\n\nWe provide the Non-stationary Transformer experiment scripts and hyperparameters of all benchmark dataset under the folder `./scripts`.\n\n```bash\n# Transformer with our framework\nbash ./scripts/ECL_script/ns_Transformer.sh\nbash ./scripts/Traffic_script/ns_Transformer.sh\nbash ./scripts/Weather_script/ns_Transformer.sh\nbash ./scripts/ILI_script/ns_Transformer.sh\nbash ./scripts/Exchange_script/ns_Transformer.sh\nbash ./scripts/ETT_script/ns_Transformer.sh\n```\n\n```bash\n# Transformer baseline\nbash ./scripts/ECL_script/Transformer.sh\nbash ./scripts/Traffic_script/Transformer.sh\nbash ./scripts/Weather_script/Transformer.sh\nbash ./scripts/ILI_script/Transformer.sh\nbash ./scripts/Exchange_script/Transformer.sh\nbash ./scripts/ETT_script/Transformer.sh\n```\n\n### Non-stationary framework to promote other Attention-based models \n\nWe also provide the scripts for other Attention-based models (Informer, Autoformer), for example:\n\n```bash\n# Informer promoted by our Non-stationary framework\nbash ./scripts/Exchange_script/Informer.sh\nbash ./scripts/Exchange_script/ns_Informer.sh\n\n# Autoformer promoted by our Non-stationary framework\nbash ./scripts/Weather_script/Autoformer.sh\nbash ./scripts/Weather_script/ns_Autoformer.sh\n```\n\n## Experiment Results\n\n### Main Results\n\nFor multivariate forecasting results, the vanilla Transformer equipped with our framework consistently achieves state-of-the-art performance in all six benchmarks and prediction lengths.\n\n![arch](./figures/main_results.png)\n\n### Model Promotion\n\nBy applying our framework to six mainstream Attention-based models. Our method consistently improves the forecasting ability. Overall, it achieves averaged **49.43%** promotion on Transformer, **47.34%** on Informer, **46.89%** on Reformer, **10.57%** on Autoformer, **5.17%** on ETSformer and **4.51%** on FEDformer, making each of them surpass previous state-of-the-art.\n\n![arch](./figures/promotion.png)\n\n## Future Work\n\nWe will keep equip the following models with our proposed Non-stationary Transformers framework:\n\n- [x] Transformer\n- [x] iTransformer\n- [x] Informer\n- [x] Autoformer\n- [x] FEDformer\n- [x] Crossformer\n- [x] Reformer\n- [x] ......\n\nNote: Series Stationarization as an architecture-free module has been widely applied for addressing non-stationarity in time series. Please refer to [time-series-library](https://github.com/thuml/Time-Series-Library/tree/main/models) for the implementationdetails.\n\n## Citation\n\nIf you find this repo useful, please cite our paper. \n\n```\n@article{liu2022non,\n  title={Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting},\n  author={Liu, Yong and Wu, Haixu and Wang, Jianmin and Long, Mingsheng},\n  booktitle={Advances in Neural Information Processing Systems},\n  year={2022}\n}\n```\n\n## Contact\n\nIf you have any questions or want to use the code, please contact liuyong21@mails.tsinghua.edu.cn.\n\n\n## Acknowledgement\n\nThis repo is built on the [Autoformer repo](https://github.com/thuml/Autoformer), we appreciate the authors a lot for their valuable code and efforts.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthuml%2FNonstationary_Transformers","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthuml%2FNonstationary_Transformers","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthuml%2FNonstationary_Transformers/lists"}