https://github.com/borealisai/scaleformer
https://github.com/borealisai/scaleformer
Last synced: about 1 year ago
JSON representation
- Host: GitHub
- URL: https://github.com/borealisai/scaleformer
- Owner: BorealisAI
- License: other
- Created: 2022-06-03T20:41:38.000Z (about 4 years ago)
- Default Branch: main
- Last Pushed: 2023-02-07T20:44:51.000Z (over 3 years ago)
- Last Synced: 2025-03-22T14:11:51.154Z (about 1 year ago)
- Language: Python
- Size: 3.32 MB
- Stars: 130
- Watchers: 7
- Forks: 13
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Scaleformer
**[Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting](https://arxiv.org/abs/2206.04038), ICLR 2023**
This repo is a modified version of the public implementation of [Autoformer paper](https://arxiv.org/abs/2106.13008) which can be find in this [repository](https://github.com/thuml/Autoformer). We also use the related parts of [FEDformer](https://github.com/MAZiqing/FEDformer), [FiLM](https://github.com/tianzhou2011/FiLM/), and [NHits](https://github.com/Nixtla/neuralforecast).
## Why Scaleformer?
Using iteratively refining a forecasted time series at multiple scales with shared weights, architecture adaptations and a specially-designed normalization scheme, we are able to achieve significant performance improvements with minimal additional computational overhead.
Figure 1. Overview of the proposed framework. (Left) Representation of a single scaling block. In each step, we pass the normalized upsampled version of the output from previous step along with the normalized downsampled version of encoder as the input. (Right) Representation of the full architecture. We process the input in a multi-scale manner iteratively from the smallest scale to the original scale.
Our experiments on various public datasets demonstrate that the proposed method outperforms the corresponding baselines. Depending on the choice of transformer architecture, our mutli-scale framework results in mean squared error reductions ranging from 5.5% to 38.5%.
Table 1.
Comparison of the MSE and MAE results for our proposed multi-scale framework version of different methods (-MSA) with respective baselines. Results are given in the multi-variate setting, for different lenghts of the horizon window. The best results are shown in Bold. Our method outperforms vanilla version of the baselines over almost all datasets and settings. The average improvement (error reduction) is shown in Green numbers at the bottom with respect the base models.
## Installation
**1. Clone our repo and install the requirements:**
```
git clone https://github.com/BorealisAI/scaleformer.git
cd scaleformer
pip install -r requirements.txt
```
**2. Download datasets and create the dataset directory**
You can download the datasets from [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/d/e1ccfff39ad541908bae/) or [Google Drive](https://drive.google.com/drive/folders/1ZOYpTUa82_jCcxIdTmyr0LXQfvaM9vIy?usp=sharing) links provided by [Autoformer](https://github.com/thuml/Autoformer) repository. For more information, please visit the repository.
Put all of the downloaded datasets in a `dataset` folder in the current directory:
```
scaleformer
├── dataset
│ ├── exchange_rate
| | └── exchange_rate.csv
│ ├── traffic
| | └── traffic.csv
| └── ...
└── data_provider
└── exp
└── ...
```
## Running the code
**1. running a single experiment**
You can run a single experiment using the following command:
```
python -u run.py --data_path {DATASET} --model {MODEL} --pred_len {L} --loss {LOSS_FUNC}
```
for example, for using **Informer-MSA** as the model for **traffic** dataset with an output length of **192** and **adaptive** loss function, you can run:
```
python -u run.py --data_path traffic.csv --model InformerMS --pred_len 192 --loss adaptive
```
To see more examples and parameters, please see `run_all.sh`.
**2. Running all of the experiments**
To run all of the experiments using slurm, you can use `run_all.sh` which uses `run_single.sh` to submit jobs with different parameters. The final errors of experiments will be available in `results.txt` and you can check `slurm` directory for the log of each experiment.
## Contact
If you have any question regarding the ScaleFormer, please contact aminshabaany@gmail.com.
## Citation
```
@article{shabani2022scaleformer,
title={Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting},
author={Shabani, Amin and Abdi, Amir and Meng, Lili and Sylvain, Tristan},
journal={arXiv preprint arXiv:2206.04038},
year={2022}
}
```
## Acknowledgement
We acknowledge the following github repositories that made the base of our work:
https://github.com/thuml/Autoformer
https://github.com/MAZiqing/FEDformer
https://github.com/jonbarron/robust_loss_pytorch.git
https://github.com/Nixtla/neuralforecast
https://github.com/tianzhou2011/FiLM/