Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/maxjcohen/transformer
Implementation of Transformer model (originally from Attention is All You Need) applied to Time Series.
https://github.com/maxjcohen/transformer
metamodel timeseries transformer
Last synced: 7 days ago
JSON representation
Implementation of Transformer model (originally from Attention is All You Need) applied to Time Series.
- Host: GitHub
- URL: https://github.com/maxjcohen/transformer
- Owner: maxjcohen
- License: gpl-3.0
- Created: 2019-11-22T10:19:35.000Z (almost 5 years ago)
- Default Branch: master
- Last Pushed: 2023-06-02T07:47:50.000Z (over 1 year ago)
- Last Synced: 2024-08-02T06:21:26.034Z (3 months ago)
- Topics: metamodel, timeseries, transformer
- Language: Jupyter Notebook
- Homepage: https://timeseriestransformer.readthedocs.io/en/latest/
- Size: 65.2 MB
- Stars: 831
- Watchers: 15
- Forks: 165
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- awesome-time-series - Implementation of Transformer model (originally from Attention is All You Need) applied to Time Series
README
# Transformers for Time Series
[](https://timeseriestransformer.readthedocs.io/en/latest/?badge=latest) [](https://www.gnu.org/licenses/gpl-3.0) [](https://github.com/maxjcohen/transformer/releases/latest)
Implementation of Transformer model (originally from [Attention is All You Need](https://arxiv.org/abs/1706.03762)) applied to Time Series (Powered by [PyTorch](https://pytorch.org/)).
## Transformer model
Transformer are attention based neural networks designed to solve NLP tasks. Their key features are:
- linear complexity in the dimension of the feature vector ;
- paralellisation of computing of a sequence, as opposed to sequential computing ;
- long term memory, as we can look at any input time sequence step directly.
This repo will focus on their application to times series.
## Dataset and application as metamodel
Our use-case is modeling a numerical simulator for building consumption prediction. To this end, we created a dataset by sampling random inputs (building characteristics and usage, weather, ...) and got simulated outputs. We then convert these variables in time series format, and feed it to the transformer.
## Adaptations for time series
In order to perform well on time series, a few adjustments had to be made:
- The embedding layer is replaced by a generic linear layer ;
- Original positional encoding are removed. A "regular" version, better matching the input sequence day/night patterns, can be used instead ;
- A window is applied on the attention map to limit backward attention, and focus on short term patterns.
## Installation
All required packages can be found in `requirements.txt`, and expect to be run with `python3.7`. Note that you may have to install pytorch manually if you are not using pip with a Debian distribution : head on to [PyTorch installation page](https://pytorch.org/get-started/locally/). Here are a few lines to get started with pip and virtualenv:
```bash
$ apt-get install python3.7
$ pip3 install --upgrade --user pip virtualenv
$ virtualenv -p python3.7 .env
$ . .env/bin/activate
(.env) $ pip install -r requirements.txt
```
## Usage
### Downloading the dataset
The dataset is not included in this repo, and must be downloaded manually. It is comprised of two files, `dataset.npz` contains all input and outputs value, `labels.json` is a detailed list of the variables. Please refer to [#2](https://github.com/maxjcohen/transformer/issues/2) for more information.
### Running training script
Using jupyter, run the default `training.ipynb` notebook. All adjustable parameters can be found in the second cell. Careful with the `BATCH_SIZE`, as we are using it to parallelize head and time chunk calculations.
### Outside usage
The `Transformer` class can be used out of the box, see the [docs](https://timeseriestransformer.readthedocs.io/en/latest/?badge=latest) for more info.
```python
from tst import Transformer
net = Transformer(d_input, d_model, d_output, q, v, h, N, TIME_CHUNK, pe)
```
### Building the docs
To build the doc:
```bash
(.env) $ cd docs && make html
```