{"id":13958502,"url":"https://github.com/gzerveas/mvts_transformer","last_synced_at":"2025-07-21T00:31:01.660Z","repository":{"id":37726304,"uuid":"374536582","full_name":"gzerveas/mvts_transformer","owner":"gzerveas","description":"Multivariate Time Series Transformer, public version","archived":false,"fork":false,"pushed_at":"2023-08-27T12:31:15.000Z","size":29,"stargazers_count":773,"open_issues_count":32,"forks_count":175,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-11-28T02:34:46.423Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/gzerveas.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2021-06-07T04:43:33.000Z","updated_at":"2024-11-26T14:24:09.000Z","dependencies_parsed_at":"2023-02-10T15:00:23.103Z","dependency_job_id":"d8025a61-6d1d-4604-a8a1-63eea8feb32c","html_url":"https://github.com/gzerveas/mvts_transformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/gzerveas/mvts_transformer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gzerveas%2Fmvts_transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gzerveas%2Fmvts_transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gzerveas%2Fmvts_transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gzerveas%2Fmvts_transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/gzerveas","download_url":"https://codeload.github.com/gzerveas/mvts_transformer/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gzerveas%2Fmvts_transformer/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":266221249,"owners_count":23894964,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-08T13:01:40.454Z","updated_at":"2025-07-21T00:31:01.361Z","avatar_url":"https://github.com/gzerveas.png","language":"Python","funding_links":[],"categories":["时间序列","Time-Series"],"sub_categories":["网络服务_其他","2024"],"readme":"# Multivariate Time Series Transformer Framework\n\nThis code corresponds to the [paper](https://dl.acm.org/doi/10.1145/3447548.3467401): George Zerveas et al. **A Transformer-based Framework for Multivariate Time Series Representation Learning**, in _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21), August 14-18, 2021_.\nArXiV version: https://arxiv.org/abs/2010.02803\n\nIf you find this code or any of the ideas in the paper useful, please consider citing:\n```buildoutcfg\n@inproceedings{10.1145/3447548.3467401,\nauthor = {Zerveas, George and Jayaraman, Srideepika and Patel, Dhaval and Bhamidipaty, Anuradha and Eickhoff, Carsten},\ntitle = {A Transformer-Based Framework for Multivariate Time Series Representation Learning},\nyear = {2021},\nisbn = {9781450383325},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nurl = {https://doi.org/10.1145/3447548.3467401},\ndoi = {10.1145/3447548.3467401},\nbooktitle = {Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery \u0026amp; Data Mining},\npages = {2114–2124},\nnumpages = {11},\nkeywords = {regression, framework, multivariate time series, classification, transformer, deep learning, self-supervised learning, unsupervised learning, imputation},\nlocation = {Virtual Event, Singapore},\nseries = {KDD '21}\n}\n```\n\n## Setup\n\n_Instructions refer to Unix-based systems (e.g. Linux, MacOS)._\n\n`cd mvts_transformer/`\n\nInside an already *existing* root directory, each experiment will create a time-stamped output directory, which contains\nmodel checkpoints, performance metrics per epoch, predictions per sample, the experiment configuration, log files etc.\nThe following commands assume that you have created a new root directory inside the project directory like this: \n`mkdir experiments`.\n\nThis code has been tested with `Python 3.7` and `3.8`.\n\n[We recommend creating and activating a `conda` or other Python virtual environment (e.g. `virtualenv`) to \ninstall packages and avoid conficting package requirements; otherwise, to run `pip`, the flag `--user` or `sudo` privileges will be necessary.]\n\n`pip install -r requirements.txt`\n\n[Note: Because sometimes newer versions of packages (e.g. `sktime`) break backward compatibility with previous versions or other packages, \nif you are encountering issues, you can instead use `failsafe_requirements.txt`, which contains specific versions \nof packages tested to work with this codebase.] \n\n### Get data from TS Archive\n\nDownload dataset files and place them in separate directories, one for regression and one for classification.\n\nClassification: http://www.timeseriesclassification.com/Downloads/Archives/Multivariate2018_ts.zip\n\nRegression: https://zenodo.org/record/3902651#.YB5P0OpOm3s\n\n### Adding your own datasets\n\nTo train and evaluate on your own data, you have to add a new data class in `datasets/data.py`.\nYou can see other examples for data classes in that file, or the template in `example_data_class.py`.\n\nThe data class sets up one or more `pandas` `DataFrame`(s) containing all data, indexed by example IDs.\nDepending on the task, these dataframes are accessed by the Pytorch `Dataset` subclasses in `dataset.py`.\n\nFor example, autoregressive tasks (e.g. imputation, transduction) require a member dataframe `self.feature_df`, \nwhile regression and classification (implemented through `ClassiregressionDataset`) additionally require a `self.labels_df` member\nvariable to be defined inside the data class in `data.py`.\n\nOnce you write your data class, you must add a string identifier for it in the `data_factory` dictionary inside `data.py`:\n```python\ndata_factory = {'weld': WeldData,\n                'tsra': TSRegressionArchive,\n                'pmu': PMUData,\n                'mydataset': MyNewDataClass}\n```\n\nYou can now train and evaluate using your own dataset through the option `--data_class mydataset`.\n\n## Example commands\n\nTo see all command options with explanations, run: `python src/main.py --help`\n\nYou should replace `$1` below with the name of the desired dataset.\nThe commands shown here specify configurations intended for `BeijingPM25Quality` for regression and `SpokenArabicDigits` for classification.\n\n_[To obtain best performance for other datasets, *use the hyperparameters as given in the Supplementary Material of the paper*. For example, for self-supervised pretraining of `BeijingPM25Quality`, the correct batch size is 128.\nAppropriate downsampling with the option `--subsample_factor` can be often used on datasets with longer time series to speedup training, without significant\nperformance degradation.]_\n\nThe configurations as shown below will evaluate the model on the TEST set periodically during training, and at the end of training.\n\nBesides the console output  and the logfile `output.log`, you can monitor the evolution of performance (after installing tensorboard: `pip install tensorboard`) with:\n```bash\ntensorboard dev upload --name my_exp --logdir path/to/output_dir\n```\n\n## Train models from scratch\n\n\n### Regression\n\n(Note: the loss reported for regression is the Mean Square Error, i.e. without the Root)\n\n```bash\npython src/main.py --output_dir path/to/experiments --comment \"regression from Scratch\" --name $1_fromScratch_Regression --records_file Regression_records.xls --data_dir path/to/Datasets/Regression/$1/ --data_class tsra --pattern TRAIN --val_pattern TEST --epochs 100 --lr 0.001 --optimizer RAdam  --pos_encoding learnable --task regression\n```\n\n### Classification\n\n```bash\npython src/main.py --output_dir experiments --comment \"classification from Scratch\" --name $1_fromScratch --records_file Classification_records.xls --data_dir path/to/Datasets/Classification/$1/ --data_class tsra --pattern TRAIN --val_pattern TEST --epochs 400 --lr 0.001 --optimizer RAdam  --pos_encoding learnable  --task classification  --key_metric accuracy\n```\n\n## Pre-train models (unsupervised learning through input masking)\n\nCan be used for any downstream task, e.g. regression, classification, imputation.\n\nMake sure that the network architecture parameters of the pretrained model match the parameters of the desired fine-tuned model (e.g. use `--d_model 64` for `SpokenArabicDigits`).\n\n```bash\npython src/main.py --output_dir experiments --comment \"pretraining through imputation\" --name $1_pretrained --records_file Imputation_records.xls --data_dir /path/to/$1/ --data_class tsra --pattern TRAIN --val_ratio 0.2 --epochs 700 --lr 0.001 --optimizer RAdam --batch_size 32 --pos_encoding learnable --d_model 128\n```\n\nAs noted above, please check the paper for the optimal hyperparameter values for each dataset. E.g. for pretraining on `BeijingPM25Quality`, one should use `--batch_size 128`.\n\n## Fine-tune pretrained models\n\nMake sure that network architecture parameters (e.g. `d_model`) used to fine-tune a model match the pretrained model.\n\n### Regression\n```bash\npython src/main.py --output_dir experiments --comment \"finetune for regression\" --name BeijingPM25Quality_finetuned --records_file Regression_records.xls --data_dir /path/to/Datasets/Regression/BeijingPM25Quality/ --data_class tsra --pattern TRAIN --val_pattern TEST  --epochs 200 --lr 0.001 --optimizer RAdam --pos_encoding learnable --d_model 128 --load_model path/to/BeijingPM25Quality_pretrained/checkpoints/model_best.pth --task regression --change_output --batch_size 128\n```\n\n### Classification\n```bash\npython src/main.py --output_dir experiments --comment \"finetune for classification\" --name SpokenArabicDigits_finetuned --records_file Classification_records.xls --data_dir /path/to/Datasets/Classification/SpokenArabicDigits/ --data_class tsra --pattern TRAIN --val_pattern TEST --epochs 100 --lr 0.001 --optimizer RAdam --batch_size 128 --pos_encoding learnable --d_model 64 --load_model path/to/SpokenArabicDigits_pretrained/checkpoints/model_best.pth --task classification --change_output --key_metric accuracy\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgzerveas%2Fmvts_transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgzerveas%2Fmvts_transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgzerveas%2Fmvts_transformer/lists"}