{"id":13571379,"url":"https://github.com/hamrel-cxu/EnbPI","last_synced_at":"2025-04-04T08:31:07.680Z","repository":{"id":125304764,"uuid":"377386757","full_name":"hamrel-cxu/EnbPI","owner":"hamrel-cxu","description":"Official code for: Conformal prediction interval for dynamic time-series (conference, ICML 21 Long Presentation) AND Conformal prediction for time-series (journal, IEEE TPAMI)","archived":false,"fork":false,"pushed_at":"2023-11-25T15:22:14.000Z","size":27433,"stargazers_count":76,"open_issues_count":0,"forks_count":20,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-02-13T07:42:38.490Z","etag":null,"topics":["anomaly-detection","conformal-prediction","machine-learning","prediction","prediction-interval","time-series"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hamrel-cxu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2021-06-16T05:57:14.000Z","updated_at":"2024-01-20T16:18:10.000Z","dependencies_parsed_at":"2023-11-25T16:36:42.157Z","dependency_job_id":null,"html_url":"https://github.com/hamrel-cxu/EnbPI","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamrel-cxu%2FEnbPI","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamrel-cxu%2FEnbPI/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamrel-cxu%2FEnbPI/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hamrel-cxu%2FEnbPI/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hamrel-cxu","download_url":"https://codeload.github.com/hamrel-cxu/EnbPI/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247147038,"owners_count":20891613,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anomaly-detection","conformal-prediction","machine-learning","prediction","prediction-interval","time-series"],"created_at":"2024-08-01T14:01:01.508Z","updated_at":"2025-04-04T08:31:02.671Z","avatar_url":"https://github.com/hamrel-cxu.png","language":"Jupyter Notebook","funding_links":[],"categories":["📝 Papers with code","Python"],"sub_categories":["Managed database services"],"readme":"# Ensemble batch prediction intervals (EnbPI)\n\n## Table of Contents\n* [News and Notes](#news-and-notes)\n* [Open-source implementation](#open-source-implementation)\n* [Extension Works and Ideas](#extension-works-and-ideas)\n* [How to use](#how-to-use)\n* [Poster Talk and Slides](#poster-talk-and-slides)\n* [FAQ](#faq)\n* [References](#references)\n\u003c!-- * [License](#license) --\u003e\n\n## News and Notes\n- The code and related material in this _main_ branch are for our ICML 2021 oral work, titled [Conformal Prediction Interval for Dynamic Time-series](http://proceedings.mlr.press/v139/xu21h.html) (Xu et al. 2021a). Please use codes in this repository, as those downloaded from PMLR are not the most updated ones. You may direct any inquiries either to [Chen Xu](https://sites.gatech.edu/chenxu97/) (cxu310@gatech.edu) or [Yao Xie](https://www2.isye.gatech.edu/~yxie77/index.html) (yao.xie@isye.gatech.edu). The work is constantly updated to incorporate new feedback and ideas. \n- We have significantly revised and extended the ICML 2021 work, which is accepted by 🌟[IEEE Transactions on Pattern Analysis and Machine Intelligence](https://ieeexplore.ieee.org/abstract/document/10121511)🌟. The most recent version is also available on [arxiv](https://arxiv.org/abs/2010.09107). The [_Journal_code_ branch](https://github.com/hamrel-cxu/EnbPI/tree/Journal_code) contains updated codes, which essentially follow the same structure as those in this branch. Nevertheless, feel free to message us if you have any question regarding either branch. \n\n## Open-source implementation\n- We are excited that the work has been integrated as a part of\n  1. [MAPIE](https://github.com/scikit-learn-contrib/MAPIE), which is a scikit-learn-compatible module for predictive inference.\n  2. [Fortuna](https://aws-fortuna.readthedocs.io/en/latest/examples/enbpi_ts_regression.html) by Amazon AWS.\n  3. [PUNCC](https://deel-ai.github.io/puncc/regression.html#deel.puncc.regression.EnbPI) by the Artificial and Natural Intelligence Toulouse Institute.\n  4. [functime.ai](https://docs.functime.ai/forecasting/#conformal-prediction) for production-ready time series models.\n  5. [ConformalPrediction](https://juliatrustworthyai.github.io/ConformalPrediction.jl/stable/how_to_guides/timeseries/) for Trustworthy AI in Julia.\n\n## Extension Works and Ideas\n- 🌟[Sequential Predictive Conformal Inference for Time Series](https://proceedings.mlr.press/v202/xu23r)🌟 (Xu et al. 2023) is our latest work on this topic, where we advance EnbPI by considering *time-adaptive re-estimation of residual quantiles*, improving the performance of EnbPI and other recent CP methods.\n- [Conformal prediction set for time-series](https://arxiv.org/abs/2206.07851) (Xu et al. 2022) focuses on time-series classification. It has been strongly accepted by the Distribution-free Uncertainty Quantification Workshop in ICML 2021 ([DFUQ 2022](https://sites.google.com/berkeley.edu/dfuq-22/)), with the [Poster here](https://github.com/hamrel-cxu/Ensemble-Regularized-Adaptive-Prediction-Set-ERAPS/blob/main/ICML%202022%20DFUQ%20Poster.jpeg).\n- [Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data](https://arxiv.org/abs/2105.11886) (Xu et al. 2021) adopts **EnbPI** for detecting anomalous traffic flows. Our method significantly outperforms competing methods (Table 1). It has been accepted by the Distribution-free Uncertainty Quantification Workshop in ICML 2021 ([DFUQ 2021](https://sites.google.com/berkeley.edu/dfuq21/)), with the [Poster here](https://github.com/hamrel-cxu/EnbPI/blob/main/DFUQ%202021%20Anomaly%20Detection%20Poster.png).\n\n\n## How to use\n- **Required Dependency:** \n  - Basic modules: `numpy, pandas, sklearn, scipy, matplotlib, seaborn`.\n  - Additional modules: `statsmodels` for implementing ARIMA, `keras` for building neural network and recurrent neural networks, and `pyod` for competing anomaly detection methods.\n- **General Info and Tests:** This work reproduces all experiments in [Conformal Prediction Interval for Dynamic Time-series](https://arxiv.org/abs/2010.09107) (Xu et al. 2021a). In particular, \n  - [tests_paper.ipynb](https://github.com/hamrel-cxu/EnbPI/blob/main/tests_paper.ipynb) provides an illustration of how to generate the main figures (Figure 1-4) in the paper. The code contents are nearly identical to those in [tests_paper.py](https://github.com/hamrel-cxu/EnbPI/blob/main/tests_paper.py). \n  - [tests_paper+supp.py](https://github.com/hamrel-cxu/EnbPI/blob/main/tests_paper%2Bsupp.py) reproduces all figures, including additional ones found in the Appendix. It is written in Jupyter notebook format, so that they are meant to be executed line by line.\n- **EnbPI implementation:** \n  - [PI_class_EnbPI.py](https://github.com/hamrel-cxu/EnbPI/blob/main/PI_class_EnbPI.py) implements the [class](https://github.com/hamrel-cxu/EnbPI/blob/85245eb51adb5276b17b320be6cf1f83629b712b/PI_class_EnbPI.py#L20) that contains **EnbPI** ([line](https://github.com/hamrel-cxu/EnbPI/blob/85245eb51adb5276b17b320be6cf1f83629b712b/PI_class_EnbPI.py#L119)), Jackknife+-after-bootstrap ([line](https://github.com/hamrel-cxu/EnbPI/blob/85245eb51adb5276b17b320be6cf1f83629b712b/PI_class_EnbPI.py#L192), [paper](https://proceedings.neurips.cc/paper/2020/hash/2b346a0aa375a07f5a90a344a61416c4-Abstract.html)), Split/Inductive Conformal ([line](https://github.com/hamrel-cxu/EnbPI/blob/85245eb51adb5276b17b320be6cf1f83629b712b/PI_class_EnbPI.py#L222), [paper](https://www.intechopen.com/books/tools_in_artificial_intelligence/inductive_conformal_prediction__theory_and_application_to_neural_networks)), and Weighted Inductive Conformal ([line](https://github.com/hamrel-cxu/EnbPI/blob/85245eb51adb5276b17b320be6cf1f83629b712b/PI_class_EnbPI.py#L266), [paper](https://www.stat.cmu.edu/~ryantibs/papers/weightedcp.pdf)). We used [ARIMA](https://github.com/hamrel-cxu/EnbPI/blob/85245eb51adb5276b17b320be6cf1f83629b712b/PI_class_EnbPI.py#L318) as another competing method. \n  - Because conditional coverage (Figure 3) and anomaly detection (Figure 4) require problem-specific modifications, the code changes are not contained in \"PI_class_EnbPI.py\" but in their respective sections within [tests_paper.py](https://github.com/hamrel-cxu/EnbPI/blob/main/tests_paper.py)/[tests_paper+supp.py](https://github.com/hamrel-cxu/EnbPI/blob/main/tests_paper%2Bsupp.py).\n- **Other Function Files:** \n  - [utils_EnbPI.py](https://github.com/hamrel-cxu/EnbPI/blob/main/utils_EnbPI.py) primarily contain plotting functions for all figures except Figure 4.\n  - [PI_class_ECAD.py](https://github.com/hamrel-cxu/EnbPI/blob/main/PI_class_ECAD.py) implements **ECAD** based on **EnbPI** (see [Xu et al. 2021a, Section 8.5, Algorithm 2]) and [utils_ECAD.py](https://github.com/hamrel-cxu/EnbPI/blob/main/utils_ECAD.py) contains helpers for anomaly detection.\n- **Additional Files** \n  - The [Data](https://github.com/hamrel-cxu/EnbPI/tree/main/Data) repository contains all dataset in our paperexcept the money laundry one for Figure 4 (due to size limit). \n  - The [Results](https://github.com/hamrel-cxu/EnbPI/tree/main/Results) repository is provided for your convenience to reproduce plots, since some experiments by neural network/recurrent neural networks can take some time to execute. It contains all .csv results files on all dataset.\n- **Broad Usage:** To wrap **EnbPI** around other regression models and/or use on other data, one should:\n  - **If other regression models:** Make sure the model has methods `.fit(X_train, Y_train)` to train a predictor and `.predict(X_predict)` to make predictions on new data. Most models in [sklearn](https://scikit-learn.org/stable/supervised_learning.html) or deep learning models built by keras/pytorch are capable of doing so.\n  - **If other data:** We have assumed that all our datasets are save as `pandas.DataFrame` and convertible to `numpy.array`. However, such assumptions are purely computational. Please feel free to adjust the data formate as long as it can be processed by regression models of choice.\n\n## Poster Talk and Slides\n\n- The poster for our work is [available via this link](https://github.com/hamrel-cxu/EnbPI/blob/main/ICML%20Poster.png).\n- We are fortunate to pre-record a long presentation and give an oral presentation at the Proceedings of the 38th International Conference on Machine Learning (ICML 2021). The long presentation is [available on Slideslive](https://recorder-v3.slideslive.com/?share=37762\u0026s=ee12530c-5218-4c9e-8bb3-5107ceb79f41) and the oral presentation will be given at the conference once the date is finalized.\n- The slide for the talk is [available here](https://github.com/hamrel-cxu/EnbPI/blob/main/ICML2021_Slide.pdf).\n\n## FAQ\n1. Encountering \"NotImplementedError: Cannot convert a symbolic Tensor (lstm_2/strided_slice:0) to a numpy array\" when using RNN as the regression model:\n- See [this github answer](https://stackoverflow.com/questions/66207609/notimplementederror-cannot-convert-a-symbolic-tensor-lstm-2-strided-slice0-t) to resolve the problem, primarily due to numpy \u0026 python version issues.\n\n## References\n- Xu, Chen and Yao Xie (2023). Sequential Predictive Conformal Inference for Time Series. *Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 2023*\n- Xu, Chen and Yao Xie (2023). Conformal prediction for time-series. *Journal version, IEEE Transactions on Pattern Analysis and Machine Intelligence.*\n- Xu, Chen and Yao Xie (2021). Conformal prediction interval for dynamic time-series. *Conference version, The Proceedings of the 38th International Conference on Machine Learning, PMLR 139, 2021.*\n- Xu, Chen and Yao Xie (2022). Conformal prediction set for time-series. *ICML 2022 Distribution-Free Uncertainty Quantification workshop.*\n- Xu, Chen and Yao Xie (2021). Conformal Anomaly Detection on Spatio-Temporal Observations with Missing Data. *ICML 2021 Distribution-Free Uncertainty Quantification workshop.*\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhamrel-cxu%2FEnbPI","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhamrel-cxu%2FEnbPI","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhamrel-cxu%2FEnbPI/lists"}