https://github.com/mamba413/cope
Off-Policy Interval Estimation withConfounded Markov Decision Process
https://github.com/mamba413/cope
causal-inference confidence-intervals off-policy-evaluation reinforcement-learning statistical-inference
Last synced: about 1 year ago
JSON representation
Off-Policy Interval Estimation withConfounded Markov Decision Process
- Host: GitHub
- URL: https://github.com/mamba413/cope
- Owner: Mamba413
- Created: 2021-06-27T12:48:20.000Z (almost 5 years ago)
- Default Branch: main
- Last Pushed: 2023-09-13T14:01:35.000Z (over 2 years ago)
- Last Synced: 2025-03-24T11:07:50.897Z (about 1 year ago)
- Topics: causal-inference, confidence-intervals, off-policy-evaluation, reinforcement-learning, statistical-inference
- Language: Python
- Homepage: https://www.tandfonline.com/doi/full/10.1080/01621459.2022.2110878
- Size: 3.14 MB
- Stars: 6
- Watchers: 1
- Forks: 4
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Off-Policy Interval Estimation with Confounded Markov Decision Process (COPE)
This repository contains the implementation for the paper "[Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process](https://arxiv.org/pdf/2202.10589.pdf)" (JASA, 2022) in Python.
## Summary of the Paper
This paper is concerned with constructing a confidence interval for a target policy's value offline based on a pre-collected observational data in infinite horizon settings. Most of the existing works assume no unmeasured variables exist that confound the observed actions. This assumption, however, is likely to be violated in real applications such as healthcare and technological industries. In this paper, we show that with some auxiliary variables that mediate the effect of actions on the system dynamics, the target policy’s value is identifiable in a confounded Markov decision process. Based on this result, we develop an efficient off-policy value estimator that is robust to potential model misspecification and provide rigorous uncertainty quantification. Our method is justified by theoretical results, simulated and real datasets obtained from ridesharing companies.
**Figure 1**: Causal diagrams for MDP, confounded MDP and confounded MDP with mediators (the focus of this paper).
## Requirements
Change your working directory to this main folder, run `conda env create --file COPE.yml` to create the Conda environment,
and then run `conda activate COPE` to activate the environment.
## Code Description
The proposed estimators:
- `opeuc.py`: direct estimator, importance sampling estimator, confounded off-policy estimator
Nuisance parameters:
- `problearner.py`: learn transition probabilities: (i) state --> action & (ii) (state, action) --> mediator
- `qlearner.py`: fitted Q evaluation
- `rnnl.py`: marginal ratio learning via neural network
- `rll.py`: marginal ratio learning via linear model
Sampling:
- `simulator_save.py`: generate observations tuple from MDP
Utilities:
- `policy.py`: target policies
- `utilize.py`: some helpful functions
- `utilize_ci.py`: helpful functions for computing confidence intervals
- `utilize_prototypemodel.py`: helpful classes for simulations
Numerical experiments:
(See ACC form detailed for instructions)
- `sim_robust.py`: simulation for demonstrating double robustness
- `sim_time_compare.py` & `sim_time_compare_multdim.py`: simulation when time points vary
- `sim_trajectory_compare.py` & `sim_trajectory_compare_multdim.py`: simulation when the number of trajectories vary
- `sim_ratiolearner_compare.py`
- `sim_ratio_features_number_compare.py`
## Citations
Please cite the following publications if you make use of the material here.
- Shi, C., Zhu, J., Ye, S., Luo, S., Zhu, H., & Song, R. (2022). Off-policy confidence interval estimation with confounded Markov decision process. Journal of the American Statistical Association, accepted.
```
@article{shi2022off,
title={Off-policy confidence interval estimation with confounded Markov decision process},
author={Shi, Chengchun and Zhu, Jin and Ye, Shen and Luo, Shikai and Zhu, Hongtu and Song, Rui},
journal={Journal of the American Statistical Association},
volume={accepted},
year={2022},
publisher={Taylor \& Francis}
}
```
## License
All content in this repository is licensed under the GPL-3 license.