{"id":16739414,"url":"https://github.com/zafarali/better-sampling","last_synced_at":"2025-03-15T22:43:53.931Z","repository":{"id":78568182,"uuid":"111598298","full_name":"zafarali/better-sampling","owner":"zafarali","description":"investigating sampling","archived":false,"fork":false,"pushed_at":"2019-02-08T19:43:40.000Z","size":3475,"stargazers_count":0,"open_issues_count":2,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-22T11:47:41.760Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zafarali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-11-21T20:33:05.000Z","updated_at":"2019-02-08T19:43:38.000Z","dependencies_parsed_at":"2023-06-26T02:04:35.872Z","dependency_job_id":null,"html_url":"https://github.com/zafarali/better-sampling","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fbetter-sampling","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fbetter-sampling/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fbetter-sampling/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Fbetter-sampling/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zafarali","download_url":"https://codeload.github.com/zafarali/better-sampling/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243801611,"owners_count":20350106,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-13T00:50:54.849Z","updated_at":"2025-03-15T22:43:53.908Z","avatar_url":"https://github.com/zafarali.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Reinforced Variational Inference for Stochastic Processes\n\n## Abstract Goals\nThe objective of this project is to learn the reverse transition dynamics of a stochastic process conditioned on a terminal observation. We will explore recent connections made between variational inference and policy-based reinforcement learning in \\citet{weber2015reinforced} and \\citet{bachman2015data} termed as reinforced variational inference. To our knowledge, this will be the first empirical evaluation of the above method and will provide insight into which techniques are transferable between the fields. We will evaluate the method on two tasks with known posterior distributions: the random walk and the two window random walk. We will then evaluate the method on two other stochastic processes from epidemiology and \\zafcomment{population genetics}.\n\n## How to Set up\n\nThis project requires installing PyTorch 0.3.1 (See requirements.txt for a full list of modules needed) and Python 3.5+\nAdditionally, this project requires installing `pg_methods`, a new tool to make implementing algorithms based policy gradients / REINFORCE simpler\n\nTo install that project:\n\n```bash\ngit clone https://github.com/zafarali/policy-gradient-methods.git\ncd policy-gradient-methods\npip install -e .\n```\n\nyou can now install this one:\n\n```bash\ngit clone https://github.com/zafarali/better-sampling.git\ncd better-sampling\npip install -r requirements.txt\npip install -e .\n```\n\nTo test if everything is working you can run:\n\n```bash\npython -m pytest ./tests\n```\n\n## Code Organization\n\nThere are two important parts of this code base: Stochastic Processes and Samplers.\n\n### Stochastic Processes\n\nA stochastic process is a sequence of random variables that are generated according to some transition kernel. The `StochasticProcess` object here is an abstraction of this.\nThey implement:\n\n1. `state_space`: the size of the random variable \n2. `action_space`: the number of possible transitions from one variable to the next\n4. `transition_prob`: Returns the transition probability from one `state_t` to `state_tp1` \n5. `simulate`: Simulates a new realization of the stochastic process\n\nWe can then interact with the stochastic process via \"agents\" or \"particles\" that start at the final observation. To interact with the stochastic process we have:\n\n1. `reset`: Resets the stochastic process to the observation\n2. `reset_task`: Generates a new realization (via `simulate`)\n3. `step`: Executes an action in the environment and moves the particle one step back in time.\n\nConcretely, we have the following workflow:\n\n```python\nfrom rvi_sampling.stochastic_processes import MyStochasticProcess\n\nenv = MyStochasticProcess(*stochastic_process_configuations, seed=2018)\nstate = env.reset_task() # generates a new task\nstate_tp1, log_prob, done, info =  env.step(action)\n...\nenv.reset() # Resets the environment (but not the task!)\n```\n\nThe `log_prob` is the log probability of taking the `action` under the stochastic process.\n\n#### Implemented Stochastic Processes\n\n1. Discrete Random Walk\n2. Two Window Discrete Random Walk\n3. SIR Epidemiology model (Work in Progress)\n\n### Samplers\n\nThe most important part of the code base are `Sampler` objects that can be found in `./rvi_sampling/samplers/`.\nThe purpose of the sampler is to estimate the posterior distribution. Each sampler you implement must take the hyperparameters you are interested in as the `__init__` arguments (along with seed) and must implement the `solve\n method so that they can be benchmarked against each other. The `Sampler` interacts with the `StochasticProcess` via the `solve` command.\n \n```python\nfrom rvi_sampling.samplers import MyAwesomeSampler\n\nsampler = MyAwesomeSampler(hyperparameter1=1, hyperparameter2=2)\nsampler.solve(env, mc_samples=5000) # runs the sampler and obtains 5k MC Samples\n```\n\n`Sampler.solve` returns a `SamplingResults` object for easy visualizations.\n\n### Testing the algorithm\n\nTo perform a single evaluation of the algorithm you can look at [`experiments/rw_experiment.py`](./experiments/rw_experiment.py)\nTo run a large scale evaluation of the algorithm on a SLURM cluster (like ComputeCanada) you can use [`./experiments/cc_random_walk_expectation_analysis.py`](./experiments/cc_random_walk_expectation_analysis.py) \nwhich will create a SLURM script to run the above experiment on 50 different tasks, averaged over 5 seeds.\nYou can visualize the results using:\n1. `visualize_proposal.py`: visualize the proposal distribution\n2. `KL_violin.py` Compare the KL values for each algorithm.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzafarali%2Fbetter-sampling","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzafarali%2Fbetter-sampling","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzafarali%2Fbetter-sampling/lists"}