{"id":16739376,"url":"https://github.com/zafarali/emdp","last_synced_at":"2025-03-21T22:31:32.074Z","repository":{"id":33465653,"uuid":"129259553","full_name":"zafarali/emdp","owner":"zafarali","description":"Easy MDPs and grid worlds with accessible transition dynamics to do exact calculations","archived":false,"fork":false,"pushed_at":"2022-04-01T07:18:30.000Z","size":84,"stargazers_count":49,"open_issues_count":3,"forks_count":14,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-18T05:34:57.225Z","etag":null,"topics":["gridworld-environment","markov-decision-processes","reinforcement-learning"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zafarali.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-04-12T13:56:08.000Z","updated_at":"2025-03-12T18:40:24.000Z","dependencies_parsed_at":"2022-08-07T21:16:57.512Z","dependency_job_id":null,"html_url":"https://github.com/zafarali/emdp","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Femdp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Femdp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Femdp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zafarali%2Femdp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zafarali","download_url":"https://codeload.github.com/zafarali/emdp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244880185,"owners_count":20525505,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["gridworld-environment","markov-decision-processes","reinforcement-learning"],"created_at":"2024-10-13T00:50:38.836Z","updated_at":"2025-03-21T22:31:31.736Z","avatar_url":"https://github.com/zafarali.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# emdp\n\nEasy MDPs implemented in a gym like interface with access to transition dynamics.\n\nJump to topics: [Installation](#installation) | [Grid World](#grid-world) | [Grid World-\u003ePlotting](#plotting-gridworlds) | [Grid World-\u003eCustomize](#customization) | [Accessing transition dynamics](#accessing-transition-dynamics)\n\n## Installation\n\n`cd` into this directory and then run:\n\n```bash\npip install -e .\n```\n\n## Usage\n\n`emdp` can simulate arbitriary MDPs with and without absorbing states.\n\n### Chain World\n\nThese are found in `emdp.chainworld`. A helper function is given to you to build worlds easily:\n\n```python\nfrom emdp.chainworld import build_chain_MDP\nfrom emdp import actions\nbuild_chain_MDP(n_states=7, p_success=0.9, reward_spec=[(5, actions.RIGHT, +1), (1, actions.LEFT, -1)]\n                    starting_distribution=np.array([0,0,0,1,0,0,0]),\n                    terminal_states=[0, 6], gamma=0.9)\n```\nThis creates a 7 state MDP where the agent starts in the middle at the two ends are two terminal states.\nOnce the agent enters a terminal state it goes into the absorbing state. The agent executes the wrong action with prob 0.1\nIf the agent is at the left of the world and it takes an action LEFT it gets a -1 and goes into the abosrbing state.\notherwise it gets nothing. If the agent is at the right of the world and it takes an action RIGHT it gets a +1 otherwise it gets nothing.\n\n### Grid World\n\nHere we provide helper functions to create gridworlds and a simple function to build an empty gridworld.\n```python\nfrom emdp.gridworld import build_simple_grid\nP = build_simple_grid(size=5, terminal_states=[(0, 4)], p_success=0.9)\n```\nBuilds a simple 5x5 grid world where there is a terminal state at (0, 4). The probability of successfully executing the action is 0.9. This function returns the transition matrix.\n\nFor a full example, see how to build this example from the S\u0026B book:\n\n```python\nimport emdp.gridworld as gw\n\ndef build_SB_example35():\n    \"\"\"\n    Example 3.5 from (Sutton and Barto, 2018) pg 60 (March 2018 version).\n    A rectangular Gridworld representation of size 5 x 5.\n\n    Quotation from book:\n    At each state, four actions are possible: north, south, east, and west, which deterministically\n    cause the agent to move one cell in the respective direction on the grid. Actions that\n    would take the agent off the grid leave its location unchanged, but also result in a reward\n    of −1. Other actions result in a reward of 0, except those that move the agent out of the\n    special states A and B. From state A, all four actions yield a reward of +10 and take the\n    agent to A'. From state B, all actions yield a reward of +5 and take the agent to B'\n    \"\"\"\n    size = 5\n    P = gw.build_simple_grid(size=size, p_success=1)\n    # modify P to match dynamics from book.\n\n    P[1, :, :] = 0 # first set the probability of all actions from state 1 to zero\n    P[1, :, 21] = 1 # now set the probability of going from 1 to 21 with prob 1 for all actions\n\n    P[3, :, :] = 0  # first set the probability of all actions from state 3 to zero\n    P[3, :, 13] = 1  # now set the probability of going from 3 to 13 with prob 1 for all actions\n\n    R = np.zeros((P.shape[0], P.shape[1])) # initialize a matrix of size |S|x|A|\n    R[1, :] = +10\n    R[3, :] = +1\n\n    p0 = np.ones(P.shape[0])/P.shape[0] # uniform starting probability (assumed)\n    gamma = 0.9\n\n    terminal_states = []\n    return gw.GridWorldMDP(P, R, gamma, p0, terminal_states, size)\n```\n\nTo actually use this there is a gym like interface where you can move around:\n\n```python\n\nmdp = build_SB_example35()\nstate, reward, done, _ = mdp.step(actions.UP) # moves the agent up.\n```\n\n#### Plotting GridWorlds\n\nThere are some tools built in for quickly plotting trajectories obtained from the `GridWorldMDP`s.\n\n```python\nfrom emdp.gridworld import GridWorldPlotter\nfrom emdp import actions\nimport random\ngwp = GridWorldPlotter(mdp.size, mdp.has_absorbing_state) # alternatively you can use GridWorldPlotter.from_mdp(mdp)\n\n# collect some trajectories from the GridWorldMDP object:\n\ntrajectories = []\nfor _ in range(3): # 3 trajectories\n  trajectory = [mdp.reset()]\n  for _ in range(10): # 10 steps maximum\n    state, reward, done, info = mdp.step(random.sample([actions.LEFT, actions.RIGHT, \n                                                        actions.UP, actions.DOWN], 1)[0])\n    trajectory.append(state)\n  trajectories.append(trajectory)\n```\n\nNow `trajectories` contains a list of lists of numpy arrays which represent the states. You can easily obtain trajectory plots and state visitation heatmaps:\n\n```python\nfig = plt.figure(figsize=(10, 4))\nax = fig.add_subplot(121)\n\n# trajectory\ngwp.plot_trajectories(ax, trajectories)\ngwp.plot_grid(ax)\n\n# heatmap\nax = fig.add_subplot(122)\ngwp.plot_heatmap(ax, trajectories)\ngwp.plot_grid(ax)\n```\n\nYou will get something like this:\n\n![image](https://user-images.githubusercontent.com/6295292/39479043-20587d32-4d32-11e8-82ae-7deddca8dc07.png)\n\n#### Customization\n\nThere is an interface to add walls and blockages to the gridworld.\n\n```python\nfrom emdp.gridworld.builder_tools import TransitionMatrixBuilder\nbuilder = TransitionMatrixBuilder(grid_size=5, has_terminal_state=False)\nbuilder.add_grid([], p_success=1)\nbuilder.add_wall_at((4, 2))\nbuilder.add_wall_at((3, 2))\nbuilder.add_wall_at((2, 2))\nbuilder.add_wall_at((1, 2))\n```\n\nUsing the above interface we have built a wall within our gridworld. Visualizing the trajectories in a similar way as above we get:\n\n![image](https://user-images.githubusercontent.com/6295292/39713910-667750c2-51f6-11e8-9e80-b24b4861afda.png)\n\n\nAlternatively, you can use `add_wall_between` which creates a straight line of walls between two positions on the grid.\nSo the following code will produce\n```python\nfrom emdp.gridworld.builder_tools import TransitionMatrixBuilder\nbuilder = TransitionMatrixBuilder(grid_size=5, has_terminal_state=False)\nbuilder.add_grid([], p_success=1)\nbuilder.add_wall_between((0,2), (1, 2))\nbuilder.add_wall_between((3,2), (4, 2))\nbuilder.add_wall_between((1,1), (1, 3))\n```\n\n![image](https://user-images.githubusercontent.com/6295292/39715133-376e147e-51fa-11e8-98c4-d14528c330a6.png)\n\n\n## Accessing transition dynamics\n\nYou can access transition dynamics by inspecting the `MDP` object:\n\n```python\nmdp.P # transition matrix\nmdp.R # reward matrix\nmdp.p0 # starting distribution\nmdp.gamma # discount factor\nmdp.terminal_states # the location of the terminal states\n```\n\n### Absorbing states\n\nIf you have an absorbing state in your MDP, it must be the last one. All actions executed in the absorbing state must lead to itself.\n\n\n## Current usage\n\nI use this code consistently in many pieces of work. If you use it, let me know and I'll be happy to add them here. Here are a few preprints that are available:\n\n\n- [Bachman, Philip, et al. \"Vfunc: a deep generative model for functions.\" arXiv preprint arXiv:1807.04106 (2018).](https://arxiv.org/abs/1807.04106)\n- [Ahmed, Zafarali, et al. \"Understanding the impact of entropy in policy learning.\" arXiv preprint arXiv:1811.11214 (2018).](https://arxiv.org/abs/1811.11214)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzafarali%2Femdp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzafarali%2Femdp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzafarali%2Femdp/lists"}