{"id":19071717,"url":"https://github.com/blackhc/mdp","last_synced_at":"2025-04-28T15:45:10.140Z","repository":{"id":57415510,"uuid":"103268878","full_name":"BlackHC/mdp","owner":"BlackHC","description":"Make it easy to specify simple MDPs that are compatible with the OpenAI Gym.","archived":false,"fork":false,"pushed_at":"2023-01-23T09:57:29.000Z","size":700,"stargazers_count":40,"open_issues_count":3,"forks_count":10,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-04-13T18:18:32.739Z","etag":null,"topics":["mdp","openai-gym","rl"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BlackHC.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-09-12T12:39:17.000Z","updated_at":"2024-12-30T22:27:48.000Z","dependencies_parsed_at":"2023-02-12T22:01:39.753Z","dependency_job_id":null,"html_url":"https://github.com/BlackHC/mdp","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fmdp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fmdp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fmdp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BlackHC%2Fmdp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BlackHC","download_url":"https://codeload.github.com/BlackHC/mdp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249436875,"owners_count":21271957,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mdp","openai-gym","rl"],"created_at":"2024-11-09T01:30:12.975Z","updated_at":"2025-04-18T05:32:35.222Z","avatar_url":"https://github.com/BlackHC.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MDP environments for the OpenAI Gym\n\nThis Python framework makes it very easy to specify simple MDPs.\n\n[![Build Status](https://travis-ci.org/BlackHC/mdp.svg?branch=master)](https://travis-ci.org/BlackHC/mdp)\n\n## Installation\n\nTo install using pip, use:\n\n```\npip install blackhc.mdp\n```\n\nTo run the tests, use:\n\n```\npython setup.py test\n```\n\n## Whitepaper\n\nA whitepaper is available at \u003chttps://arxiv.org/abs/1709.09069\u003e. Here is a BibTeX entry that you can use in publications (or download [CITE_ME.bib](CITE_ME.bib)):\n```\n@techreport{blackhc.mdp,\n    Author = {Andreas Kirsch},\n    Title = {MDP environments for the OpenAI Gym},\n    Year = {2017},\n    Url = {https://arxiv.org/abs/1709.09069}\n}\n```\n\n\n## Introduction\n\nIn reinforcement learning, agents learn to maximize accumulated rewards from an environment that they can interact with by observing and taking actions. Usually, these environments satisfy a Markov property and are treated as *Markov Decision Processes* (*MDPs*).\n\nThe OpenAI Gym is a standardized and open framework that provides many different environments to train agents against through a simple API.\n\nEven the simplest of these environments already has a level of complexity that is interesting for research but can make it hard to track down bugs. However, the gym provides four very simple environments that are useful for testing. The `gym.envs.debugging` package contains a one-round environment with deterministic rewards and one with non-deterministic rewards, and a two-round environment with deterministic rewards and another one with non-deterministic rewards.\nThe author has found these environments very useful for smoke-testing code changes.\n\nThis Python framework makes it very easy to specify simple MDPs like the ones described above in an extensible way. With it, one can validate that agents converge correctly as well as examine other properties.\n\n## Specification of MDPs\n\nMDPs are Markov processes that are augmented with a reward function and discount factor. An MDP can be fully specified by a tuple of:\n\n* a finite set of states,\n* a finite set of actions,\n* a matrix that specifies probabilities of transitions to a new state for a given a state and action,\n* a reward function that specifies the reward for a given action taken in a certain state, and\n* a discount rate.\n\nThe reward function can be either deterministic, or it can be a probability distribution.\n\nWithin this framework, MDPs can be specified in Python using a simple *domain-specific language* (*DSL*).\nFor example, the one-round deterministic environment defined in `gym.envs.debugging.one_round_deterministic_reward` could be specified as follows:\n\n```python\nfrom blackhc.mdp import dsl\n\nstart = dsl.state()\nend = dsl.terminal_state()\n\naction_0 = dsl.action()\naction_1 = dsl.action()\n\nstart \u0026 (action_0 | action_1) \u003e end\nstart \u0026 action_1 \u003e dsl.reward(1.)\n```\n\nThe DSL is based on the following grammar (using EBNF[@ebnf]): \n\n    TRANSITION ::= STATE '\u0026' ACTION '\u003e' OUTCOME\n    OUTCOME ::= (REWARD | STATE) ['*' WEIGHT]\n    \n    ALTERNATIVES ::= ALTERNATIVE ('|' ALTERNATIVE)* \n    \nSee below for how alternatives work. Alternatives can be used in place of states, actions and outcomes, and comprise of states, actions and outcomes.\n\nFor a given state and action, outcomes can be specified. Outcomes are state transitions or rewards.\nIf multiple state transitions or rewards are specified for the same state and action, the MDP is non-deterministic and the state transition (or reward) are determined using a categorical distribution. By default, each outcome is weighted uniformly, except if specified otherwise by either having duplicate transitions or by using an explicit weight factor. \n\nFor example, to specify that a state receives a reward of +1 or -1 with equal probability and does not change states with probability 3/4 and only transitions to the next state with probability 1/4, we could write:\n   \n```python\nstate \u0026 action \u003e dsl.reward(-1.) | dsl.reward(1.)\nstate \u0026 action \u003e state * 3 | next_state\n```\n\nAlternatives are distributive with respect to both conjunctions (`\u0026`) and outcome mappings (`\u003e`), so:\n\n    (a | b) \u0026 (c | d) \u003e (e | f) ===\n    (a \u0026 c \u003e e) | (a \u0026 c \u003e f) | (a \u0026 d \u003e e) | \n    (a \u0026 d \u003e f) | (b \u0026 c \u003e e) | ... \n\nAlternatives can consist of states, actions, outcomes, conjunctions or partial transitions. For example, the following are valid alternatives:\n\n    stateA \u0026 actionA | stateB \u0026 actionB\n    (actionA \u003e stateC) | (actionB \u003e stateD)\n\nAs the DSL is implemented within Python, operator overloading is used to implement the semantics. Operator precedence is favorable as `*` has higher precedence than `\u0026`, which has higher precedence than `|`, which has higher precedence than `\u003e`. This allows for a natural formulation of transitions.\n\n## Conventional API\n\nThe framework also supports specifying an MDP using a conventional API as DSLs are not always preferred.\n\n```python\nfrom blackhc import mdp\n\nspec = mdp.MDPSpec()\nstart = spec.state('start')\nend = spec.state('end', terminal_state=True)\naction_0 = spec.action()\naction_1 = spec.action()\n\nspec.transition(start, action_0, mdp.NextState(end))\nspec.transition(start, action_1, mdp.NextState(end))\nspec.transition(start, action_1, mdp.Reward(1))\n```\n\n## Visualization\n\nTo make debugging easier, MDPs can be converted to `networkx` graphs and rendered using `pydotplus` and `GraphViz`.\n\n```python\nfrom blackhc import mdp\nfrom blackhc.mdp import example\n\nspec = example.ONE_ROUND_DMDP\n\nspec_graph = spec.to_graph()\nspec_png = mdp.graph_to_png(spec_graph)\n\nmdp.display_mdp(spec)\n```\n\n\u003cdiv\u003e\n\u003cimg src=\"docs/one_round_dmdp.png\" alt=\"One round deterministic MDP\" width=\"96\" /\u003e\n\u003c/div\u003e\n\u003cb\u003eFigure 1: One round deterministic MDP\u003c/b\u003e\n\n## Optimal values\n\nThe framework also contains a small module that can compute the optimal value functions using linear programming.\n\n```python\nfrom blackhc.mdp import lp\nfrom blackhc.mdp import example\n\nsolver = lp.LinearProgramming(example.ONE_ROUND_DMDP)\nprint(solver.compute_q_table())\nprint(solver.compute_v_vector())\n```\n\n## Gym environment\n\nAn environment that is compatible with the OpenAI Gym can be created easily by using the `to_env()` method. It supports rendering into Jupyter notebooks, as RGB array for storing videos, and as png byte data.\n\n```python\nfrom blackhc import mdp\nfrom blackhc.mdp import example\n\nenv = example.MULTI_ROUND_NMDP.to_env()\n\nenv.reset()\nenv.render()\n\nis_done = False\nwhile not is_done:\n    state, reward, is_done, _ = env.step(env.action_space.sample())\n    env.render()\n```\n\n\u003cdiv\u003e\n\u003cimg src=\"docs/multi_round_nmdp_render.png\" alt=\"env.render() of `example.MULTI_ROUND_NMDP`\" width=\"192\" /\u003e\n\u003c/div\u003e\n\u003cb\u003eFigure 2: env.render() of `example.MULTI_ROUND_NMDP`\u003c/b\u003e\n\n# Examples\n\nThe `blackhc.mdp.example` package provides 5 MDPs. Four of them match the ones in `gym.envs.debugging`, and the fifth one is depicted in figure 2. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackhc%2Fmdp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fblackhc%2Fmdp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fblackhc%2Fmdp/lists"}