{"id":18616026,"url":"https://github.com/bjoern-hempel/js-reinforcement-learning-framework","last_synced_at":"2026-06-19T18:31:51.554Z","repository":{"id":110076526,"uuid":"145627384","full_name":"bjoern-hempel/js-reinforcement-learning-framework","owner":"bjoern-hempel","description":"A reinforcement learning framework","archived":false,"fork":false,"pushed_at":"2018-10-08T22:08:20.000Z","size":3577,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-09-05T01:38:30.835Z","etag":null,"topics":["javascript","mathematics","mit-license","reinforcement-learning","reinforcement-learning-algorithms","statistics"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bjoern-hempel.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-08-21T22:47:48.000Z","updated_at":"2018-10-08T22:08:21.000Z","dependencies_parsed_at":"2023-04-01T15:50:28.135Z","dependency_job_id":null,"html_url":"https://github.com/bjoern-hempel/js-reinforcement-learning-framework","commit_stats":null,"previous_names":[],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/bjoern-hempel/js-reinforcement-learning-framework","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bjoern-hempel%2Fjs-reinforcement-learning-framework","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bjoern-hempel%2Fjs-reinforcement-learning-framework/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bjoern-hempel%2Fjs-reinforcement-learning-framework/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bjoern-hempel%2Fjs-reinforcement-learning-framework/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bjoern-hempel","download_url":"https://codeload.github.com/bjoern-hempel/js-reinforcement-learning-framework/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bjoern-hempel%2Fjs-reinforcement-learning-framework/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34544403,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-19T02:00:06.005Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["javascript","mathematics","mit-license","reinforcement-learning","reinforcement-learning-algorithms","statistics"],"created_at":"2024-11-07T03:33:49.500Z","updated_at":"2026-06-19T18:31:51.540Z","avatar_url":"https://github.com/bjoern-hempel.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# A javascript reinforcement learning framework\n\nThis is a javascript reinforcement learning framework.\n\n## 1. Introduction\n\nIn progress.. \n\n## 2. Markov decision process (MDP)\n\n### 2.1 Theory\n\n#### 2.1.1 The bellman equation\n\n\u003cimg src=\"https://latex.ixno.de/?r=300\u0026p=1pt\u0026c=1\u0026f=V%5E%2A%28s%29%20%3D%20%5Csubstack%7B%5Ctextbf%7Bmax%7D%5C%5C%20%7B%5Ctiny%20a%7D%7D%5Csum_%7Bs%27%7D%5E%7B%7D%20T%28s%2C%20a%2C%20s%27%29%5BR%28s%2C%20a%2C%20s%27%29%20%2B%20%5Cgamma%20%5Ccdot%20V%5E%2A%28s%27%29%5D%20%5Cquad%20%5Cforall%20s\" width=\"499\" alt=\"V^*(s) = \\substack{\\textbf{max}\\\\ {\\tiny a}}\\sum_{s'}^{} T(s, a, s')[R(s, a, s') + \\gamma \\cdot V^*(s')] \\quad \\forall s\"\u003e\n\n#### 2.1.2 The value iteration algorithm\n\n\u003cimg src=\"https://latex.ixno.de/?r=300\u0026p=1pt\u0026c=1\u0026f=V_%7Bk%2B1%7D%28s%29%20%5Cleftarrow%20%5Csubstack%7B%5Ctextbf%7Bmax%7D%5C%5C%20%7B%5Ctiny%20a%7D%7D%5Csum_%7Bs%27%7D%5E%7B%7D%20T%28s%2C%20a%2C%20s%27%29%5BR%28s%2C%20a%2C%20s%27%29%20%2B%20%5Cgamma%20%5Ccdot%20V_k%28s%27%29%5D%20%5Cquad%20%5Cforall%20s\" width=\"516\" alt=\"V_{k+1}(s) \\leftarrow \\substack{\\textbf{max}\\\\ {\\tiny a}}\\sum_{s'}^{} T(s, a, s')[R(s, a, s') + \\gamma \\cdot V_k(s')] \\quad \\forall s\"\u003e\n\n#### 2.1.3 The Q-value iteration algorithm\n\n\u003cimg src=\"https://latex.ixno.de/?r=300\u0026p=1pt\u0026c=1\u0026f=Q_%7Bk%2B1%7D%28s%2Ca%29%20%5Cleftarrow%20%5Csum_%7Bs%27%7D%5E%7B%7D%20T%28s%2C%20a%2C%20s%27%29%5BR%28s%2C%20a%2C%20s%27%29%20%2B%20%5Cgamma%20%5Ccdot%20%5Csubstack%7B%5Ctextbf%7Bmax%7D%5C%5C%20%7B%5Ctiny%20a%27%7D%7D%20Q_k%28s%27%2Ca%27%29%5D%20%5Cquad%20%5Cforall%20%28s%2Ca%29\" width=\"604\" alt=\"Q_{k+1}(s,a) \\leftarrow \\sum_{s'}^{} T(s, a, s')[R(s, a, s') + \\gamma \\cdot \\substack{\\textbf{max}\\\\ {\\tiny a'}} Q_k(s',a')] \\quad \\forall (s,a)\"\u003e\n\n### 2.2 Usage\n\n#### 2.2.1 Super basic example\n\nLet's look at a state s\u003csub\u003e0\u003c/sub\u003e which contains 3 actions which point back to the state s\u003csub\u003e0\u003c/sub\u003e. The first action a\u003csub\u003e0\u003c/sub\u003e receives a reward of 1. The second action a\u003csub\u003e1\u003c/sub\u003e receives a penalty of 1 (-1). The third action a\u003csub\u003e2\u003c/sub\u003e remains neutral and results in neither reward nor punishment. In this case, each action contains only one possible state change, so the transition probability is 1,0 (100%). The whole setup does not make much sense, but it should show the procedure in more detail:\n\n\u003cimg src=\"/images/SuperBasic.png\" width=\"480\" alt=\"super basic example\"\u003e\n\nAs one can logically see, a\u003csub\u003e0\u003c/sub\u003e is the best option and leads to maximum reward. a\u003csub\u003e1\u003c/sub\u003e teaches us punishment and is the most unfavorable variant, while a\u003csub\u003e2\u003c/sub\u003e is the neutral version without any reward. Let's calculate that:\n\n##### 2.2.1.1 Code\n\n**The written-out version:**\n\n```javascript\nvar discountFactor = 0;\n\nvar rl = new ReinforcementLearning.mdp();\n\n/* s0 */\nvar s0 = rl.addState();\n\n/* create a0, a1 and a2 */\nvar a0 = rl.addAction(s0);\nvar a1 = rl.addAction(s0);\nvar a2 = rl.addAction(s0);\n\n/* add the action to state connections (state changes) */\nrl.addStateChange(a0, s0, 1.0,  1);\nrl.addStateChange(a1, s0, 1.0, -1);\nrl.addStateChange(a2, s0, 1.0,  0);\n\nvar Q = rl.calculateQ(discountFactor);\n\nconsole.log(JSON.stringify(Q));\n```\n\n**The short version:**\n\n```javascript\nvar discountFactor = 0;\n\nvar rl = new ReinforcementLearning();\n\n/* s0 */\nvar s0 = rl.addState();\n\n/* s0.a0, s0.a1 and s0.a2 */\nrl.addAction(s0, new StateChange(s0, 1.0,  1));\nrl.addAction(s0, new StateChange(s0, 1.0, -1));\nrl.addAction(s0, new StateChange(s0, 1.0,  0));\n\nvar Q = rl.calculateQ(discountFactor);\n\nconsole.log(JSON.stringify(Q));\n```\n\n**It returns:**\n\n```json\n[\n    [1, -1, 0]\n]\n```\n\nAs we suspected above: a\u003csub\u003e0\u003c/sub\u003e is the winner and with the maximum value of Q\u003csub\u003e(s=0)\u003c/sub\u003e (Q\u003csub\u003e(s=0,a=0)\u003c/sub\u003e = 1). The discountFactor is set to 0, because we only want to consider one iteration step. The discountFactor determines the importance of future rewards: A factor of 0 makes the agent \"short-sighted\" by considering only the current rewards, while a factor of close to 1 makes him strive for a high long-sighted reward. Because it is set to 0, only the next step is important and it shows the previously shown result.\n\nThe situation doesn't change if we look a little bit more far-sighted and we set the discount factor close to 1 (e.g. 0,9):\n\n```javascript\nvar discountFactor = 0.9;\n```\n\n**It returns:**\n\n```json\n[\n    [9.991404955442832, 7.991404955442832, 8.991404955442832]\n]\n```\n\nQ\u003csub\u003e(s=0,a=0)\u003c/sub\u003e is still the winner with the maximum of Q\u003csub\u003e(s=0)\u003c/sub\u003e ≈ 10. The algorithm of the `calculateQ` function stops the iteration of the above Markov formula until the Q change difference falls below a certain threshold: the default value of this threshold is 0,001.\n\n##### 2.2.1.2 Watch at the [demo](demo/rl-super-basic.html)\n\nDiscount rate 0,9:\n\n\u003cimg src=\"/images/SuperBasicDemo.png\" width=\"793\" alt=\"super basic example\"\u003e\n\n#### 2.2.2 Basic example\n\nLet's look at the next following example:\n\n\u003cimg src=\"/images/Basic.png\" width=\"512\" alt=\"super basic example\"\u003e\n\nIf we look at this example in the short term, it is a good idea to permanently go through a\u003csub\u003e1\u003c/sub\u003e from s\u003csub\u003e0\u003c/sub\u003e (discountRate = 0) and stay on state s\u003csub\u003e0\u003c/sub\u003e: Because we always get a reward of 2 and we don't want to receive any punishment of -5. From a far-sighted point of view, it's better to go through a\u003csub\u003e0\u003c/sub\u003e (discountRate = 0,9), because in future we will receive a reward of 10 in addition to the punishment of -5 (it means the sum of 5 reward instead of only 2). Let's calculate that:  \n\n##### 2.2.2.1 Code\n\n```javascript\nvar discountRate = 0.9;\n\nvar rl = new ReinforcementLearning.mdp();\n\n/* s0 and s1 */\nvar s0 = rl.addState();\nvar s1 = rl.addState();\n\n/* s0.a0, s0.a1 and s1.a0 */\nrl.addAction(s0, new StateChange(s1, 1.0, -5));\nrl.addAction(s0, new StateChange(s0, 1.0,  2));\n\n/* s1.a0 */\nrl.addAction(s1, new StateChange(s0, 1.0, 10));\n\nvar Q = rl.calculateQ(discountRate);\n\nconsole.log(JSON.stringify(Q));\n```\n\n**It returns:**\n\n```json\n[\n    [21.044799074176453, 20.93957918978874],\n    [28.93957918978874]\n]\n```\n\nAs we expected, far-sighted it is better to choose s\u003csub\u003e0\u003c/sub\u003e.a\u003csub\u003e0\u003c/sub\u003e with the reward of Q\u003csub\u003e(s=0,a=0)\u003c/sub\u003e.\n\n##### 2.2.2.2 Watch at the [demo](demo/rl-basic.html)\n\nDiscount rate 0,9:\n\n\u003cimg src=\"/images/BasicDemo.png\" width=\"828\" alt=\"basic example\"\u003e\n\n##### 2.2.2.3 Comparison of different discount rates\n\n| discountRate | type               | s\u003csub\u003e0\u003c/sub\u003e    | s\u003csub\u003e1\u003c/sub\u003e | s\u003csub\u003e0\u003c/sub\u003e (winner) | s\u003csub\u003e1\u003c/sub\u003e (winner) |\n|--------------|--------------------|------------------|---------------|------------------------|------------------------|\n| 0.0          | short-sighted      | `[-5, 2]`        | `[10]`        | a\u003csub\u003e1\u003c/sub\u003e          | a\u003csub\u003e0\u003c/sub\u003e          |\n| 0.1          | short-sighted      | `[-3.98, 2.22]`  | `[10.22]`     | a\u003csub\u003e1\u003c/sub\u003e          | a\u003csub\u003e0\u003c/sub\u003e          |\n| 0.5          | half short-sighted | `[1.00, 4.00]`   | `[12.00]`     | a\u003csub\u003e1\u003c/sub\u003e          | a\u003csub\u003e0\u003c/sub\u003e          |\n| 0.9          | far-sighted        | `[21.04, 20.94]` | `[28.94]`     | a\u003csub\u003e0\u003c/sub\u003e          | a\u003csub\u003e0\u003c/sub\u003e          |\n\n**Graphic:**\n\n\u003cimg src=\"/images/BasicResult.png\" width=\"792\" alt=\"basic example result\"\u003e\n\n#### 2.2.3 More complex example\n\nLet's look at the somewhat more complex example:\n\n\u003cimg src=\"/images/Complex.png\" width=\"960\" alt=\"super basic example\"\u003e\n\nShort-sightedly it is a good idea to permanently go through a\u003csub\u003e0\u003c/sub\u003e and stay on state s\u003csub\u003e0\u003c/sub\u003e. But how is it farsighted? Is courage rewarded in this case? Let's calculate that:\n\n##### 2.2.3.1 Code\n\n```javascript\nvar discountRate =  0.9;\n\nvar rl = new ReinforcementLearning.mdp();\n\n/* s0, s1 and s2 */\nvar s0 = rl.addState();\nvar s1 = rl.addState();\nvar s2 = rl.addState();\n\n/* s0.a0 and s0.a1 */\nrl.addAction(s0, new StateChange(s0, 1.0,   1));\nrl.addAction(s0, new StateChange(s0, 0.5,  -2), new StateChange(s1, 0.5, 0));\n\n/* s1.a0 and s1.a1 */\nrl.addAction(s1, new StateChange(s1, 1.0,   0));\nrl.addAction(s1, new StateChange(s2, 1.0, -50));\n\n/* s2.a0 */\nrl.addAction(s2, new StateChange(s0, 0.8, 100), new StateChange(s1, 0.1, 0), new StateChange(s2, 0.1, 0));\n\nvar Q = rl.calculateQ(discountRate);\n\nconsole.log(JSON.stringify(Q));\n```\n\n**It returns:**\n\n```json\n[\n    [61.75477734479686, 67.50622243150205],\n    [76.25766751820726, 84.73165595751362],\n    [149.70275422340958]\n]\n```\n\nLooking at the example far-sightedly (`discountRate = 0.9`) it is a good idea to take action a\u003csub\u003e1\u003c/sub\u003e in status s\u003csub\u003e0\u003c/sub\u003e, take action a\u003csub\u003e1\u003c/sub\u003e in status s\u003csub\u003e1\u003c/sub\u003e and take action a\u003csub\u003e0\u003c/sub\u003e in status s\u003csub\u003e2\u003c/sub\u003e. \n\n##### 2.2.3.2 Watch at the [demo](demo/rl-more-complex.html)\n\nDiscount rate 0,9:\n\n\u003cimg src=\"/images/MoreComplexDemo.png\" width=\"852\" alt=\"more complex example\"\u003e\n\n##### 2.2.3.3 Comparison of different discount rates\n\n| discountRate | type | s\u003csub\u003e0\u003c/sub\u003e | s\u003csub\u003e1\u003c/sub\u003e | s\u003csub\u003e2\u003c/sub\u003e | s\u003csub\u003e0\u003c/sub\u003e (winner) | s\u003csub\u003e1\u003c/sub\u003e (winner) | s\u003csub\u003e2\u003c/sub\u003e (winner) |\n|---|-------------------------------------|----------------|----------------|----------|----------------|----------------|----------|\n| 0.0 | short-sighted | `[1, -1]` | `[0, -50]` | `[80]` | a\u003csub\u003e0\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e |\n| 0.1 | short-sighted | `[1.11, -0.94]` | `[0, -41.91]` | `[80.90]` | a\u003csub\u003e0\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e |\n| 0.5 | half short-sighted | `[2, -0.5]` | `[0, -7.47]` | `[85.05]` | a\u003csub\u003e0\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e |\n| 0.9 | far-sighted | `[61.76, 67.51]` | `[76.27, 84.74]` | `[149.71]` | a\u003csub\u003e1\u003c/sub\u003e | a\u003csub\u003e1\u003c/sub\u003e | a\u003csub\u003e0\u003c/sub\u003e |\n\n**Graphic:**\n\n\u003cimg src=\"/images/ComplexResult0.0.png\" width=\"960\" alt=\"more complex example result\"\u003e\n\n\u003cimg src=\"/images/ComplexResult0.1.png\" width=\"960\" alt=\"more complex example result\"\u003e\n\n\u003cimg src=\"/images/ComplexResult0.5.png\" width=\"960\" alt=\"more complex example result\"\u003e\n\n\u003cimg src=\"/images/ComplexResult0.9.png\" width=\"960\" alt=\"more complex example result\"\u003e\n\n#### 2.2.4 Real example\n\n##### 2.2.4.1 Code\n\nIn progress..\n\n##### 2.2.4.2 Watch at the [demo](demo/rl-real.html)\n\nIn progress..\n\n##### 2.2.4.3 Comparison of different discount rates\n\nIn progress..\n\n## 3. Temporal Difference Learning and Q-Learning\n\n### 3.1 Theory\n\n#### 3.1.1 Formula\n\nIn Progress\n\n### 3.2 Usage\n\n#### 3.2.1 Real example\n\n##### 3.2.1.1 Code\n\nIn progress..\n\n##### 3.2.1.2 Watch at the [demo](demo/rl-real-q-learning.html)\n\nIn progress..\n\n#### 3.2.2 Simple Grid World\n\nImagine we have a person who is currently on the field x=5 and y=3. The goal is the safe way to x=1 and y=3. The red fields must be avoided. They are chasms that endanger the person (negative rewards or punishments). Which way should the person go?\n\n\u003cimg src=\"/images/GridWorldRawSmall.png\" width=\"345\" alt=\"grid world raw small\"\u003e\n\nLet's calculate that:\n\n##### 3.2.2.1 Code\n\n```javascript\nvar discountRate = 0.95;\nvar width  = 5;\nvar height = 3;\nvar R      = {\n    0: {2: 100},\n    1: {2: -10},\n    2: {2: -10, 1: -10},\n    3: {2: -10},\n    4: {2: 0, 0: -10}\n};\n\n/* create the q-learning instance */\nvar rlQLearning = new ReinforcementLearning.qLearning();\n\n/* build the grid world */\nrlQLearning.buildGridWorld(width, height, R);\n\n/* calculate Q */\nvar Q = rlQLearning.calculateQ(discountRate, {\n    iterations: 100000,\n    useSeededRandom: true,\n    useOptimizedRandom: true\n});\n\n/* print result */\nrlQLearning.printTableGridWorld(Q, width, R);\n```\n\n**It returns:**\n\n\u003cimg src=\"/images/GridWorldCalculatedSmall.png\" width=\"345\" alt=\"grid world calculated small\"\u003e\n\n##### 3.2.2.2 Watch at the [demo](demo/rl-grid-world.html)\n\nIn progress.\n\n#### 3.2.3 Extended Grid World\n\nAs in the example 3.2.2 but just a bigger grid world:\n\n```javascript\nvar width  = 10;\nvar height = 5;\nvar R      = {\n    0: {4: 100},\n    2: {4: -10},\n    3: {4: -10, 3: -10},\n    4: {4: -10},\n    5: {4: 0, 0: -10}\n};\n```\n\nThat's easy now:\n\n\u003cimg src=\"/images/GridWorldCalculatedWide.png\" width=\"682\" alt=\"grid world raw wide\"\u003e\n\nNow imagine that the person is drunk. That means with a certain probability the person goes to the right or left, although he wanted to go straight out. Depending on how drunk the person is we choose a probability of 2.5% to go left and a probability of 2.5% to go right (`splitT = 0.025`). What is the safest way now? **Preliminary consideration:** Moving away from the chasms first and staying away from them might now be better than taking the shortest route.\n\nLet's calculate that:\n\n##### 3.2.3.1 Code\n\n```javascript\nvar discountRate = 0.95;\nvar width  = 10;\nvar height = 5;\nvar R      = {\n    0: {4: 100},\n    2: {4: -10},\n    3: {4: -10, 3: -10},\n    4: {4: -10},\n    5: {4: 0, 0: -10}\n};\nvar splitT = 0.025;\n\n/* create the q-learning instance */\nvar rlQLearning = new ReinforcementLearning.qLearning();\nrlQLearning.adoptConfig({splitT: splitT});\n\n/* build the grid world */\nrlQLearning.buildGridWorld(width, height, R);\n\n/* calculate Q */\nvar Q = rlQLearning.calculateQ(discountRate, {\n    iterations: 100000,\n    useSeededRandom: true,\n    useOptimizedRandom: true\n});\n\n/* print result */\nrlQLearning.printTableGridWorld(Q, width, R);\n```\n\n**It returns:**\n\n\u003cimg src=\"/images/GridWorldCalculatedWideDrunk.png\" width=\"682\" alt=\"grid world calculated small drunk\"\u003e\n\n##### 3.2.3.2 Watch at the [demo](demo/rl-grid-world.html)\n\nIn progress.\n\n## A. Tools\n\n* All flowcharts were gratefully created with [Google Drive](https://www.google.com/drive/)\n\n## B. Authors\n\n* Björn Hempel \u003cbjoern@hempel.li\u003e - _Initial work_ - [https://github.com/bjoern-hempel](https://github.com/bjoern-hempel)\n\n## C. Licence\n\nThis tutorial is licensed under the MIT License - see the [LICENSE.md](/LICENSE.md) file for details\n\n## D. Closing words\n\nHave fun! :)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbjoern-hempel%2Fjs-reinforcement-learning-framework","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbjoern-hempel%2Fjs-reinforcement-learning-framework","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbjoern-hempel%2Fjs-reinforcement-learning-framework/lists"}