{"id":19509404,"url":"https://github.com/ondrejbiza/bandits","last_synced_at":"2025-10-25T13:37:21.692Z","repository":{"id":111604716,"uuid":"135427910","full_name":"ondrejbiza/bandits","owner":"ondrejbiza","description":"Comparison of bandit algorithms from the Reinforcement Learning bible.","archived":false,"fork":false,"pushed_at":"2018-06-06T09:31:01.000Z","size":1415,"stargazers_count":17,"open_issues_count":1,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T07:23:36.849Z","etag":null,"topics":["machine-learning","reinforcement-learning","reinforcement-learning-agent","reinforcement-learning-algorithms","sutton-book"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ondrejbiza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-30T10:37:48.000Z","updated_at":"2023-06-19T13:23:06.000Z","dependencies_parsed_at":null,"dependency_job_id":"076da3c3-5cd4-4835-bb41-9624d454fa68","html_url":"https://github.com/ondrejbiza/bandits","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fbandits","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fbandits/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fbandits/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fbandits/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ondrejbiza","download_url":"https://codeload.github.com/ondrejbiza/bandits/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250926823,"owners_count":21509044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","reinforcement-learning","reinforcement-learning-agent","reinforcement-learning-algorithms","sutton-book"],"created_at":"2024-11-10T23:11:53.691Z","updated_at":"2025-10-25T13:37:16.660Z","avatar_url":"https://github.com/ondrejbiza.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Bandits\n\nExperiments with bandit algorithms from the 2nd chapter of Sutton and Barto's Reinforcement Learning: An Introduction.\n\n## Results\n\nYou can generate each plot with the command written under it.\n\n### Stationary Environment\n\nThe experimental setup follows the one described in the book:\n\nThe values for each action are drawn from a normal distribution with zero mean and unit variance and do not change \nduring the experiment. The bandits take 1000 steps in the environment choosing from 10 actions at each step.\nFurthermore, the bandits observe rewards with noise drawn from normal distribution with zero mean and unit variance \nadded to the action values. The experiments are repeated 2000 times.\n\nI plot the average reward and the percentage of times the bandit chose the optimal actions.\n\n#### Comparison from the book\n\nI replicated Figure 2.1 from the book to check my implementation. ε-greedy bandit outperforms a greedy bandit in this \nsimple testbed.\n\n![plot_from_book_1](images/book_1_rewards.svg)\n![plot_from_book_2](images/book_1_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/book_1 -a epsilon epsilon epsilon -s 0.0 0.01 0.1 -l \"ε=0\", \"ε=0.01\" \"ε=0.1\" -t \"ε-greedy bandits\"\n```\n\n#### Epsilon-greedy bandits\n\nNext, I compare ε-greedy bandits with different exploration settings. ε=0.1 performs the best.\n\n![epsilon_1](images/epsilon_rewards.svg)\n![epsilon_2](images/epsilon_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/epsilon -a epsilon epsilon epsilon epsilon epsilon epsilon -s 0.0 0.01 0.1 0.2 0.5 1.0 -l \"ε=0\" \"ε=0.01\" \"ε=0.1\" \"ε=0.2\" \"ε=0.5\" \"ε=1.0\" -t \"ε-greedy bandits\"\n```\n\n#### Softmax bandits\n\nAnother type of bandit presented in the book is the softmax bandit. Softmax bandits should perform better than \nε-greedy bandits because they avoid bad actions during exploration. However, they are quite sensitive to the \ntemperature (τ) parameter setting.\n\n![softmax_1](images/softmax_rewards.svg)\n![softmax_2](images/softmax_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/softmax -a softmax softmax softmax softmax softmax -s 0.1 0.2 0.5 1.0 2.0 -l \"τ=0.1\" \"τ=0.2\" \"τ=0.5\" \"τ=1.0\" \"τ=2.0\" -t \"softmax bandits\"\n```\n\n#### Optimistic initialization\n\nOptimistic Initialization is an alternative to ε-greedy or softmax exploration policies. It outperforms the ε-greedy \nbandit in this simple environment but has some drawback (e.g. it cannot track non-stationary rewards). Interestingly,\nthe optimistically initialized bandit chooses the optimal action with lower frequency than the ε-greedy bandit\nbut still achieves higher average reward.\n\n![optimistic_init_1](images/optimistic_init_rewards.svg)\n![optimistic_init_2](images/optimistic_init_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/optimistic_init -a epsilon epsilon -s 0.0 0.1 -i 5.0 0.0 -l \"ε=0, init=5\" \"ε=0.1, init=0\" -t \"Optimistic Initialization\"\n```\n\n#### Final Comparison\n\nFinally, I compare the best ε-greedy, softmax and optimistically initialized bandits. The softmax bandit wins by a \nsmall margin.\n\n![epsilon_vs_softmax_vs_optimistic_1](images/epsilon_vs_softmax_vs_optimistic_rewards.svg)\n![epsilon_vs_softmax_vs_optimistic_2](images/epsilon_vs_softmax_vs_optimistic_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/epsilon_vs_softmax_vs_optimistic -a epsilon epsilon softmax -s 0.1 0.0 0.2 -l \"ε=0.1, init=0\", \"ε=0, init=5\" \"τ=0.2, init=0\" -i 0.0 5.0 0.0\n```\n\n### Non-stationary Environment\n\nIn this experiment, all action values start at 0. After all agents perform a single action, the action values take a \nsmall random step drawn from a normal distribution. Therefore, the action values change as the bandits interact with \nthe environment.\n\nI compare the ε-greedy bandit from the previous section with a modified version that uses a constant α during sample \naveraging. Constant α value causes it to prioritize recent rewards, which models the non-stationary environment better.\n\nThe agents take 5000 steps in the environment instead of 1000, so that we can see the gap between the two agents \nincrease.\n\n![non_stationary_bandits_1](images/nonstationary_rewards.svg)\n![non_stationary_bandits_2](images/nonstationary_actions.svg)\n\n```\npython -m scripts.compare_bandits_nonstationary images/nonstationary -a epsilon epsilon -s 0.1 0.1 --alphas 0.0 0.1 -l \"α=1/k\" \"α=0.1\" -t \"ε-greedy bandits, ε=0.1\"\n```\n\n### Advanced Bandits in a Stationary Environment ###\n\n#### UCB ####\n\n[UCB bandits](https://webdocs.cs.ualberta.ca/~games/go/seminar/notes/2007/slides_ucb.pdf) \nestablish an upper bound on regret–how much we loose for not playing the optimal action.\nUCB1 beats ε-greedy while not having any parameters to tune.\n\n![ucb1_1](images/ucb1_rewards.svg)\n![ucb1_2](images/ucb1_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/ucb1 -a ucb1 epsilon -s 0.0 0.1 -l \"UCB1\" \"ε-greedy (ε=0.1)\"\n```\n\nUCB2 tightens the bound on the regret but introduces an additional parameter α.\n\n![ucb2_alpha_1](images/ucb2_alpha_rewards.svg)\n![ucb2_alpha_2](images/ucb2_alpha_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/ucb2_alpha -a ucb2 ucb2 ucb2 ucb2 -s 0.001 0.01 0.1 0.5 -l \"α=0.001\" \"α=0.01\" \"α=0.1\" \"α=0.5\" -t UCB2\n```\n\nUCB2 reaches optimal performance faster than UCB1..\n\n![ucb2_1](images/ucb2_rewards.svg)\n![ucb2_2](images/ucb2_actions.svg)\n\n```\npython -m scripts.compare_bandits_stationary images/ucb2 -a ucb2 ucb1 epsilon -s 0.5 0.0 0.1 -l \"UCB2 (α=0.5)\" \"UCB1\" \"ε-greedy (ε=0.1)\"\n```\n\n## Setup\n\nInstall Python 3 and all packages listed in requirements.txt.\n\n## Usage\n\nEach scripts contains documentation for all arguments.\n\nFor stationary experiments, execute:\n\n```\npython -m scripts.compare_bandits_stationary -h\n```\n\nand for non-stationary experiments:\n\n```\npython -m scripts.compare_bandits_nonstationary -h\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrejbiza%2Fbandits","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fondrejbiza%2Fbandits","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrejbiza%2Fbandits/lists"}