{"id":19509394,"url":"https://github.com/ondrejbiza/racetrack","last_synced_at":"2025-04-26T03:31:53.778Z","repository":{"id":111605720,"uuid":"136221951","full_name":"ondrejbiza/racetrack","owner":"ondrejbiza","description":"An environment for tabular Reinforcement Learning agents.","archived":false,"fork":false,"pushed_at":"2018-06-13T11:36:30.000Z","size":211,"stargazers_count":13,"open_issues_count":0,"forks_count":5,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-04T07:23:25.949Z","etag":null,"topics":["machine-learning","reinforcement-learning","reinforcement-learning-agent","reinforcement-learning-algorithms","sutton-book"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ondrejbiza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-06-05T18:59:38.000Z","updated_at":"2024-04-21T20:49:50.000Z","dependencies_parsed_at":"2023-06-13T13:00:42.543Z","dependency_job_id":null,"html_url":"https://github.com/ondrejbiza/racetrack","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fracetrack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fracetrack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fracetrack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Fracetrack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ondrejbiza","download_url":"https://codeload.github.com/ondrejbiza/racetrack/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250926823,"owners_count":21509044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["machine-learning","reinforcement-learning","reinforcement-learning-agent","reinforcement-learning-algorithms","sutton-book"],"created_at":"2024-11-10T23:11:51.926Z","updated_at":"2025-04-26T03:31:53.772Z","avatar_url":"https://github.com/ondrejbiza.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Racetrack with Monte Carlo #\n\nThis is an implementation of the Racetrack environment and the Monte Carlo agent described in Sutton and Barto's \nReinforcement Learning: An Introduction, Chapter 5. My agent can solve the problem posed in the book and I further \nexperiment with increasing the difficult of the environment.\n\n## Environment ##\n\nThe book recommends two environments (the 2nd and 3rd picture), I added one more for debugging.\nThe third environment is bigger than the second one but is in fact easier to solve.\n\nThe race car starts at the bottom and should move to the finish, which is in the top right corner, as fast as possible.\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"images/track_1.svg\" alt=\"track_1\" height=\"320px\"  /\u003e\n  \u003cimg src=\"images/track_2.svg\" alt=\"track_2\" height=\"320px\" /\u003e\n  \u003cimg src=\"images/track_3.svg\" alt=\"track_3\" height=\"320px\" /\u003e\n\u003c/p\u003e\n\n```\nfor track in track_1 track_2 track_3;\n  do python -m scripts.show_racetrack \"$track\" --save-path \"images/$track.svg\";\ndone\n```\n\n## Results ##\n\n### Original Problem ###\n\nIn the original problem, the car is penalized when it attempts to leave the track but the episode does not terminate.\nInstead, the car is placed back on the track and the environment makes sure that the car always moves by at least one square towards the finish.\nMoreover, the car can only move to the top and to the right.\nTo make it harder, the car's position is randomly shifted up or right by one square with a probability of 50%.\n\nA (tabular) Monte Carlo agent can successfully solve this task after 50000 training episode. I show three testing episodes\nfor each of the environments below.\n\nThe agent learns to drive in all three tracks. However, it sometimes gets penalized for driving off the track\nbecause of the random noise added to the car's position.\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"images/track_1_episode_1.svg\" alt=\"track_1_episode_1\" height=\"320px\" /\u003e\n  \u003cimg src=\"images/track_1_episode_2.svg\" alt=\"track_1_episode_2\" height=\"320px\" /\u003e\n  \u003cimg src=\"images/track_1_episode_3.svg\" alt=\"track_1_episode_3\" height=\"320px\" /\u003e\n\u003c/p\u003e\n\n```\npython -m scripts.solve_racetrack track_1 0.1 --save-path images/track_1_episode\n```\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"images/track_2_episode_1.svg\" alt=\"track_2_episode_1\" height=\"460px\" /\u003e\n  \u003cimg src=\"images/track_2_episode_3.svg\" alt=\"track_2_episode_3\" height=\"460px\" /\u003e\n  \u003cimg src=\"images/track_2_episode_5.svg\" alt=\"track_2_episode_5\" height=\"460px\" /\u003e\n\u003c/p\u003e\n\n```\npython -m scripts.solve_racetrack track_2 0.1 --save-path images/track_2_episode\n```\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"images/track_3_episode_1.svg\" alt=\"track_3_episode_1\" height=\"260px\" /\u003e\n  \u003cimg src=\"images/track_3_episode_9.svg\" alt=\"track_3_episode_9\" height=\"260px\" /\u003e\n  \u003cimg src=\"images/track_3_episode_19.svg\" alt=\"track_3_episode_19\" height=\"260px\" /\u003e\n\u003c/p\u003e\n\n\n```\npython -m scripts.solve_racetrack track_3 0.1 --save-path images/track_3_episode\n```\n\n### Strict Version ###\n\nThe strict version of the environment terminates the episode when the car attempts to leave the track. Unlike the original problem,\nthe car rarely reaches the finish line when taking random actions. I disabled the random noise added to the car's position because\nthe environment is already difficult to solve.\n\nThe agent succeeds in the first racetrack.\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"images/track_1_strict_episode_1.svg\" alt=\"track_1_strict_episode_1\" height=\"320px\" /\u003e\n  \u003cimg src=\"images/track_1_strict_episode_2.svg\" alt=\"track_1_strict_episode_2\" height=\"320px\" /\u003e\n  \u003cimg src=\"images/track_1_strict_episode_3.svg\" alt=\"track_1_strict_episode_3\" height=\"320px\" /\u003e\n\u003c/p\u003e\n\n```\npython -m scripts.solve_racetrack track_1 0.1 --save-path images/track_1_strict_episode --strict\n```\n\nHowever, the agent always terminates the episode as early as possible in the second track. This happens because\nthe agent almost never reaches the finish line during training, even with 100% exploration (ε=1.0).\n\n\u003cp float=\"left\"\u003e\n  \u003cimg src=\"images/track_2_strict_episode_3.svg\" alt=\"track_2_strict_episode_2\" height=\"460px\" /\u003e\n  \u003cimg src=\"images/track_2_strict_eps_0.5_episode_2.svg\" alt=\"track_2_strict_eps_0.5_episode_2\" height=\"460px\" /\u003e\n  \u003cimg src=\"images/track_2_strict_eps_1.0_episode_2.svg\" alt=\"track_2_strict_eps_1.0_episode_2\" height=\"460px\" /\u003e\n\u003c/p\u003e\n\n```\npython -m scripts.solve_racetrack track_2 0.1 --save-path images/track_2_strict_episode --strict\npython -m scripts.solve_racetrack track_2 0.5 --save-path images/track_2_strict_eps_0.5_episode --strict\npython -m scripts.solve_racetrack track_2 1.0 --save-path images/track_2_strict_eps_1.0_episode --strict\n```\n\nEven though the third track is bigger, it is much more likely that the agent will reach the finish by taking random actions.\nThe trained agent usually wins but the learned policy is not perfect, as you can see in the second picture.\n\n![track_3_strict_episode_1](images/track_3_strict_episode_1.svg)\n![track_3_strict_episode_23](images/track_3_strict_episode_23.svg)\n\n```\npython -m scripts.solve_racetrack track_3 0.1 --save-path images/track_3_strict_episode --strict\n```\n\n## Setup ##\n\nInstall Python3 and all Python packages listed in **requirements.txt**.\n\n## Usage ##\n\nThe environments are implemented in **environment.py** and the agent in **agent.py**. You can replicate all the \nplots by running the commands listed under them.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrejbiza%2Fracetrack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fondrejbiza%2Fracetrack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrejbiza%2Fracetrack/lists"}