{"id":31944431,"url":"https://github.com/kuohaozeng/cartpole_model_based_control","last_synced_at":"2025-10-14T10:25:58.412Z","repository":{"id":40961489,"uuid":"256369888","full_name":"KuoHaoZeng/cartpole_model_based_control","owner":"KuoHaoZeng","description":"Behaviour Cloning of Cartpole Swing-up Policy with Model-Predictive Uncertainty Regularization (UW CSE571 Guided Project)","archived":false,"fork":false,"pushed_at":"2023-07-06T21:58:54.000Z","size":41360,"stargazers_count":1,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-02-04T10:58:56.607Z","etag":null,"topics":["behavioral-cloning","cartpole","computer-vision","deep-learning","model-based-rl","uncertainty-estimation","uncertainty-neural-networks"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KuoHaoZeng.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-04-17T01:25:54.000Z","updated_at":"2024-02-04T10:58:56.608Z","dependencies_parsed_at":"2022-08-27T02:49:54.718Z","dependency_job_id":null,"html_url":"https://github.com/KuoHaoZeng/cartpole_model_based_control","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KuoHaoZeng/cartpole_model_based_control","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KuoHaoZeng%2Fcartpole_model_based_control","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KuoHaoZeng%2Fcartpole_model_based_control/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KuoHaoZeng%2Fcartpole_model_based_control/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KuoHaoZeng%2Fcartpole_model_based_control/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KuoHaoZeng","download_url":"https://codeload.github.com/KuoHaoZeng/cartpole_model_based_control/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KuoHaoZeng%2Fcartpole_model_based_control/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279018775,"owners_count":26086452,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["behavioral-cloning","cartpole","computer-vision","deep-learning","model-based-rl","uncertainty-estimation","uncertainty-neural-networks"],"created_at":"2025-10-14T10:25:57.656Z","updated_at":"2025-10-14T10:25:58.405Z","avatar_url":"https://github.com/KuoHaoZeng.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Behaviour Cloning of Cartpole Swing-up Policy with Model-Predictive Uncertainty Regularization\n\n### (UW CSE571 Guided Project 1)\n\nBy Kuo-Hao Zeng, Pengcheng Chen, Mengying Leng, Xiaojuan Wang\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"fig/cartpole_BC.gif\" style=\"zoom:75%;\" /\u003e\u003c/p\u003e\n\n### **Introduction**\n\nIn this project, we adopt the idea of uncertainty regularization [1] to learn a swing-up policy via behaviour cloning without interacting with the simulator. We make several modifications to adapt the learning framework for our focused task. We make several modifications to adapt the learning framework for cartpole swing-up task:\n\n- Since our policy learning entirely relies on BC, our policy network does not need to interact with the environment during the training phase. Therefore, we remove the simulator from our learning framework, except for the data collection process.\n- We use state observation instead of image observation to ease the learning of dynamic model. In this case, we are able to focus on the effectiveness of uncertainty regularization approach.\n- We slightly modify the learning framework by changing the policy cost to behaviour cloning objective to fit our problem setting.\n- To make the focused task simple, we do not adopt the z-dropout technique proposed by original authors, we rather directly utilize the simplest dropout technique to perform Bayesian Neural Network (e.g., generate sub-network on-the-fly by different dropout masks).\n\nCheck out our [introduction video](https://drive.google.com/file/d/1WZJ4ulTXD3X9bEGl0UXfBG9y3CgtK0Ts/view), [final report](pdf/final_report.pdf), and some [qualitative results](https://drive.google.com/open?id=1vRpvi3G-4KpnD6k95rBvm104NjcYg501).\n\n### Set Up\n\n1. Clone this repository\n\n   ```\n   git clone git@github.com:KuoHaoZeng/cartpole_model_based_control.git\n   ```\n   \n4. Using `python 3.6`, create a `venv`\n\n   **Note**: The `python` version needs to be above `3.6` to match the original carpole codebase provided by ETAs\n   \n   ```\n   # Create venv and execute it\n   python -m venv venv \u0026\u0026 source venv/bin/activate\n   ```\n   \n4. Install the requirements with\n\n   ```\n   # Make sure you execute this under (venv) environment\n   pip install -r requirements.txt\n   ```\n\n### Train and evaluate it!\n\nYou can always change or adjust the hyperparameters defined in the config file to change the setting such as how often you want to store a checkpoint, how large the initial learning rate you are going to use, what batch size you are going to use etc.\n\n#### Pretrain a dynamic model\n\n```\n# Train and test\npython main.py --config configs/dm_state.yaml\n```\n\nThe default model is dropout LSTM with dropout rate = 0.05. You can change them in the config file:\n\n```\nmodel:\n    ...\n    backbone: dlstm # {fc, gru, lstm, dfc, dgru, dlstm} \u003c--- change the model backbone here\n    ...\n    dropout_p: 0.05 # only work for the model has dropout layer \u003c--- change the dropout rate here\n```\n\n#### Main Results for dynamics model\n\n|    Model     | L2 difference with simulator |\n| :----------: | :--------------------------: |\n|      FC      |         0.528±0.079          |\n|     GRU      |         0.354±0.063          |\n|     LSTM     |         0.229±0.058          |\n|  Dropout FC  |         0.559±0.114          |\n| Dropout GRU  |         0.416±0.064          |\n| Dropout LSTM |         0.252±0.040          |\n\n#### Learn a swing-up by uncertainty regularization with the pretrained dynamic model\n\n```\n# Train and test\npython main.py --config configs/mp_state.yaml\n```\n\n**Note**: you need to make sure the dynamics model defined in the `mp_state.yaml` pointing to the correct pretrained dynamics model:\n\n```\ndm_model:\n    ......\n    model:\n        protocol: state\n        backbone: dlstm # {fc, gru, lstm, dfc, dgru, dlstm} \u003c--- change the model backbone here\n        ...\n        dropout_p: 0.05 # only work for the model has dropout layer \u003c--- change the dropout rate here\n```\n\n#### Do experiments on policy learning with a pretrained drop LSTM model with various experimental settings\n\nAssuming you have pretrained the dynamics model with `dlstm`, the following script performs experiments with different hyparparameters setting defined in `experiment.py`.\n\n```\n# Train and test model with different experimental settings\n# --n: indicates how many workers (n) you want to spawn for doing the experiments\npython experiment.py --config configs/mp_state.yaml --n 4\n```\n\nYou can change the hyparparameters which you would like to try in the `experiment.py`:\n\n```\nif __name__ == \"__main__\":\n    options = {\n        \"framework.seed\": [12345], # \u003c--- indicates what are the random seeds you want to try\n        \"dm_model.model.backbone\": [\"dlstm\"], # \u003c--- indicates what are the backbones for dynamics model you want to try\n        \"model.backbone\": [\"fc\", \"dfc\", \"gru\", \"dgru\", \"lstm\", \"dlstm\"], # \u003c--- indicates what are the backbones for policy network you want to try\n        \"train.LAMBDA\": [0.0, 0.01, 0.1, 0.15], # \u003c--- indicates what are the lambda for policy learning you want to try\n    }\n```\n\nYou can easily add experimental options based on the hyperparameters defined in the config files. For example, do experiments with different initial learning rate:\n\n```\nif __name__ == \"__main__\":\n    options = {\n        \"train.lr\": [0.1, 0.01, 0.001],\n    }\n```\n\n#### Main Results for policy learning with uncertainty regularzation\n\n|            Dynamics Model \\ Policy Network             |    FC     |    GRU    |   LSTM    | Drpopout LSTM |\n| :----------------------------------------------------: | :-------: | :-------: | :-------: | :-----------: |\n| Dropout LSTM w/ **λ** = 0 (original behaviour cloning) |   0.649   | **0.537** |   0.534   |     0.539     |\n|              Dropout LSTM w/ **λ** = 0.01              | **0.629** |   0.543   |   0.516   |   **0.527**   |\n|              Dropout LSTM w/ **λ** = 0.1               |   0.631   |   0.540   |   0.527   |     0.554     |\n|              Dropout LSTM w/ **λ** = 0.15              |   0.646   |   0.550   | **0.510** |     0.539     |\n\n### Reference\n\n[1] Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with uncertainty regularization for driving in dense traffic. In *ICLR*, 2019.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuohaozeng%2Fcartpole_model_based_control","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkuohaozeng%2Fcartpole_model_based_control","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkuohaozeng%2Fcartpole_model_based_control/lists"}