{"id":19509397,"url":"https://github.com/ondrejbiza/aamas_19","last_synced_at":"2025-09-18T23:26:08.910Z","repository":{"id":111604661,"uuid":"172695907","full_name":"ondrejbiza/aamas_19","owner":"ondrejbiza","description":"Source code for the paper \"Online Abstraction with MDP Homomorphisms for Deep Learning\".","archived":false,"fork":false,"pushed_at":"2021-03-02T18:13:33.000Z","size":122,"stargazers_count":5,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-04-04T07:23:10.353Z","etag":null,"topics":["aamas","abstraction","deep-learning","deep-neural-networks","reinforcement-learning","reinforcement-learning-algorithms"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1811.12929","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ondrejbiza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-02-26T11:09:59.000Z","updated_at":"2024-03-21T08:08:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"797a45eb-71c2-4530-9157-4f8ccda9d24a","html_url":"https://github.com/ondrejbiza/aamas_19","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Faamas_19","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Faamas_19/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Faamas_19/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ondrejbiza%2Faamas_19/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ondrejbiza","download_url":"https://codeload.github.com/ondrejbiza/aamas_19/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250926823,"owners_count":21509044,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aamas","abstraction","deep-learning","deep-neural-networks","reinforcement-learning","reinforcement-learning-algorithms"],"created_at":"2024-11-10T23:11:52.502Z","updated_at":"2025-09-18T23:26:03.844Z","avatar_url":"https://github.com/ondrejbiza.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Online Abstraction with MDP Homomorphisms for Deep Learning—source code\n\nThis repository contains the source code to our [AAMAS'19 paper](https://arxiv.org/abs/1811.12929). \nThe aim of the paper is to find abstractions\nin the form of MDP homomorphisms based on experience collected by a Deep Reinforcement Learning agent.\nWe use a fully-convolutional deep Q-network to collect the experience.\n\n##  Setup\n\n* Install Python \u003e= 3.5.\n* Install all packages listed in requirements.txt: `pip install -r requirements.txt`.\n* I use tensorflow-gpu 1.7 with CUDA 9.1 and cuDNN 7.1; any other setup might produce different results.\n\n## Usage\n\n### Train a deep Q-network\n\n#### Discrete environments\n\nTrain a deep Q-network to stack 2 pucks in a grid world environment:\n```\npython -m abstract.scripts.solve.puck_stack_n.dqn_branch 2 4 \n    --max-time-steps 2500 --max-episodes 200 --learning-rate 0.0001 --batch-size 30\n```\nStacking three pucks:\n```\npython -m abstract.scripts.solve.puck_stack_n.dqn_branch 3 4 \n    --max-episodes 1000 --max-time-steps 20000 --exploration-fraction 0.25 \n    --learning-rate 0.0001 --batch-size 30\n```\n\n#### Fully convolutional network for pseudo-continuous environments\n\nTrain a deep Q-network on the continuous component task:\n```\n# 2, 3 or 4 pucks should work\nnum_pucks=2\n\npython -m abstract.scripts.solve.continuous_component.dqn_fc 4 112 ${num_pucks} \\\n        --max-time-steps 400000 --max-episodes 15000 \\\n        --learning-rate 0.0001 --exploration-fraction 0.025 \\\n        --num-filters 32 64 64 32 --filter-sizes 8 8 3 1 --strides 4 2 1 1 \\\n        --upsample upsample_after\n```\nBuilding stairs:\n```\n# 3 or 6 pucks; the latter would require a lot more time steps (perhaps in the millions)\nnum_pucks=3\n\npython -m abstract.scripts.solve.continuous_stairs.dqn_fc 4 112 ${num_pucks} \\\n        --max-time-steps 400000 --max-episodes 15000 \\\n        --learning-rate 0.0001 --exploration-fraction 0.025 \\\n        --num-filters 32 64 64 32 --filter-sizes 8 8 3 1 --strides 4 2 1 1 \\\n        --upsample upsample_after\n```\nStacking pucks:\n```\n# 2 or 3 pucks; stacking 4 pucks would require a lot of time steps\nnum_pucks=2\n\npython -m abstract.scripts.solve.continuous_puck_stack_n.dqn_fc 4 112 ${num_pucks} \\\n        --max-time-steps 400000 --max-episodes 15000 \\\n        --learning-rate 0.0001 --exploration-fraction 0.025 \\\n        --num-filters 32 64 64 32 --filter-sizes 8 8 3 1 --strides 4 2 1 1 \\\n        --upsample upsample_after\n```\n\n### Collect data for the abstraction algorithm\n\n#### Discrete environment\n\nThe transfer script for the discrete environment collects the initial experience during each run.\n\n#### Pseudo-continuous environments\nYou need to collect the data for abstraction using the following shell scripts:\n\n```\n./abstract/shell_scripts/abstraction/continuous_component/collect_data_dqn.sh\n./abstract/shell_scripts/abstraction/continuous_puck_stack_n/collect_data_dqn.sh\n./abstract/shell_scripts/abstraction/continuous_stairs/collect_data_dqn.sh\n```\n\n### Transfer options between environments using MDP homomorphisms\n\n#### Discrete environments\n\nTransfer from 2 to 3 pucks stacking in a grid world environment:\n```\n# transfer options\npython -m abstract.scripts.abstract.puck_stack_n.dqn_exp_goal_transfer 4 1 --num-pucks-list 2 3 \\\n        --num-start-episodes 1000 --num-episodes 0 --max-buffer-size 10000 \\\n        --min-radius 7 --max-radius 12 --reuse --max-blocks 10 \\\n        --reward-threshold 0.98 --early-stop 1000 --softmax-selection --no-sharing \\\n        --dqn-final-epsilon 0.1 --dqn-num-exp-steps 5000 --state-action-threshold 400\n\n# transfer weights\npython -m abstract.scripts.abstract.puck_stack_n.dqn_exp_goal_transfer 4 1 --num-pucks-list 2 3 \\\n        --num-start-episodes 1000 --num-episodes 0 --max-buffer-size 10000 \\\n        --min-radius 7 --max-radius 12 --reuse --no-sharing \\\n        --dqn-final-epsilon 0.1 --dqn-num-exp-steps 5000 --no-option \\\n        --share-dqn --share-dqn-reset-buffer\n```\n\nTransfer from 3 pucks stacking to 2 and 2 puck stacking in a grid world environment:\n```\n# transfer options\npython -m scripts.abstract.puck_stack_subgoal.dqn_exp_option_transfer 4 1 \\\n        --num-start-episodes 1500 --num-episodes 0 --max-buffer-size 10000 \\\n        --min-radius 7 --max-radius 12 --reuse --max-blocks 10 \\\n        --reward-threshold 0.98 --early-stop 1000 --softmax-selection --no-sharing \\\n        --dqn-final-epsilon 0.1 --dqn-num-exp-steps 10000 \\\n        --state-action-threshold 600 --option-learning-rate 0.1\n\n# transfer weights\npython -m scripts.abstract.puck_stack_subgoal.dqn_exp_option_transfer 4 1 \\\n        --num-start-episodes 1500 --num-episodes 0 --max-buffer-size 10000 \\\n        --min-radius 7 --max-radius 12 --reuse --max-blocks 10 \\\n        --reward-threshold 0.98 --early-stop 1000 --softmax-selection --no-sharing \\\n        --dqn-final-epsilon 0.1 --dqn-num-exp-steps 10000 --share-dqn \\\n        --no-option --share-dqn-reset-buffer\n```\n\n#### Pseudo-continuous environments\n\nWe ran many transfer experiments in the pseudo-continuous environments. The following is one example:\n```\n# transfer from 2 puck stacking to 3 component\n\n# transfer options\npython -m scripts.abstract.continuous_component.transfer_drn \"dataset/dqn/continuous_puck_stack_2_112x112.pickle\" \\\n        3 1000 10 --deduplicate --max-time-steps 400000 \\\n        --max-episodes 15000 --learning-rate 0.0001 --exploration-fraction 0.025 \\\n        --num-filters 32 64 64 32 --filter-sizes 8 8 3 1 --strides 4 2 1 1 \\\n        --upsample upsample_after --proportional-selection\n\n# transfer weights\npython -m scripts.solve.continuous_component.dqn_fc 3 112 3 \\\n        --max-time-steps 400000 --max-episodes 15000 \\\n        --learning-rate 0.0001 --exploration-fraction 0.025 \\\n        --num-filters 32 64 64 32 --filter-sizes 8 8 3 1 --strides 4 2 1 1 \\\n        --upsample upsample_after --load-weights \"dataset/dqn/continuous_puck_stack_2_112x112\"\n```\n\n## Environments\n\n* **envs/puck_stack**: stack N pucks in a discrete grid world\n* **envs/puck_stack_subgoal**: make two stacks of N pucks in a continuous grid world\n* **envs/continuous_puck_stack**: stack N pucks in a psedo-continuous environment\n* **envs/continuous_two_stack**: make two stacks of N pucks in a pseudo-continuous environment\n* **envs/continuous_component**: arrange N pucks so that they form a connected component\n* **envs/continuous_stairs**: build stairs from 3 or 6 pucks\n\n## Authors\n\n[Ondrej Biza](https://sites.google.com/view/obiza), supervised by [Robert Platt](http://www.ccs.neu.edu/home/rplatt/).","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrejbiza%2Faamas_19","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fondrejbiza%2Faamas_19","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fondrejbiza%2Faamas_19/lists"}