{"id":15683913,"url":"https://github.com/xuehaipan/soft-actor-critic","last_synced_at":"2025-05-07T14:31:46.986Z","repository":{"id":109485534,"uuid":"247768517","full_name":"XuehaiPan/Soft-Actor-Critic","owner":"XuehaiPan","description":"PyTorch Implementation of Soft Actor-Critic Algorithm","archived":false,"fork":false,"pushed_at":"2020-09-13T08:05:10.000Z","size":530,"stargazers_count":11,"open_issues_count":0,"forks_count":5,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-31T11:01:43.996Z","etag":null,"topics":["actor-critic","actor-critic-algorithm","reinforcement-learning-algorithms","soft-actor-critic"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/XuehaiPan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-03-16T16:47:07.000Z","updated_at":"2024-10-08T14:27:43.000Z","dependencies_parsed_at":"2023-04-13T13:16:54.853Z","dependency_job_id":null,"html_url":"https://github.com/XuehaiPan/Soft-Actor-Critic","commit_stats":{"total_commits":179,"total_committers":1,"mean_commits":179.0,"dds":0.0,"last_synced_commit":"9018199f28351f4106dab73a9dc3631c52b72260"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XuehaiPan%2FSoft-Actor-Critic","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XuehaiPan%2FSoft-Actor-Critic/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XuehaiPan%2FSoft-Actor-Critic/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/XuehaiPan%2FSoft-Actor-Critic/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/XuehaiPan","download_url":"https://codeload.github.com/XuehaiPan/Soft-Actor-Critic/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252895768,"owners_count":21821211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["actor-critic","actor-critic-algorithm","reinforcement-learning-algorithms","soft-actor-critic"],"created_at":"2024-10-03T17:09:13.349Z","updated_at":"2025-05-07T14:31:46.962Z","avatar_url":"https://github.com/XuehaiPan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Soft-Actor-Critic\nPyTorch Implementation of Soft Actor-Critic Algorithm\n\nReference: Soft Actor-Critic Algorithms and Applications ([https://arxiv.org/pdf/1812.05905.pdf](https://arxiv.org/pdf/1812.05905.pdf))\n\n# Main Features\n\n- fully customizable network structure\n- parallel data sampling\n- comprehensive logging\n- good performance\n\n# Requirements\n\n- [PyTorch](https://pytorch.org/)\n- [TensorBoard](https://www.tensorflow.org/tensorboard/)\n- [Gym](http://gym.openai.com/)\n- [PyBullet](https://pybullet.org/wordpress/)\n\n# Architecture\n\n![architecture](images/architecture.png)\n\n# Usage\n\n```bash\n# clone the repo\ngit clone https://github.com/XuehaiPan/Soft-Actor-Critic.git\ncd Soft-Actor-Critic\n\n# install dependencies\npip3 install -r requirements.txt\n\n# modify hyperparameters before running\n# train/test FC controller without state encoder\nbash scripts/train_identity.sh\nbash scripts/test_identity.sh\n# override arguments in script file\nbash scripts/train_identity.sh --env \"BipedalWalker-v3\" --n-epochs 5000\n\n# train/test FC controller with FC state encoder\nbash scripts/train_fc.sh\nbash scripts/test_fc.sh\n\n# train/test FC controller with RNN state encoder\nbash scripts/train_rnn.sh\nbash scripts/test_rnn.sh\n\n# train/test FC controller with CNN state encoder\nbash scripts/train_cnn.sh\nbash scripts/test_cnn.sh\n\n# for device arguments\n# train and sample on 'cuda:0'\nbash scripts/train_identity.sh --gpu --n-samplers 4\n# train on 'cuda:0' and sample on ('cuda:1', 'cuda:2', 'cuda:3', 'cuda:4')\nbash scripts/train_identity.sh --gpu 0 1 2 3 4 --n-samplers 4\n# train on 'cuda:0' and sample on ('cuda:1', 'cuda:2', 'cuda:0', 'cuda:1')\nbash scripts/train_identity.sh --gpu 0 1 2 --n-samplers 4\n# train on 'cuda:0' and sample on ('cuda:1', 'cuda:2', 'cuda:1', 'cuda:2')\nbash scripts/train_identity.sh --gpu 0 1 2 1 2 --n-samplers 4\n# train on 'cuda:0' and sample on ('cuda:1', 'cuda:2', 'cpu', 'cpu')\nbash scripts/train_identity.sh --gpu 0 1 2 c c --n-samplers 4\n```\n\nYou can use `python3 main.py --help` for more details:\n```\nusage: main.py [-h] [--mode {train,test}]\n               [--gpu CUDA_DEVICE [CUDA_DEVICE ...]] [--env ENV]\n               [--n-frames N_FRAMES] [--render] [--vision-observation]\n               [--image-size SIZE] [--hidden-dims DIM [DIM ...]]\n               [--activation {ReLU,LeakyReLU}] [--encoder-arch {FC,RNN,CNN}]\n               [--state-dim DIM] [--encoder-activation ACTIVATION]\n               [--encoder-hidden-dims DIM [DIM ...]]\n               [--encoder-hidden-dims-before-rnn DIM [DIM ...]]\n               [--encoder-hidden-dims-rnn DIM [DIM ...]]\n               [--encoder-hidden-dims-after-rnn DIM [DIM ...]]\n               [--skip-connection] [--trainable-hidden]\n               [--step-size STEP_SIZE]\n               [--encoder-hidden-channels CHN [CHN ...]]\n               [--kernel-sizes K [K ...]] [--strides S [S ...]]\n               [--paddings P [P ...]] [--poolings K [K ...]]\n               [--batch-normalization] [--max-episode-steps MAX_EPISODE_STEPS]\n               [--n-epochs N_EPOCHS] [--n-episodes N_EPISODES]\n               [--n-updates N_UPDATES] [--batch-size BATCH_SIZE]\n               [--n-samplers N_SAMPLERS] [--buffer-capacity CAPACITY]\n               [--update-sample-ratio RATIO] [--gamma GAMMA] [--soft-tau TAU]\n               [--normalize-rewards] [--reward-scale SCALE] [--deterministic]\n               [--lr LR] [--critic-lr CRITIC_LR] [--actor-lr ACTOR_LR]\n               [--alpha-lr ALPHA_LR] [--initial-alpha ALPHA]\n               [--adaptive-entropy] [--weight-decay WEIGHT_DECAY]\n               [--clip-gradient] [--random-seed SEED] [--log-episode-video]\n               [--log-dir LOG_DIR] [--checkpoint-dir CHECKPOINT_DIR]\n               [--load-checkpoint]\n\nTrain or test Soft Actor-Critic controller.\n\noptional arguments:\n  -h, --help            show this help message and exit\n  --mode {train,test}   mode (default: train)\n  --gpu CUDA_DEVICE [CUDA_DEVICE ...]\n                        GPU device indexes (int for CUDA device or 'c'/'cpu'\n                        for CPU) (use 'cuda:0' if no following arguments; use\n                        CPU if not present)\n  --env ENV             environment to train on (default: Pendulum-v0)\n  --n-frames N_FRAMES   concatenate original N consecutive observations as a\n                        new observation (default: 1)\n  --render              render the environment\n  --vision-observation  use rendered images as observation\n  --image-size SIZE     image size of vision observation (default: 96)\n  --hidden-dims DIM [DIM ...]\n                        hidden dimensions of FC controller\n  --activation {ReLU,LeakyReLU}\n                        activation function in controller networks (default:\n                        ReLU)\n  --max-episode-steps MAX_EPISODE_STEPS\n                        max steps per episode (default: 10000)\n  --n-epochs N_EPOCHS   number of training epochs (default: 1000)\n  --n-episodes N_EPISODES\n                        number of test episodes (default: 100)\n  --n-updates N_UPDATES\n                        number of learning updates per epoch (default: 256)\n  --batch-size BATCH_SIZE\n                        batch size (default: 256)\n  --n-samplers N_SAMPLERS\n                        number of parallel samplers (default: 4)\n  --buffer-capacity CAPACITY\n                        capacity of replay buffer (default: 1000000)\n  --update-sample-ratio RATIO\n                        speed ratio of training and sampling (sample speed \u003c=\n                        training speed / ratio (ratio should be larger than\n                        1.0)) (default: 2.0)\n  --gamma GAMMA         discount factor for rewards (default: 0.99)\n  --soft-tau TAU        soft update factor for target networks (default: 0.01)\n  --normalize-rewards   normalize rewards for training\n  --reward-scale SCALE  reward scale factor for normalized rewards (default:\n                        1.0)\n  --deterministic       deterministic in evaluation\n  --weight-decay WEIGHT_DECAY\n                        weight decay (default: 0.0)\n  --clip-gradient       clip gradient on optimizer step\n  --random-seed SEED    random seed (default: 0)\n  --log-episode-video   save rendered episode videos to TensorBoard logs\n  --log-dir LOG_DIR     folder to save TensorBoard logs\n  --checkpoint-dir CHECKPOINT_DIR\n                        folder to save checkpoint\n  --load-checkpoint     load latest checkpoint in checkpoint dir\n\nstate encoder:\n  --encoder-arch {FC,RNN,CNN}\n                        architecture of state encoder network (default: FC)\n  --state-dim DIM       target state dimension of encoded state (use\n                        env.observation_space.shape if not present)\n  --encoder-activation ACTIVATION\n                        activation function in state encoder networks (use\n                        activation function in controller if not present)\n\nFC state encoder:\n  --encoder-hidden-dims DIM [DIM ...]\n                        hidden dimensions of FC state encoder\n\nRNN state encoder:\n  --encoder-hidden-dims-before-rnn DIM [DIM ...]\n                        hidden FC dimensions before GRU layers in RNN state\n                        encoder\n  --encoder-hidden-dims-rnn DIM [DIM ...]\n                        GRU hidden dimensions of RNN state encoder\n  --encoder-hidden-dims-after-rnn DIM [DIM ...]\n                        hidden FC dimensions after GRU layers in RNN state\n                        encoder\n  --skip-connection     add skip connection beside GRU layers in RNN state\n                        encoder\n  --trainable-hidden    set initial hidden of GRU layers trainable (use zeros\n                        as initial hidden if not present)\n  --step-size STEP_SIZE\n                        number of continuous steps for update (default: 16)\n\nCNN state encoder:\n  --encoder-hidden-channels CHN [CHN ...]\n                        channels of hidden conv layers in CNN state encoder\n  --kernel-sizes K [K ...]\n                        kernel sizes of conv layers in CNN state encoder\n                        (defaults: 3)\n  --strides S [S ...]   strides of conv layers in CNN state encoder (defaults:\n                        1)\n  --paddings P [P ...]  paddings of conv layers in CNN state encoder\n                        (defaults: K // 2)\n  --poolings K [K ...]  max pooling kernel size after activation function in\n                        CNN state encoder (defaults: 1)\n  --batch-normalization\n                        use batch normalization in CNN state encoder\n\nlearning rate:\n  --lr LR               learning rate (can be override by the following\n                        specific learning rate) (default: 0.0001)\n  --critic-lr CRITIC_LR\n                        learning rate for critic networks (use LR above if not\n                        present)\n  --actor-lr ACTOR_LR   learning rate for actor networks (use LR above if not\n                        present)\n\ntemperature parameter:\n  --alpha-lr ALPHA_LR   learning rate for temperature parameter (use ACTOR_LR\n                        above if not present)\n  --initial-alpha ALPHA\n                        initial value of temperature parameter (default: 1.0)\n  --adaptive-entropy    auto update temperature parameter while training\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuehaipan%2Fsoft-actor-critic","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxuehaipan%2Fsoft-actor-critic","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxuehaipan%2Fsoft-actor-critic/lists"}