{"id":24331152,"url":"https://github.com/yhs0602/minecraft-simulator-benchmark","last_synced_at":"2025-12-24T06:05:56.368Z","repository":{"id":271658399,"uuid":"912450539","full_name":"yhs0602/minecraft-simulator-benchmark","owner":"yhs0602","description":"Performance Comparison Between Base Simulators for Minecraft in Reinforcement Learning","archived":false,"fork":false,"pushed_at":"2025-07-17T15:57:45.000Z","size":73482,"stargazers_count":4,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-17T18:08:38.780Z","etag":null,"topics":["craftground","malmo","minecraft","minerl","performance-comparison","reinforcement-learning","simulation","simulation-optimization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yhs0602.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-05T16:01:03.000Z","updated_at":"2025-07-17T15:53:28.000Z","dependencies_parsed_at":"2025-01-26T15:25:58.200Z","dependency_job_id":"6056c459-3fe8-47a5-a95b-dbe5f5d20151","html_url":"https://github.com/yhs0602/minecraft-simulator-benchmark","commit_stats":null,"previous_names":["yhs0602/minecraft-simulator-benchmark"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/yhs0602/minecraft-simulator-benchmark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yhs0602%2Fminecraft-simulator-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yhs0602%2Fminecraft-simulator-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yhs0602%2Fminecraft-simulator-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yhs0602%2Fminecraft-simulator-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yhs0602","download_url":"https://codeload.github.com/yhs0602/minecraft-simulator-benchmark/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yhs0602%2Fminecraft-simulator-benchmark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":27995883,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-24T02:00:07.193Z","response_time":83,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["craftground","malmo","minecraft","minerl","performance-comparison","reinforcement-learning","simulation","simulation-optimization"],"created_at":"2025-01-18T01:18:09.861Z","updated_at":"2025-12-24T06:05:56.354Z","avatar_url":"https://github.com/yhs0602.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Optimizing Minecraft Simulations: Performance Comparison between Malmö and CraftGround\nThis experiment compares the performance of [Malmö](https://github.com/microsoft/malmo) and [Craftground](https://github.com/yhs0602/CraftGround) in terms of FPS and memory usage. Here we use [MineRL](https://github.com/minerllabs/minerl) which is based on Malmö, because MineRL is most widely used in the research community.\n\n## Caveats\nWhile CraftGround shows significant performance improvements in certain scenarios, there are some caveats to consider:\n\n1. **Feature Gaps**\n   - CraftGround currently does not support some advanced features available in Malmö-based environments, such as multi-agent scenarios, custom mission scripting, or integration with specific competitions like IGLU 2021.\n   - The API interface of CraftGround differs significantly from Malmö, requiring users to adapt their codebase to use it effectively.\n2. **Experimental Nature**: \n    - CraftGround is a relatively new framework and is still under active development. As a result, some features may be less stable or lack documentation compared to mature Malmö-based environments.\n    - Certain optimizations, such as ZeroCopy mode, achieve high performance by bypassing abstractions and safety checks, which might introduce unexpected behavior in complex scenarios.\n    - The ecosystem and community support for CraftGround are still growing, and users may encounter fewer readily available resources or troubleshooting guides compared to MineRL or Malmö.\n  \n\u003e We are exploring the possibility of providing extensions for Malmö-based environments, such as MineRL, to enable the use of CraftGround as a backend with minimal code changes. This could enhance compatibility and performance for users looking for alternative backend solutions.\n\n\n## Configurations\n- Targets: Malmö(MineRL 0.4.4, MineRL 1.0.0) vs Craftground RAW vs Craftground ZeroCopy\n- Steps: 100_000\n- Image Width:\n    - 64 x 64\n    - 114 x 64\n    - 640 x 360\n- Settings\n    - (Simulation)\n    - (Simulation + Render)\n    - (Simulation + PPO)\n    - (Simulation + Render + PPO)\n    - (Simulation + Optimized Render)\n    - (Simulation + Optimized PPO)\n    - (Simulation + Optimized Render + Optimized PPO)\n- Metrics\n    - FPS\n    - Memory Usage\n\n\n## Environment Setup\n- Conda, python=3.11\n- Ubuntu Ubuntu 18.04.6 LTS, cuda driver version 525.105.17, NVIDIA GeForce RTX 3090 Ti, VirtualGL, RAM 188GB, AMD Ryzen Threadripper 3960X 24-Core Processor\n- Apple M1 Pro, single process at once,  single process at once, normal load (not strict setting)\n\n### MineRL 0.4.4 (exp_minerl044, on Ubuntu 18.04.6 LTS)\nThis may help solving issues such as https://github.com/minerllabs/minerl/issues/788.\n- Use this table to get latest gcc version your OS supports: https://askubuntu.com/a/1163021/901082\n- To solve MixinGradle issue, follow the steps as mentioned here: https://github.com/MineDojo/MineDojo/issues/113#issuecomment-1908997704\n```bash\nconda create -n exp_minerl044 python=3.11\nconda activate exp_minerl044\nconda install conda-forge::openjdk=8 \npip install setuptools==65.5.0 pip==21 wheel==0.38.0\npip install gym==0.19.0\npip install --upgrade pip\nsudo add-apt-repository ppa:ubuntu-toolchain-r/test\nsudo apt update\nsudo apt install -y libxml2-dev libxslt1-dev gfortran libopenblas-dev software-properties-common\n# Ensure you have the latest version of gcc:\n# To check the version of gcc, run `gcc --version`\nsudo apt-get install gcc-10 g++-10\nsudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 120\nsudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 120\nOPENSSL_ROOT_DIR=$CONDA_PREFIX pip download --no-binary :all: minerl==0.4.4\ntar -xvf minerl-0.4.4.tar.gz\ncd minerl-0.4.4\n# remove gym line from requirements.txt\npip install -r requirements.txt\n# Edit minerl-0.4.4/minerl/Malmo/Minecraft/build.gradle:L19 based on\n# https://github.com/MineDojo/MineDojo/issues/113#issuecomment-1908997704\n# classpath('MixinGradle-dcfaf61:MixinGradle:dcfaf61'){ // 0.6\n# Add repository maven to the build.gradle\n#         maven { url 'file:file:/absolute-path/to/that/repo's/parent' }\npip install .\npip install wandb tensorboard moviepy stable-baselines3\npip install --upgrade git+https://github.com/DLR-RM/stable-baselines3.git # To ensure correct video rendering\nvglrun python experiments/minerl044.py --image_width 64x64 --load simulation\n```\n\n### MineRL 1.0.0 (exp_minerl100)\nInstalling MineRL 1.0.0 is much easier than 0.4.4.\n```bash\nconda create -n exp_minerl100 python=3.11\nconda activate exp_minerl100\nconda install conda-forge::openjdk=8\npip install git+https://github.com/minerllabs/minerl\npip install wandb tensorboard moviepy stable-baselines3\npip install --upgrade git+https://github.com/DLR-RM/stable-baselines3.git # To ensure correct video rendering\nconda install -c anaconda cudnn # for ppo\n# On cuda devices\npip install jax[cuda]\n# On apple devices\npip install jax-metal\nvglrun python experiments/minerl100_exp.py --image_width 64x64 --load simulation\n```\n\n### MineRL 1.0.0 on MacOS\nWe should apply  https://github.com/MineDojo/MineDojo/pull/56/ to make it work on MacOS.\n```gradle\n    def schemaIndexFile = new File('src/main/resources/schemas.index')\n```\nto\n```gradle\n    def schemaIndexFile = new File(\"$projectDir/src/main/resources/schemas.index\")\n```\nor use the patched version:\n```\npip install git+https://github.com/yhs0602/minerl  \n```\n\n### Craftground\nLatest cmake is required for Craftground to ensure it find the cuda libraries correctly.\n```bash\nconda create -n exp_craftground python=3.11\nconda activate exp_craftground\nconda install conda-forge::openjdk=21 conda-forge::cmake conda-forge::glew conda-forge::libpng conda-forge::libzlib conda-forge::libopengl conda-forge::libflite\npip install craftground\npip install wandb tensorboard moviepy stable-baselines3\npip install --upgrade git+https://github.com/DLR-RM/stable-baselines3.git # To ensure correct video rendering\n# Test SBX, install JAX and Jaxlib\npip install jax jaxlib sbx\n# On apple, to use metal backend\npip install jax-metal\n# On other systems, to use cuda backend\npip install jaxlib\n\npython experiments/craftground_exp.py --mode raw --image_width 64x64 --load simulation\n```\n\n\n## Run options\n- environment: minerl044, minerl100, craftground_raw, craftground_zerocopy\n- image_width: 64x64, 114x64, 640x360\n- load: simulation, render, ppo, render_ppo, optimized_render, optimized_ppo, optimized_render_ppo\n\n## Experiment Results\n![64x64](images/64x64-middle9.png)\n![640x360](images/640x360-middle9.png)\n\n\n## Experiment Results (Frames Per Second, CUDA, 9 seeds)\nBoth used vglrun to run the experiments on headless server.\n\nFor PPO, used stable-baselines3 (SB3) and stable-baselines-jax (SBX). To mitigate the influence of extreme values, we applied outlier removal using the Interquartile Range (IQR) method. Data points below Q1 - 1.5 × IQR or above Q3 + 1.5 × IQR were excluded.\n\n| Configuration           | Malmö      | CraftGround RAW | Speedup |\n| ----------------------- | ---------- | --------------- | ------- |\n| 64x64 Render            | 57.7 (0.7) | 160.7 (17.8)    | 2.79x   |\n| 64x64 PPO Render(SB3)   | 44.7 (2.2) | 115.2 (23.7)    | 2.58x   |\n| 64x64 PPO Render(SBX)   | 54.0 (0.0) | 125.0 (12.5)    | 2.31x   |\n| 640x360 Render          | 55.9 (1.6) | 110.0 (8.8)     | 1.97x   |\n| 640x360 PPO Render(SB3) | 30.0 (0.0) | 52.1 (2.3)      | 1.74x   |\n| 640x360 PPO Render(SBX) | 33.8 (1.3) | 49.7 (4.1)      | 1.47x   |\n\n\n## Experiment Results (Frames Per Second, CUDA, 2 - 9 seeds, not strict setting)\nBoth used vglrun to run the experiments on headless server.\n\nFor PPO, used stable-baselines3. For optimized version which uses tensor directly, we are planning to implement it in the future.\n| Configuration            | Malmö | CraftGround RAW | CraftGround ZeroCopy | Speedup |\n| ------------------------ | ----- | --------------- | -------------------- | ------- |\n| 64x64 Simul              | 57    | 192             | 145                  | 3.36x   |\n| 640x360 Simul            | 56    | 140             | 151                  | 2.7x    |\n| 64x64 Render             | 58.2  | 279.8           | 155                  | 4.80x   |\n| 640x360 Render           | 57.2  | 141.0           | 128                  | 2.47x   |\n| 64x64 PPO                | 45    | 103             | 87                   | 2.29x   |\n| 640x360 PPO              | 33    | 56.5            | 46                   | 1.71x   |\n| 64x64 PPO Render         | 46.5  | 134.5           | 76                   | 2.89x   |\n| 640x360 PPO Render       | 30.2  | 53.5            | 47                   | 1.77x   |\n| 64x64 Render Optim       | *58.5 | *175            | ?                    |         |\n| 640x360 Render Optim     | *55   | *115            | ?                    |         |\n| 64x64 PPO Optim          | *45   | *103            | ?                    |         |\n| 640x360 PPO Optim        | *33   | *56.5           | ?                    |         |\n| 64x64 PPO Render Optim   | *44.5 | *102            | ?                    |         |\n| 640x360 PPO Render Optim | *29.5 | *49             | ?                    |         |\n\n* Since the optimized version is not implemented on Malmö and CraftGround RAW mode, the results are the same as the non-optimized version.\n\n## Experiment Results (Frames Per Second, CUDA, SBX, 2-9 seeds)\n| Configuration            | Malmö | CraftGround RAW | CraftGround ZeroCopy | Speedup |\n| ------------------------ | ----- | --------------- | -------------------- | ------- |\n| 64x64 PPO(CPU)           | 45.6  | 131.6           |                      | 2.89x   |\n| 64x64 PPO(CUDA)          | 55    | 230.3           |                      | 4.19x   |\n| 640x360 PPO(CPU)         | 19    | 23              |                      | 1.21x   |\n| 640x360 PPO(CUDA)        | 34    | 53              |                      | 1.56x   |\n| 64x64 PPO Render(CPU)    | 45.3  | 94              |                      | 2.08x   |\n| 64x64 PPO Render(CUDA)   | 54.5  | 264.8           |                      | 4.86x   |\n| 640x360 PPO Render(CPU)  | 18.5  | 23.5            |                      | 1.27x   |\n| 640x360 PPO Render(CUDA) | 34.8  | 53.2            |                      | 1.53x   |\n\n\n## Experiment Results (Frames Per Second, Apple M1 Pro, 2 seeds average)\n| Configuration      | Malmö | CraftGround RAW | CraftGround ZeroCopy | Speedup |\n| ------------------ | ----- | --------------- | -------------------- | ------- |\n| 64x64 Simul        | -     | 138             | 133.5                |         |\n| 640x360 Simul      |       | 90.5            | 117.5                |         |\n| 64x64 Render       |       | 129             | 144                  |         |\n| 640x360 Render     |       | 111.5           | 134.5                |         |\n| 64x64 PPO          |       | 26              | 26                   |         |\n| 640x360 PPO        |       | 13              | 13.5                 |         |\n| 64x64 PPO Render   |       | 27.5            | 25                   |         |\n| 640x360 PPO Render |       | 13              | 13.5                 |         |\n\n## Experiment Results (Frames Per Second, Apple M1 Pro, SBX, 2 seeds average)\n| Configuration                 | Malmö | CraftGround RAW | CraftGround ZeroCopy | Speedup |\n| ----------------------------- | ----- | --------------- | -------------------- | ------- |\n| 64x64 PPO(SBX, CPU)           |       | 133.5           | -                    |         |\n| 64x64 PPO(SBX, MPS)           |       | 103             | -                    |         |\n| 640x360 PPO (SBX, CPU)        |       | 12              | -                    |         |\n| 640x360 PPO (SBX, MPS)        |       | 44              | -                    |         |\n| 64x64 PPO Render (SBX, CPU)   |       | 147             | -                    |         |\n| 64x64 PPO Render (SBX, MPS)   |       | 103             | -                    |         |\n| 640x360 PPO Render (SBX, CPU) |       | 11.5            | -                    |         |\n| 640x360 PPO Render (SBX, MPS) |       | 42.5            | -                    |         |\n\n\n# TroubleShooting\n## Installing MineRL 0.4.4\n### Error because of MixinGradle\nEdit minerl-0.4.4/minerl/Malmo/Minecraft/build.gradle:L19 based on https://github.com/MineDojo/MineDojo/issues/113#issuecomment-1908997704\n```gradle\nclasspath('MixinGradle-dcfaf61:MixinGradle:dcfaf61'){ // 0.6\n```\nAdd repository maven to the build.gradle\n```\nmaven { url 'file:file:/absolute-path/to/that/repo's/parent' }\n```\n\n### Hangs in installing build dependencies\n```bash\npip install --upgrade pip\n```\n\n### OpenSSL error\nAppend `OPENSSL_ROOT_DIR=$CONDA_PREFIX` to the command if you are using conda. Otherwise, you can set the environment variable after installing openssl-dev.\n```bash\nOPENSSL_ROOT_DIR=$CONDA_PREFIX pip download --no-binary :all: minerl==0.4.4\n```\n\n### NumPy requires GCC \u003e= 8.4, SciPy requires GCC \u003e= 9.1\n- Use this table to get latest gcc version your OS supports: https://askubuntu.com/a/1163021/901082\n\n```bash\nsudo add-apt-repository ppa:ubuntu-toolchain-r/test\nsudo apt update\nsudo apt install -y software-properties-common\nsudo apt-get install gcc-10 g++-10\nsudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 120\nsudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-10 120\n```\n\n### gfortran not found\n```bash\nsudo apt install gfortran\n```\n\n### OpenBLAS not found\n```bash\nsudo apt install libopenblas-dev\n```\n\n\n## Installing MineRL 1.0.0\nEnsure you have the correct version of JDK. MineRL 1.0.0 requires JDK 8.\n```bash\nconda install conda-forge::openjdk=8\n```\nor \n```bash\nsudo apt install openjdk-8-jdk\n```\nor \n```bash\njenv local 1.8\n```\nYou can get the JDK 8 from various sources such as:\n- https://www.azul.com/downloads/?version=java-8-lts\u0026package=jdk#zulu\n\n## Installing Craftground\nEnsure you have the latest version of cmake. Currently the apt repository has cmake 3.10.2, which is not enough for Craftground. To install the latest version of cmake, you should use pip or conda to install it.\n```bash\nconda install cmake\n```\nor \n```bash\npip install --upgrade cmake\n```\n\n\n## Malmö, MineRL, and CraftGround is not using GPU on CUDA devices\nYou should install VirtualGL and run the experiments. Take a look at this MineRL documentation:\n\n- https://minerl.readthedocs.io/en/latest/notes/performance-tips.html#faster-alternative-to-xvfb\n\nAlso you can check this guide:\n\n- https://yhs0602.github.io/CraftGround/headless.html\n\n```bash\necho $WAYLAND_DISPLAY\necho $XDG_SESSION_TYPE\nps aux | grep -E ’weston|sway’\nsudo apt install virtualgl\nwget https://sourceforge.net/projects/virtualgl/files/3.1/virtualgl_3.1_amd64.\ndeb/download\nmv download vgl3.1.deb\nsudo dpkg -i vgl3.1.deb\nsudo vglserver_config\n# During configuration, select the option to install both GLX and EGL and adjust device permissions as required\n# In case you meed the following error, run the following command\n# modprobe: FATAL: Module nvidia_drm is in use. You must execute modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia’ with the display manager stopped in order for the new device permission settings to become effective.\nsudo systemctl stop gdm\nsudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia\n# If you meet modprobe: FATAL: Module nvidia_drm is in use.\nsudo lsof /dev/nvidia*\npkill \u003cpid\u003e\n# Restart the display manager\nsudo modprobe nvidia_uvm nvidia_drm nvidia_modeset nvidia\nsudo systemctl restart gdm\n# Install Xvfb\nsudo apt install xvfb\nXvfb :2 -screen 0 1024x768x24 +extension GLX -ac +extension RENDER \u0026 \nexport DISPLAY=:2\nVGL_DISPLAY=:0 vglrun /opt/VirtualGL/bin/glxspheres64\nsudo nvidia-xconfig --query-gpu-info\nsudo nvidia-xconfig -a --allow-empty-initial-configuration \\\n--use-display-device=None --virtual=1920x1200 \\\n--busid PCI:\u003cBusID\u003e\nsudo systemctl restart gdm\nVGL_DISPLAY=:0 vglrun /opt/VirtualGL/bin/glxspheres64\n# OpenGL Renderer: NVIDIA GeForce RTX 3090/PCIe/SSE2.\n```\n\n\n## Running Craftground\n### FileExistsError\n```\nFileExistsError: Socket file /tmp/minecraftrl_8001.sock already exists. Please choose another port.\n```\nThen\n```bash\nrm /tmp/minecraftrl_8001.sock \n```\n### Zombie Minecraft process\n```bash\njps -l # find the pid of something like DevLaunchInjector.Main\nkill -9 \u003cpid\u003e\n```\n\n# Setup on windows\nNote: you may need to enable long file path due to windows limitation. You can enable it by editing registry as mentioned [here](https://docs.python.org/3/using/windows.html#removing-the-max-path-limitation)\n\n\u003e In the latest versions of Windows, this limitation can be expanded to approximately 32,000 characters. Your administrator will need to activate the “Enable Win32 long paths” group policy, or set LongPathsEnabled to 1 in the registry key HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\FileSystem.\n\nThe following command does.\n```cmd\nreg add \"HKLM\\SYSTEM\\CurrentControlSet\\Control\\FileSystem\" /v LongPathsEnabled /t REG_DWORD /d 1 /f\n```\n\n\n```ps\nconda create -n exp_craftground python=3.11\nconda activate exp_craftground\nconda install conda-forge::openjdk=21 conda-forge::cmake conda-forge::glew conda-forge::libpng conda-forge::libzlib conda-forge::libopengl conda-forge::libflite\npip install craftground\npip install wandb tensorboard moviepy\npip install --upgrade git+https://github.com/DLR-RM/stable-baselines3.git # To ensure correct video rendering\n# Test SBX, install JAX and Jaxlib\npip install jax jaxlib sbx\n# On other systems, to use cuda backend\npip install jaxlib\npip install git+https://github.com/yhs0602/sbx-gpu\n$env:PYTHONPATH = \".\"\npython experiments/craftground_exp.py --mode raw --image_width 64x64 --load simulation\n```\n\n\n# License\nThis repository is basically licensed under the MIT License. However, the following files follows the original license of [stable-baselines3](https://github.com/DLR-RM/stable-baselines3/), which is [MIT License](https://github.com/DLR-RM/stable-baselines3/blob/master/LICENSE):\n- experiments/tensor_optimized/async_vec_video_recorder.py\n- experiments/tensor_optimized/async_video_recorder.py\n- experiments/optim_dummy_vec_env.py\n\nWe copied and modified a bit to make it work with tensor observation directly.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyhs0602%2Fminecraft-simulator-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyhs0602%2Fminecraft-simulator-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyhs0602%2Fminecraft-simulator-benchmark/lists"}