{"id":15136368,"url":"https://github.com/openai/video-pre-training","last_synced_at":"2025-04-08T08:14:58.744Z","repository":{"id":38745676,"uuid":"506356072","full_name":"openai/Video-Pre-Training","owner":"openai","description":"Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos","archived":false,"fork":false,"pushed_at":"2024-06-10T16:13:40.000Z","size":51,"stargazers_count":1436,"open_issues_count":16,"forks_count":146,"subscribers_count":26,"default_branch":"main","last_synced_at":"2025-04-08T08:14:54.215Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/openai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-06-22T18:06:56.000Z","updated_at":"2025-04-07T16:35:07.000Z","dependencies_parsed_at":"2024-01-14T17:05:41.029Z","dependency_job_id":"dbd25abb-9b27-454f-95e4-8bf7dacf00a5","html_url":"https://github.com/openai/Video-Pre-Training","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FVideo-Pre-Training","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FVideo-Pre-Training/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FVideo-Pre-Training/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/openai%2FVideo-Pre-Training/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/openai","download_url":"https://codeload.github.com/openai/Video-Pre-Training/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247801175,"owners_count":20998339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-09-26T06:21:21.033Z","updated_at":"2025-04-08T08:14:58.713Z","avatar_url":"https://github.com/openai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n\n# Video-Pre-Training\nVideo PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos\n\n\n\u003e :page_facing_up: [Read Paper](https://cdn.openai.com/vpt/Paper.pdf) \\\n  :mega: [Blog Post](https://openai.com/blog/vpt) \\\n  :space_invader: [MineRL Environment](https://github.com/minerllabs/minerl) (note version 1.0+ required) \\\n  :checkered_flag: [MineRL BASALT Competition](https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition)\n\n\n# Running agent models\n\nInstall pre-requirements for [MineRL](https://minerl.readthedocs.io/en/latest/tutorials/index.html).\nThen install requirements with:\n\n```\npip install git+https://github.com/minerllabs/minerl\npip install -r requirements.txt\n```\n\n\u003e ⚠️ Note: For reproducibility reasons, the PyTorch version is pinned as `torch==1.9.0`, which is incompatible with Python 3.10 or higher versions. If you are using Python 3.10 or higher, install a [newer version of PyTorch](https://pytorch.org/get-started/locally/) (usually, `pip install torch`). However, note that this *might* subtly change model behaviour (e.g., still act mostly as expected, but not reaching the reported performance).\n\nTo run the code, call\n\n```\npython run_agent.py --model [path to .model file] --weights [path to .weight file]\n```\n\nAfter loading up, you should see a window of the agent playing Minecraft.\n\n\n\n# Agent Model Zoo\nBelow are the model files and weights files for various pre-trained Minecraft models.\nThe 1x, 2x and 3x model files correspond to their respective model weights width.\n\n* [:arrow_down: 1x Model](https://openaipublic.blob.core.windows.net/minecraft-rl/models/foundation-model-1x.model)\n* [:arrow_down: 2x Model](https://openaipublic.blob.core.windows.net/minecraft-rl/models/2x.model)\n* [:arrow_down: 3x Model](https://openaipublic.blob.core.windows.net/minecraft-rl/models/foundation-model-3x.model)\n\n### Demonstration Only - Behavioral Cloning\nThese models are trained on video demonstrations of humans playing Minecraft\nusing behavioral cloning (BC) and are more general than later models which\nuse reinforcement learning (RL) to further optimize the policy.\nFoundational models are trained across all videos in a single training run\nwhile house and early game models refine their respective size foundational\nmodel further using either the housebuilding contractor data or early game video\nsub-set. See the paper linked above for more details.\n\n#### Foundational Model :chart_with_upwards_trend:\n  * [:arrow_down: 1x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/foundation-model-1x.weights)\n  * [:arrow_down: 2x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/foundation-model-2x.weights)\n  * [:arrow_down: 3x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/foundation-model-3x.weights)\n\n#### Fine-Tuned from House :chart_with_upwards_trend:\n  * [:arrow_down: 3x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/bc-house-3x.weights)\n\n#### Fine-Tuned from Early Game :chart_with_upwards_trend:\n  * [:arrow_down: 2x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/bc-early-game-2x.weights)\n  * [:arrow_down: 3x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/bc-early-game-3x.weights)\n\n### Models With Environment Interactions\nThese models further refine the above demonstration based models with a reward\nfunction targeted at obtaining diamond pickaxes. While less general then the behavioral\ncloning models, these models have the benefit of interacting with the environment\nusing a reward function and excel at progressing through the tech tree quickly.\nSee the paper for more information\non how they were trained and the exact reward schedule.\n\n#### RL from Foundation :chart_with_upwards_trend:\n  * [:arrow_down: 2x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/rl-from-foundation-2x.weights)\n\n#### RL from House :chart_with_upwards_trend:\n  * [:arrow_down: 2x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/rl-from-house-2x.weights)\n\n#### RL from Early Game :chart_with_upwards_trend:\n  * [:arrow_down: 2x Width Weights](https://openaipublic.blob.core.windows.net/minecraft-rl/models/rl-from-early-game-2x.weights)\n\n# Running Inverse Dynamics Model (IDM)\n\nIDM aims to predict what actions player is taking in a video recording.\n\nSetup:\n* Install requirements: `pip install -r requirements.txt`\n* Download the IDM model [.model :arrow_down:](https://openaipublic.blob.core.windows.net/minecraft-rl/idm/4x_idm.model) and [.weight :arrow_down:](https://openaipublic.blob.core.windows.net/minecraft-rl/idm/4x_idm.weights) files\n* For demonstration purposes, you can use the contractor recordings shared below to. For this demo we use\n  [this .mp4](https://openaipublic.blob.core.windows.net/minecraft-rl/data/10.0/cheeky-cornflower-setter-02e496ce4abb-20220421-092639.mp4)\n  and [this associated actions file (.jsonl)](https://openaipublic.blob.core.windows.net/minecraft-rl/data/10.0/cheeky-cornflower-setter-02e496ce4abb-20220421-092639.jsonl).\n\nTo run the model with above files placed in the root directory of this code:\n```\npython run_inverse_dynamics_model.py --weights 4x_idm.weights --model 4x_idm.model --video-path cheeky-cornflower-setter-02e496ce4abb-20220421-092639.mp4 --jsonl-path cheeky-cornflower-setter-02e496ce4abb-20220421-092639.jsonl\n```\n\nA window should pop up which shows the video frame-by-frame, showing the predicted and true (recorded) actions side-by-side on the left.\n\nNote that `run_inverse_dynamics_model.py` is designed to be a demo of the IDM, not code to put it into practice.\n\n# Using behavioural cloning to fine-tune the models\n\n**Disclaimer:** This code is a rough demonstration only and not an exact recreation of what original VPT paper did (but it contains some preprocessing steps you want to be aware of)! As such, do not expect replicate the original experiments with this code. This code has been designed to be run-able on consumer hardware (e.g., 8GB of VRAM).\n\nSetup:\n* Install requirements: `pip install -r requirements.txt`\n* Download `.weights` and `.model` file for model you want to fine-tune.\n* Download contractor data (below) and place the `.mp4` and `.jsonl` files to the same directory (e.g., `data`). With default settings, you need at least 12 recordings.\n\nIf you downloaded the \"1x Width\" models and placed some data under `data` directory, you can perform finetuning with\n\n```\npython behavioural_cloning.py --data-dir data --in-model foundation-model-1x.model --in-weights foundation-model-1x.weights --out-weights finetuned-1x.weights\n```\n\nYou can then use `finetuned-1x.weights` when running the agent. You can change the training settings at the top of `behavioural_cloning.py`.\n\nMajor limitations:\n- Only trains single step at the time, i.e., errors are not propagated through timesteps.\n- Computes gradients one sample at a time to keep memory use low, but also slows down the code.\n\n# Contractor Demonstrations\n\n### Versions\nOver the course of the project we requested various demonstrations from contractors\nwhich we release as index files below. In general, major recorder versions change for a new\nprompt or recording feature while bug-fixes were represented as minor version changes.\nHowever, some\nrecorder versions we asked contractors to change their username when recording particular\nmodalities. Also, as contractors internally ask questions, clarification from one contractor may\nresult in a behavioral change in the other contractor. It is intractable to share every contractor's\nview for each version, but we've shared the prompts and major clarifications for each recorder\nversion where the task changed significantly.\n\n  \u003cdetails\u003e\n  \u003csummary\u003eInitial Prompt\u003c/summary\u003e\n\n  We are collecting data for training AI models in Minecraft. You'll need to install java, download the modified version of minecraft (that collects and uploads your play data), and play minecraft survival mode! Paid per hour of gameplay. Prior experience in minecraft not. necessary. We do not collect any data that is unrelated to minecraft from your computer.\n\n  \u003c/details\u003e\n\nThe following is a list of the available versions:\n\n* **6.x** Core recorder features subject to change [:arrow_down: index file](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/all_6xx_Jun_29.json)\n  * 6.9 First feature complete recorder version\n  * 6.10 Fixes mouse scaling on Mac when gui is open\n  * 6.11 Tracks the hotbar slot\n  * 6.13 Sprinting, swap-hands, ... (see commits below)\n    \u003cdetails\u003e\n    \u003csummary\u003eCommits\u003c/summary\u003e\n\n    * improve replays that are cut in the middle of gui; working on riding boats / replays cut in the middle of a run\n    * improve replays by adding dwheel action etc, also, loosen up replay tolerances\n    * opencv version bump\n    * add swap hands, and recording of the step timestamp\n    * implement replaying from running and sprinting and tests\n    * do not record sprinting (can use stats for that)\n    * check for mouse button number, ignore \u003e2\n    * handle the errors when mouse / keyboard are recorded as null\n\n    \u003c/details\u003e\n* **7.x** Prompt changes [:arrow_down: index file](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/all_7xx_Apr_6.json)\n  * 7.6 Bump version for internal tracking\n    \u003cdetails\u003e\n    \u003csummary\u003eAdditional ask to contractors\u003c/summary\u003e\n\n    Right now, early game data is especially valuable to us. As such, we request that at least half of the data you upload is from the first 30 minutes of the game. This means that, for every hour of gameplay you spend in an older world, we ask you to play two sessions in which you create a new world and play for 30 minutes. You can play for longer in these worlds, but only the first 30 minutes counts as early game data.\n\n    \u003c/details\u003e\n* **8.x** :clipboard: House Building from Scratch Task [:arrow_down: index](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/all_8xx_Jun_29.json)\n  \u003cdetails\u003e\n  \u003csummary\u003eChanges and Prompt\u003c/summary\u003e\n\n  Hi all! Thank you for your hard work so far.\n\n  This week we would like to have you all collect data on a specific task.\n\n  This comes with a new recorder version 8.0 which you will need to update your recording script to download.\n\n  This week we would like you to use a new world each time you play, so loading existing worlds is disabled.\n\n  The new task is as follows:\n\n  Starting in a new world, build a simple house in 10-15 minutes. This corresponds to one day and a bit of the night. Please use primarily wood, dirt, and sand, as well as crafted wood items such as doors, fences, ect. in constructing your house. Avoid using difficult items such as stone. Aside from those constraints, you may decorate the structure you build as you wish. It does not need to have any specific furniture. For example, it is OK if there is no bed in your house. If you have not finished the house by the sunrise (20 minutes) please exit and continue to another demonstration. Please continue to narrate what you are doing while completing this task.\n\n  Since you will be unable to resume building after exiting Minecraft or going back to the main menu, you must finish these demonstrations in one session. Pausing via the menu is still supported. If you want to view your creations later, they will be saved locally so you can look at them in your own time. We may use these save files in a future task so if you have space, please leave the save files titled “build-house-15-min-“.\n\n  For this week try to avoid all cobblestone / stone / granite\n\n  For this week we just want simple houses without sleeping. If 10 minutes is too short, let us know and we can think of how to adjust!\n\n  Stone tools are ok but I think you may run-out of time\n\n  Changes:\n    * Timer ends episode after 10 realtime minutes\n    * Worlds are named: `\"build-house-15-min-\" + Math.abs(random.nextInt());`\n\n  \u003c/details\u003e\n\n  * Note this version introduces 10-minute timer that ends the episode. It\n  cut experiments short occasionally and was fixed in 9.1\n  * 8.0 Simple House\n  * 8.2 Update upload script\n* **9.x** :clipboard: House Building from Random Starting Materials Task [:arrow_down: index](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/all_9xx_Jun_29.json)\n    \u003cdetails\u003e\n    \u003csummary\u003eChanges and Prompt\u003c/summary\u003e\n\n    You now will have 10 minutes to use the provided resources to build your house / home / or structure. In this version, the experiment will time out after 10 minutes if you are not complete so don't be alarmed if that happens, it is intentional.\n\n    No need to use up all the resources! It's ok to collect a few things but spend the majority of the time placing blocks (the act of placing seems to be harder to learn)\n\n    Changes:\n    * Worlds are named: `\"design-house-10-min-\" + Math.abs(random.nextInt());`\n    * Starting inventory given by code below\n    \u003c/details\u003e\n\n    \u003cdetails\u003e\n    \u003csummary\u003eRandom Starting Inventory Code\u003c/summary\u003e\n\n  ```java\n        Random random = new Random();\n        List\u003cItemStack\u003e hotbar = new ArrayList\u003c\u003e();\n        List\u003cItemStack\u003e inventory = new ArrayList\u003c\u003e();\n\n        // Ensure we give the player the basic tools in their hot bar\n        hotbar.add(new ItemStack(Items.STONE_AXE));\n        hotbar.add(new ItemStack(Items.STONE_PICKAXE));\n        hotbar.add(new ItemStack(Items.STONE_SHOVEL));\n        hotbar.add(new ItemStack(Items.CRAFTING_TABLE));\n\n        // Add some random items to the player hotbar as well\n        addToList(hotbar, inventory, Items.TORCH, random.nextInt(16) * 2 + 2);\n\n        // Next add main building blocks\n        if (random.nextFloat() \u003c 0.7) {\n           addToList(hotbar, inventory, Items.OAK_FENCE_GATE, random.nextInt(5));\n           addToList(hotbar, inventory, Items.OAK_FENCE, random.nextInt(5) * 64);\n           addToList(hotbar, inventory, Items.OAK_DOOR, random.nextInt(5));\n           addToList(hotbar, inventory, Items.OAK_TRAPDOOR, random.nextInt(2) * 2);\n           addToList(hotbar, inventory, Items.OAK_PLANKS, random.nextInt(3) * 64 + 128);\n           addToList(hotbar, inventory, Items.OAK_SLAB, random.nextInt(3) * 64);\n           addToList(hotbar, inventory, Items.OAK_STAIRS, random.nextInt(3) * 64);\n           addToList(hotbar, inventory, Items.OAK_LOG, random.nextInt(2) * 32);\n           addToList(hotbar, inventory, Items.OAK_PRESSURE_PLATE, random.nextInt(5));\n        } else {\n           addToList(hotbar, inventory, Items.BIRCH_FENCE_GATE, random.nextInt(5));\n           addToList(hotbar, inventory, Items.BIRCH_FENCE, random.nextInt(5) * 64);\n           addToList(hotbar, inventory, Items.BIRCH_DOOR, random.nextInt(5));\n           addToList(hotbar, inventory, Items.BIRCH_TRAPDOOR, random.nextInt(2) * 2);\n           addToList(hotbar, inventory, Items.BIRCH_PLANKS, random.nextInt(3) * 64 + 128);\n           addToList(hotbar, inventory, Items.BIRCH_SLAB, random.nextInt(3) * 64);\n           addToList(hotbar, inventory, Items.BIRCH_STAIRS, random.nextInt(3) * 64);\n           addToList(hotbar, inventory, Items.BIRCH_LOG, random.nextInt(2) * 32);\n           addToList(hotbar, inventory, Items.BIRCH_PRESSURE_PLATE, random.nextInt(5));\n        }\n\n        // Now add some random decoration items to the player inventory\n        addToList(hotbar, inventory, Items.CHEST, random.nextInt(3));\n        addToList(hotbar, inventory, Items.FURNACE, random.nextInt(2) + 1);\n        addToList(hotbar, inventory, Items.GLASS_PANE,  random.nextInt(5) * 4);\n        addToList(hotbar, inventory, Items.WHITE_BED, (int) (random.nextFloat() + 0.2)); // Bed 20% of the time\n        addToList(hotbar, inventory, Items.PAINTING, (int) (random.nextFloat() + 0.1)); // Painting 10% of the time\n        addToList(hotbar, inventory, Items.FLOWER_POT, (int) (random.nextFloat() + 0.1) * 4); // 4 Flower pots 10% of the time\n        addToList(hotbar, inventory, Items.OXEYE_DAISY, (int) (random.nextFloat() + 0.1) * 4); // 4 Oxeye daisies 10% of the time\n        addToList(hotbar, inventory, Items.POPPY, (int) (random.nextFloat() + 0.1) * 4); // 4 Poppies 10% of the time\n        addToList(hotbar, inventory, Items.SUNFLOWER, (int) (random.nextFloat() + 0.1) * 4); // 4 Sunflowers 10% of the time\n\n        // Shuffle the hotbar slots and inventory slots\n        Collections.shuffle(hotbar);\n        Collections.shuffle(inventory);\n\n        // Give the player the items\n        this.mc.getIntegratedServer().getPlayerList().getPlayers().forEach(p -\u003e {\n           if (p.getUniqueID().equals(this.getUniqueID())) {\n               hotbar.forEach(p.inventory::addItemStackToInventory);\n               inventory.forEach(p.inventory::addItemStackToInventory);\n           }\n        });\n  ```\n\n    \u003c/details\u003e\n\n     * 9.0 First version\n     * 9.1 Fixed timer bug\n* **10.0** :clipboard: Obtain Diamond Pickaxe Task [:arrow_down: index](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/all_10xx_Jun_29.json)\n  \u003cdetails\u003e\n  \u003csummary\u003eChanges and Prompt\u003c/summary\u003e\n  Prompt:\n\n  For this new task we have given you 20 minutes to craft a diamond pickaxe. We ask that you do not try to search for villages or other ways of getting diamonds, but if you are spawned in view of one, or happen to fall into a cave structure feel free to explore it for diamonds.\n  If 20 min is not enough that is OK. It will happen on some seeds because of bad luck. Please do not use glitches to find the diamonds.\n\n  Changes:\n  * change to 20 minute time limit\n  * _don't count gui time as part of the time limit_\n  * World are named `\"collect-diamond-pickaxe-15min-\" + Math.abs(random.nextInt());`\n\n  \u003c/details\u003e\n\n\nSometimes we asked the contractors to signify other tasks besides changing the version. This\nprimarily occurred in versions 6 and 7 as 8, 9 and 10 are all task specific.\n\n\u003cdetails\u003e\n\u003csummary\u003ePrompt to contractors (click to show)\u003c/summary\u003e\nAnother request about additional time - please use some of it to chop trees. Specifically, please start the recorder by adding --username treechop argument to the script (i.e. use play --username treechop on windows, ./play.sh --username treechop on osx/linux), and spend some time chopping trees! Getting wooden or stone tools is ok, but please spend the majority of the with username treechop specifically chopping. I did it myself for about 15 minutes, and it does get boring pretty quickly, so I don't expect you to do it all the time, but please do at least a little bit of chopping. Feel free to play normally the rest of the time (but please restart without --username treechop argument when you are not chopping)\nHowever, it is preferable that you start a new world though, and use only the tools that are easily obtainable in that world. I'll see what I can do about getting player an iron axe - that sounds reasonable, and should not be hard, but will require a code update.\n\u003c/details\u003e\n\n### Environment\nWe restrict the contractors to playing Minecraft in windowed mode at 720p which we downsample at 20hz to 360p\nto minimize space. We also disabled the options screen to prevent the contractor from\nchanging things such as brightness, or rendering options. We ask contractors not to press keys\nsuch as f3 which shows a debug overlay, however some contractors may still do this.\n\n\n### Data format\n\nDemonstrations are broken up into up to 5 minute segments consisting of a series of\ncompressed screen observations, actions, environment statistics, and a checkpoint\nsave file from the start of the segment. Each relative path in the index will\nhave all the files for that given segment, however if a file was dropped while\nuploading, the corresponding relative path is not included in the index therefore\nthere may be missing chunks from otherwise continuous demonstrations.\n\nIndex files are provided for each version as a json file:\n```json\n{\n  \"basedir\": \"https://openaipublic.blob.core.windows.net/data/\",\n  \"relpaths\": [\n    \"8.0/cheeky-cornflower-setter-74ae6c2eae2e-20220315-122354\",\n    ...\n  ]\n}\n```\nRelative paths follow the following format:\n* `\u003crecorder-version\u003e/\u003ccontractor-alias\u003e-\u003csession-id\u003e-\u003cdate\u003e-\u003ctime\u003e`\n\n\u003e Note that due to network errors, some segments may be missing from otherwise\ncontinuous demonstrations.\n\nYour data loader can then find following files:\n* Video observation: `\u003cbasedir\u003e/\u003crelpath\u003e.mp4`\n* Action file: `\u003cbasedir\u003e/\u003crelpath\u003e.jsonl`\n* Options file: `\u003cbasedir\u003e/\u003crelpath\u003e-options.json`\n* Checkpoint save file: `\u003cbasedir\u003e/\u003crelpath\u003e.zip`\n\nThe action file is **not**  a valid json object: each line in\naction file is an individual action dictionary.\n\nFor v7.x, the actions are in form\n```json\n{\n  \"mouse\": {\n    \"x\": 274.0,\n    \"y\": 338.0,\n    \"dx\": 0.0,\n    \"dy\": 0.0,\n    \"scaledX\": -366.0,\n    \"scaledY\": -22.0,\n    \"dwheel\": 0.0,\n    \"buttons\": [],\n    \"newButtons\": []\n  },\n  \"keyboard\": {\n    \"keys\": [\n      \"key.keyboard.a\",\n      \"key.keyboard.s\"\n    ],\n    \"newKeys\": [],\n    \"chars\": \"\"\n  },\n  \"isGuiOpen\": false,\n  \"isGuiInventory\": false,\n  \"hotbar\": 4,\n  \"yaw\": -112.35006,\n  \"pitch\": 8.099996,\n  \"xpos\": 841.364694513396,\n  \"ypos\": 63.0,\n  \"zpos\": 24.956354839537802,\n  \"tick\": 0,\n  \"milli\": 1649575088006,\n  \"inventory\": [\n    {\n      \"type\": \"oak_door\",\n      \"quantity\": 3\n    },\n    {\n      \"type\": \"oak_planks\",\n      \"quantity\": 59\n    },\n    {\n      \"type\": \"stone_pickaxe\",\n      \"quantity\": 1\n    },\n    {\n      \"type\": \"oak_planks\",\n      \"quantity\": 64\n    }\n  ],\n  \"serverTick\": 6001,\n  \"serverTickDurationMs\": 36.3466,\n  \"stats\": {\n    \"minecraft.custom:minecraft.jump\": 4,\n    \"minecraft.custom:minecraft.time_since_rest\": 5999,\n    \"minecraft.custom:minecraft.play_one_minute\": 5999,\n    \"minecraft.custom:minecraft.time_since_death\": 5999,\n    \"minecraft.custom:minecraft.walk_one_cm\": 7554,\n    \"minecraft.use_item:minecraft.oak_planks\": 5,\n    \"minecraft.custom:minecraft.fall_one_cm\": 269,\n    \"minecraft.use_item:minecraft.glass_pane\": 3\n  }\n}\n```\n\n# BASALT 2022 dataset\n\nWe also collected a dataset of demonstrations for the [MineRL BASALT 2022](https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition) competition, with around 150GB of data per task.\n\n**Note**: To avoid confusion with the competition rules, the action files (.jsonl) have been stripped of information that is not allowed in the competition. We will upload unmodified dataset after the competition ends.\n\n* **FindCave** [:arrow_down: index file](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/find-cave-Jul-28.json)\n  * \u003cdetails\u003e\n    \u003csummary\u003ePrompt to contractors (click to show)\u003c/summary\u003e\n\n    ```\n    Look around for a cave. When you are inside one, quit the game by opening main menu and pressing \"Save and Quit To Title\".\n    You are not allowed to dig down from the surface to find a cave.\n\n    Timelimit: 3 minutes.\n    Example recordings: https://www.youtube.com/watch?v=TclP_ozH-eg\n    ```\n    \u003c/details\u003e\n* **MakeWaterfall** [:arrow_down: index file](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/waterfall-Jul-28.json)\n  * \u003cdetails\u003e\n    \u003csummary\u003ePrompt to contractors (click to show)\u003c/summary\u003e\n\n    ```\n    After spawning in a mountainous area with a water bucket and various tools, build a beautiful waterfall and then reposition yourself to “take a scenic picture” of the same waterfall, and then quit the game by opening the menu and selecting \"Save and Quit to Title\"\n\n    Timelimit: 5 minutes.\n    Example recordings: https://youtu.be/NONcbS85NLA\n    ```\n    \u003c/details\u003e\n* **MakeVillageAnimalPen** [:arrow_down: index file](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/pen-animals-Jul-28.json)\n  * \u003cdetails\u003e\n    \u003csummary\u003ePrompt to contractors (click to show)\u003c/summary\u003e\n\n    ```\n    After spawning in a village, build an animal pen next to one of the houses in a village. Use your fence posts to build one animal pen that contains at least two of the same animal. (You are only allowed to pen chickens, cows, pigs, sheep or rabbits.) There should be at least one gate that allows players to enter and exit easily. The animal pen should not contain more than one type of animal. (You may kill any extra types of animals that accidentally got into the pen.) Don’t harm the village.\n    After you are done, quit the game by opening the menu and pressing \"Save and Quit to Title\".\n\n    You may need to terraform the area around a house to build a pen. When we say not to harm the village, examples include taking animals from existing pens, damaging existing houses or farms, and attacking villagers. Animal pens must have a single type of animal: pigs, cows, sheep, chicken or rabbits.\n\n    The food items can be used to lure in the animals: if you hold seeds in your hand, this attracts nearby chickens to you, for example.\n\n    Timelimit: 5 minutes.\n    Example recordings: https://youtu.be/SLO7sep7BO8\n    ```\n    \u003c/details\u003e\n* **BuildVillageHouse** [:arrow_down: index file](https://openaipublic.blob.core.windows.net/minecraft-rl/snapshots/build-house-Jul-28.json)\n  * \u003cdetails\u003e\n    \u003csummary\u003ePrompt to contractors (click to show)\u003c/summary\u003e\n\n    ```\n    Taking advantage of the items in your inventory, build a new house in the style of the village (random biome), in an appropriate location (e.g. next to the path through the village), without harming the village in the process.\n    Then give a brief tour of the house (i.e. spin around slowly such that all of the walls and the roof are visible).\n\n    * You start with a stone pickaxe and a stone axe, and various building blocks. It’s okay to break items that you misplaced (e.g. use the stone pickaxe to break cobblestone blocks).\n    * You are allowed to craft new blocks.\n\n    Please spend less than ten minutes constructing your house.\n\n    You don’t need to copy another house in the village exactly (in fact, we’re more interested in having slight deviations, while keeping the same \"style\"). You may need to terraform the area to make space for a new house.\n    When we say not to harm the village, examples include taking animals from existing pens, damaging existing houses or farms, and attacking villagers.\n\n    After you are done, quit the game by opening the menu and pressing \"Save and Quit to Title\".\n\n    Timelimit: 12 minutes.\n    Example recordings: https://youtu.be/WeVqQN96V_g\n    ```\n    \u003c/details\u003e\n\n\n\n# Contribution\nThis was a large effort by a dedicated team at OpenAI:\n[Bowen Baker](https://github.com/bowenbaker),\n[Ilge Akkaya](https://github.com/ilge),\n[Peter Zhokhov](https://github.com/pzhokhov),\n[Joost Huizinga](https://github.com/JoostHuizinga),\n[Jie Tang](https://github.com/jietang),\n[Adrien Ecoffet](https://github.com/AdrienLE),\n[Brandon Houghton](https://github.com/brandonhoughton),\n[Raul Sampedro](https://github.com/samraul),\nJeff Clune\nThe code here represents a minimal version of our model code which was\nprepared by [Anssi Kanervisto](https://github.com/miffyli) and others so that these models could be used as\npart of the MineRL BASALT competition.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fvideo-pre-training","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fopenai%2Fvideo-pre-training","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fopenai%2Fvideo-pre-training/lists"}