{"id":28664700,"url":"https://github.com/qxcv/magical","last_synced_at":"2025-06-13T13:09:51.504Z","repository":{"id":54935044,"uuid":"221098440","full_name":"qxcv/magical","owner":"qxcv","description":"The MAGICAL benchmark suite for robust imitation learning (NeurIPS 2020)","archived":false,"fork":false,"pushed_at":"2023-12-05T22:45:38.000Z","size":54794,"stargazers_count":77,"open_issues_count":1,"forks_count":11,"subscribers_count":6,"default_branch":"pyglet1.5","last_synced_at":"2025-05-27T17:57:02.143Z","etag":null,"topics":["imitation-learning","preference-learning","reinforcement-learning","reinforcement-learning-environments"],"latest_commit_sha":null,"homepage":"https://arxiv.org/pdf/2011.00401.pdf","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"isc","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/qxcv.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2019-11-12T00:52:11.000Z","updated_at":"2025-02-15T13:24:44.000Z","dependencies_parsed_at":"2022-08-14T07:00:42.046Z","dependency_job_id":"6cb4545d-2d27-4fb7-9131-997c505fae0d","html_url":"https://github.com/qxcv/magical","commit_stats":{"total_commits":138,"total_committers":2,"mean_commits":69.0,"dds":0.07246376811594202,"last_synced_commit":"d4708872d098bbd687c788719bf95ba40eb8d349"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/qxcv/magical","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qxcv%2Fmagical","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qxcv%2Fmagical/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qxcv%2Fmagical/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qxcv%2Fmagical/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/qxcv","download_url":"https://codeload.github.com/qxcv/magical/tar.gz/refs/heads/pyglet1.5","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/qxcv%2Fmagical/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259650956,"owners_count":22890385,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["imitation-learning","preference-learning","reinforcement-learning","reinforcement-learning-environments"],"created_at":"2025-06-13T13:09:46.472Z","updated_at":"2025-06-13T13:09:51.493Z","avatar_url":"https://github.com/qxcv.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Multitask Assessment of Generalisation in Imitative Control Algorithms (MAGICAL)\n\n[![PyPI package](https://badge.fury.io/py/magical-il.svg)](https://badge.fury.io/py/magical-il)\n[![CircleCI pipelines](https://circleci.com/gh/qxcv/magical.svg?style=svg)](https://app.circleci.com/pipelines/github/qxcv/magical)\n[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/qxcv/magical/blob/pyglet1.5/demo-notebook.ipynb)\n[![arXiv](https://img.shields.io/badge/arXiv-2011.00401-b31b1b.svg)](https://arxiv.org/abs/2011.00401)\n\n## A Benchmark Suite for Robust imitation Learning\n\nMAGICAL is a benchmark suite to evaluate the generalisation capabilities of\nimitation learning algorithms. Rather than using the same setting for training\nand testing, MAGICAL provides one set of \"training\" environments where\ndemonstrations are observed, and another, distinct set of \"testing\" environments\nwhich each vary in different ways. MAGICAL is a multitask suite, and we refer to\nthe training environment for a given task as the \"demo variant\", and the testing\nenvironments for task as \"test variants\". This structure makes it possible to\nevaluate how well an imitation learning (or reward learning) algorithm is able\nto generalise the intent behind a set of demonstrations to a substantially\ndifferent setting.\n\n![demonstration variant of one environment presented alongside three labelled test variants](images/lead.png)\n\nThe different tasks that comprise the MAGICAL suite each require similar skills,\nsuch as manipulation of 2D blocks, perception of shape and colour, relational\nreasoning, and so on. This makes it possible, in principle, to use multi-task\nand meta-IL algorithms that allow for transfer of skills between tasks, and\n(hopefully) extrapolation of demonstrator intent across the different variants\nfor each task.\n\nMAGICAL was originally published [at NeurIPS 2020](https://proceedings.neurips.cc//paper_files/paper/2020/hash/d464b5ac99e74462f321c06ccacc4bff-Abstract.html):\n\n```bibtex\n@inproceedings{toyer2020magical,\n  author    = {Sam Toyer and Rohin Shah and Andrew Critch and Stuart Russell},\n  title     = {The {MAGICAL} Benchmark for Robust Imitation},\n  booktitle = {Advances in Neural Information Processing Systems},\n  year      = {2020}\n}\n```\n\nWe'd appreciate a citation if you benchmark on MAGICAL or otherwise find it\nuseful for your work!\n\n## Installing and using MAGICAL\n\nYou can install MAGICAL using `pip`:\n\n```sh\npip install magical-il\n```\n\nIf you have an X server and input device, you can try controlling the robot in\none of the environments:\n\n```sh\npython -m magical --env-name FindDupe-Demo-v0\n```\n\nUse the arrow keys to move, space bar to close the grippers, and `R` key to\nreset the environment.\n\nAt an API level, MAGICAL tasks and variants are just Gym environments. Once\nyou've installed MAGICAL, you can use the Gym environments as follows:\n\n```python\nimport gym\nimport magical\n\n# magical.register_envs() must be called before making any Gym envs\nmagical.register_envs()\n\n# creating a demo variant for one task\nenv = gym.make('FindDupe-Demo-v0')\nenv.reset()\nenv.render(mode='human')\nenv.close()\n\n# We can also make the test variant of the same environment, or add a\n# preprocessor to the environment. In this case, we are creating a\n# TestShape variant of the original environment, and applying the\n# LoRes4E preprocessor to observations. LoRes4E stacks four\n# egocentric frames together and downsamples them to 96x96.\nenv = gym.make('FindDupe-TestShape-LoRes4E-v0')\ninit_obs = env.reset()\nprint('Observation type:', type(obs))  # np.ndarray\nprint('Observation shape:', obs.shape)  # (96, 96, 3)\nenv.close()\n```\n\nIn general, Gym environment names for MAGICAL take the form\n`\u003ctask-name\u003e-\u003cvariant\u003e[-\u003cpreprocessor]-v0`, where the final preprocessor name is\noptional. For instance,`FindDupe-Demo-v0`, `MoveToCorner-Demo-LoResStack-v0` and\n`ClusterColour-TestAll-v0` are all available Gym environments. Keep reading to\nsee a list of all available tasks and variants, as well as all the builtin\nobservation preprocessors that ship with MAGICAL.\n\nNote that the reference demonstration data for MAGICAL is *not* included in the\nPyPI package. Rather, it is packaged as [another Github\nrepository](https://github.com/qxcv/magical-data/). See the \"Using pre-recorded\ndemonstrations\" section below for instructions on using this data.\n\n## Tasks and variants\n\nHere are the available tasks, along with a picture of the initial state of the\ndemo variant:\n\n| | Task |\n| :---: | --- |\n| ![movetocorner task](images/static-movetocorner-demo-v0.png) | **MoveToCorner:** Robot must push a block from the bottom right corner to the top left. Does not support Layout and CountPlus variants. |\n| ![movetoregion task](images/static-movetoregion-demo-v0.png) | **MoveToRegion:** Move into the goal region and stay there. Does not support Shape and CountPlus variants. |\n| ![matchregions task](images/static-matchregions-demo-v0.png) | **MatchRegions:** Fill the goal region with all blocks of the same colour as the goal region. |\n| ![makeline task](images/static-makeline-demo-v0.png) | **MakeLine:** Arrange all blocks into a line. |\n| ![finddupe task](images/static-finddupe-demo-v0.png) | **FindDupe:** The goal region starts with one block inside it. The robot must find a duplicate of the block, and push it inside the goal region too. |\n| ![fixcolour task](images/static-fixcolour-demo-v0.png) | **FixColour:** There are several goal regions, each containing one block. One goal region has a block which doesn't match the regions colour. That block should be pushed out of its region, while keeping all other colours the same. |\n| ![clustercolour task](images/static-clustercolour-demo-v0.png) | **ClusterColour:** Sort the blocks into spatially separated clusters of uniform colour. |\n| ![clustershape task](images/static-clustershape-demo-v0.png) | **ClusterShape:** Sort the blocks into spatially separated clusters of uniform shape. |\n\nEach task can be instantiated in one of several _variants_, each of which\nchanges one or more aspects of the environment to evaluate combinatorial\ngeneralisation:\n\n- **Demo:** the default variant in which all demonstrations are observed.\n  Initial states of the demo variants for each task are pictured above.\n- **Jitter:** the rotations and orientations of all objects, and the size of\n  goal regions, is jittered by up to 5% of the maximum range.\n- **Layout:** positions and rotations of all objects are completely randomised.\n- **Colour:** colours of blocks and goal reginos are randomised, subject to\n  task-specific constraints (e.g. that there should always be at least one block\n  of each colour in ClusterColour).\n- **Shape:** shapes of pushable blocks are randomised, again subject to\n  task-specific constraints.\n- **CountPlus:** the layout, colour, shape, and _number_ of objects is\n  randomised.\n- **Dynamics:** mass and friction of objects is randomsied.\n- **All:** all applicable randomisations are applied.\n\nExcept where listsed in the task table above, all tasks support all variants.\n\n## Preprocessors\n\nThe default observation type is a `dict` containing two keys, `ego` and `allo`,\ncorresponding to egocentric and allocentric views of the environment,\nrespectively. The values of this dictionary are 384×384 RGB images with the the\ncorresponding view. If you don't want to work with a dict of views for single\ntime steps, then you can also get observation spaces by appending one of the\nfollowing preprocessor names to the env name:\n\n- `-LoResStack`: rather than showing only one image of the environment, values\n  of the observation dict will contain the four most recent frames, concatenated\n  along the channels axis. Additionally, observations will be resized to 96×96\n  pixels.\n- `-LoRes4E`: rather than having a dict observation space, observations will now\n  be 96×96×12 numpy arrays containing only the four most recent egocentric\n  views, stacked along the channels axis.\n- `-LoRes4A`: like `-LoRes4E`, but with allocentric views instead of egocentric\n  views.\n- `-LoRes3EA`: like `-LoRes4E`, but contains the three most recent egocentric\n  views concatenated with the most recent allocentric view. Useful for\n  maintaining full observability of the workspace while retaining the\n  ease-of-learning afforded by an egocentric perspective.\n- `LoResCHW4E`: like `-LoRes4E`, but transposes observations to be\n  channels-first (i.e. the Torch default).\n\nThe complete list of preprocessors is defined in\n`magical.benchmarks.DEFAULT_PREPROC_ENTRY_POINT_WRAPPERS`.\n\n## Using pre-recorded demonstrations\n\nWe distribute [reference\ndemonstrations](https://github.com/qxcv/magical-data.git) for use with the\nMAGICAL benchmark. To download the demonstrations, use\n`magical.reference_demos`:\n\n```python\nimport magical\nmagical.try_download_demos(dest=\"demos\")\n```\n\nNow the `demos` directory will contain a set of demonstration trajectories as\ncompressed pickles, like so:\n\n```\ndemos\n├── cluster-colour\n│   ├── demo-ClusterColour-Demo-v0-000.pkl.gz\n│   ├── demo-ClusterColour-Demo-v0-001.pkl.gz\n│   ├── demo-ClusterColour-Demo-v0-002.pkl.gz\n│   ├── …\n│   └── demo-ClusterColour-Demo-v0-024.pkl.gz\n├── cluster-shape\n│   ├── demo-ClusterShape-Demo-v0-000.pkl.gz\n│   └── …\n├── find-dupe\n│   ├── demo-FindDupe-Demo-v0-000.pkl.gz\n│   └── …\n├── fix-colour\n│   ├── demo-FixColour-Demo-v0-000.pkl.gz\n│   └── …\n├── make-line\n│   ├── demo-MakeLine-Demo-v0-000.pkl.gz\n│   └── …\n├── match-regions\n│   ├── demo-MatchRegions-Demo-v0-000.pkl.gz\n│   └── …\n├── move-to-corner\n│   ├── demo-MoveToCorner-Demo-v0-000.pkl.gz\n│   └── …\n├── move-to-region\n│   ├── demo-MoveToRegion-Demo-v0-000.pkl.gz\n└── └── …\n```\n\nTo load these files, you can use `magical.load_demos`:\n\n```python\nimport magical, glob\ndemo_trajs = list(magical.load_demos(glob.glob(\"demos/move-to-corner/demo-*.pkl.gz\")))\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqxcv%2Fmagical","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fqxcv%2Fmagical","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fqxcv%2Fmagical/lists"}