{"id":31755709,"url":"https://github.com/hk-zh/spil","last_synced_at":"2025-10-09T19:00:21.644Z","repository":{"id":129470335,"uuid":"605016462","full_name":"hk-zh/spil","owner":"hk-zh","description":"This is our SPIL model","archived":false,"fork":false,"pushed_at":"2024-05-22T09:29:54.000Z","size":114840,"stargazers_count":9,"open_issues_count":0,"forks_count":3,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-05-22T10:45:16.229Z","etag":null,"topics":["imitation-learning","language-conditioned-learning"],"latest_commit_sha":null,"homepage":"https://hk-zh.github.io/spil/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hk-zh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-02-22T09:09:51.000Z","updated_at":"2024-05-27T16:39:24.307Z","dependencies_parsed_at":"2024-03-29T22:39:27.819Z","dependency_job_id":"1b9f69ec-1605-429f-91d3-e3ee4036a54e","html_url":"https://github.com/hk-zh/spil","commit_stats":null,"previous_names":["hk-zh/spil"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hk-zh/spil","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Fspil","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Fspil/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Fspil/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Fspil/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hk-zh","download_url":"https://codeload.github.com/hk-zh/spil/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hk-zh%2Fspil/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279001866,"owners_count":26083226,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-09T02:00:07.460Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["imitation-learning","language-conditioned-learning"],"created_at":"2025-10-09T19:00:16.539Z","updated_at":"2025-10-09T19:00:21.633Z","avatar_url":"https://github.com/hk-zh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"#  Language-Conditioned Robot Manipulation With Base Skill Priors Under Unstructured Data\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n## [Paper](https://arxiv.org/pdf/2305.19075.pdf) | [Project Page](https://hk-zh.github.io/spil/)\n![architecture](https://github.com/hk-zh/spil/assets/57254021/939dd916-f325-42a4-b02c-d1e8b0c1345a)\n\n## Installation\nTo begin, clone this repository locally\n```bash\ngit clone --recurse-submodules https://github.com/Hongkuan-Zhou/spil\nexport ROOT=$(pwd)/spil\n\n```\nInstall requirements:\n```bash\ncd $ROOT\nconda create -n spil_venv python=3.8  # or use virtualenv\nconda activate spil_venv\nsh install.sh\n```\nIf you encounter problems installing pyhash, you might have to downgrade setuptools to a version below 58.\n\n## Download\n### CALVIN Dataset\nIf you want to train on the [CALVIN](https://github.com/mees/calvin) dataset, choose a split with:\n```bash\ncd $ROOT/dataset\nsh download_data.sh D | ABC | ABCD | debug\n```\nIf you want to get started without downloading the whole dataset, use the argument `debug` to download a small debug dataset (1.3 GB).\n### Language Embeddings\nWe provide the precomputed embeddings of the different Language Models we evaluate in the paper.\nThe script assumes the corresponding split has been already downloaded.\n```bash\ncd $ROOT/dataset\nsh download_lang_embeddings.sh D | ABC | ABCD\n```\n\n### Pre-trained Models\n#### D -\u003e D\n- Pre-trained **SKILL Generator** can be downloaded [here](https://drive.google.com/drive/folders/1y4DM45ltB6mecrkjwF48d9NpJD0eYA1M?usp=sharing) (epoch-98)\n- Pre-trained **Model** can be downloaded [here](https://drive.google.com/drive/folders/1CTcwDwhoSocZ5PdHROmOOqr3MXAM4thN?usp=sharing)\n#### ABC -\u003e D\n- Pre-trained **SKILL Generator** can be downloaded [here](https://drive.google.com/drive/folders/1EbpG5zW4siQi5BJxXc2Js_4gxmTum2jA?usp=sharing) (epoch-86)\n- Pre-trained **Model** can be downloaded [here](https://drive.google.com/drive/folders/1BDw8NXykYlsEyTAidVUqN1A-V-6VXUtV?usp=sharing)\n\n## Hardware Requirements\nTrained with:\n- **GPU** - 1x NVIDIA Tesla V100 16GB\n- **RAM** - 256GB\n- **OS** - Ubuntu 20.04\n\n## Training\nBefore you start your training, please remember to update the wandb account at\n- conf/logger/wandb.yaml\n- skill_generator/conf_sg/logger/wandb.yaml\n\nTo login your wandb account, first run:\n```\nwandb login --relogin\n```\n\n### SPIL model\nTo train the spil model with the maximum amount of available GPUS, run:\n```\npython spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset model.action_decoder.sg_chk_path=path/to/skill_generator datamodule/datasets=vision_lang loss=your_loss_setting\n```\nTo accelerate training process, the dataset can be first loaded into shared memory. (Note this way requires more RAM, please make sure your server has enough RAM)\n```\npython spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset model.action_decoder.sg_chk_path=path/to/skill_generator datamodule/datasets=vision_lang_shm loss=your_loss_setting\n```\n- The `vision_lang_shm` option loads the CALVIN dataset into shared memory at the beginning of the training,\nspeeding up the data loading during training.\nThe preparation of the shared memory cache will take some time\n(approx. 20 min at our SLURM cluster). \n- You can either use the following command to train the `skill-generator` or use a pre-trained one.\n\n### Skill-Generator \nTo train the skill generator, run:\n```\npython skill_generator/skill_generator/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset \n```\nNote that you should first train the skill-generator if you did not download the pre-trained skill generator.\n### Ablations\nHierarchical Universal Language Conditioned Policies (HULC), (Oier et al. 2022)\n```\npython spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset datamodule/datasets=vision_lang_shm model=mcil\ndatamodule=hulc loss=hulc\n```\n\nMulti-context imitation learning (MCIL), (Lynch et al., 2019):\n```\npython spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset datamodule/datasets=vision_lang_shm model=mcil\ndatamodule=mcil\n```\n\nGoal-conditioned behavior cloning (GCBC), (Lynch et al., 2019):\n```\npython spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset datamodule/datasets=vision_lang_shm model=gcbc\n```\n\n\n## Evaluation\nSee detailed inference instructions on the [CALVIN repo](https://github.com/mees/calvin#muscle-evaluation-the-calvin-challenge).\n```\npython spil/evaluation/evaluate_policy.py --dataset_path \u003cPATH/TO/DATASET\u003e --train_folder \u003cPATH/TO/TRAINING/FOLDER\u003e\n```\n\nOptional arguments:\n\n- `--checkpoint \u003cPATH/TO/CHECKPOINT\u003e`: by default, the evaluation loads the last checkpoint in the training log directory.\nYou can instead specify the path to another checkpoint by adding this to the evaluation command.\n- `--debug`: print debug information and visualize environment.\n## Real-world Experiments\n\n\n\n\nhttps://github.com/hk-zh/spil/assets/57254021/6715986a-51c0-4159-9fcf-6bd644752c62\n\n\n\n## Acknowledgements\n\nThis work uses code from the following open-source projects and datasets:\n\n#### HULC\nOriginal: [https://github.com/lukashermann/hulc](https://github.com/lukashermann/hulc)\nLicense: [MIT](https://github.com/mees/calvin/blob/main/LICENSE)\n\n#### CALVIN\nOriginal:  [https://github.com/mees/calvin](https://github.com/mees/calvin)\nLicense: [MIT](https://github.com/mees/calvin/blob/main/LICENSE)\n\n#### Sentence-Transformers\nOriginal:  [https://github.com/UKPLab/sentence-transformers](https://github.com/UKPLab/sentence-transformers)\nLicense: [Apache 2.0](https://github.com/UKPLab/sentence-transformers/blob/master/LICENSE)\n\n#### OpenAI CLIP\nOriginal: [https://github.com/openai/CLIP](https://github.com/openai/CLIP)\nLicense: [MIT](https://github.com/openai/CLIP/blob/main/LICENSE)\n\n## Citation\nIf you find this article intersting, please cite:\n```\n@ARTICLE{zhou2024languageconditioned,\n  author={Zhou, Hongkuan and Bing, Zhenshan and Yao, Xiangtong and Su, Xiaojie and Yang, Chenguang and Huang, Kai and Knoll, Alois},\n  journal={IEEE Robotics and Automation Letters}, \n  title={Language-Conditioned Imitation Learning with Base Skill Priors under Unstructured Data}, \n  year={2024},\n  volume={},\n  number={},\n  pages={1-8},\n  keywords={Imitation Learning;Robotic Manipulation},\n  doi={10.1109/LRA.2024.3466076}\n}\n```\n\n## License\nMIT License\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhk-zh%2Fspil","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhk-zh%2Fspil","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhk-zh%2Fspil/lists"}