{"id":24396809,"url":"https://github.com/ai4ce/seedo","last_synced_at":"2025-08-02T15:33:02.206Z","repository":{"id":261558691,"uuid":"868449859","full_name":"ai4ce/SeeDo","owner":"ai4ce","description":"Human Demo Videos to Robot Action Plans","archived":false,"fork":false,"pushed_at":"2024-11-14T07:50:05.000Z","size":162084,"stargazers_count":48,"open_issues_count":1,"forks_count":9,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-04-11T17:00:00.298Z","etag":null,"topics":["learning-from-demonstration","robotics","task-planning","vlm"],"latest_commit_sha":null,"homepage":"https://ai4ce.github.io/SeeDo","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ai4ce.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-10-06T12:18:24.000Z","updated_at":"2025-04-03T07:17:36.000Z","dependencies_parsed_at":"2025-04-11T17:02:56.699Z","dependency_job_id":null,"html_url":"https://github.com/ai4ce/SeeDo","commit_stats":null,"previous_names":["ai4ce/seedo"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ai4ce/SeeDo","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FSeeDo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FSeeDo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FSeeDo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FSeeDo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ai4ce","download_url":"https://codeload.github.com/ai4ce/SeeDo/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ai4ce%2FSeeDo/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":268410747,"owners_count":24246066,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-02T02:00:12.353Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["learning-from-demonstration","robotics","task-planning","vlm"],"created_at":"2025-01-19T21:58:32.899Z","updated_at":"2025-08-02T15:33:02.190Z","avatar_url":"https://github.com/ai4ce.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# SeeDo: Human Demo Video to Robot Action Plan via Vision Language Model\n\u003ca href='https://arxiv.org/abs/2410.08792'\u003e\u003cimg src='https://img.shields.io/badge/Paper-arXiv-red'\u003e\u003c/a\u003e \u003ca href='https://ai4ce.github.io/SeeDo/'\u003e\u003cimg src='https://img.shields.io/badge/Project-website-green'\u003e\u003c/a\u003e \u003ca href='https://huggingface.co/datasets/ai4ce/SeeDo'\u003e\u003cimg src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-blue'\u003e\u003c/a\u003e\n\n**VLM See, Robot Do (SeeDo)** is a method that uses large vision models, tracking models and vision-language models to extract robot action plans from human demonstration videos, specifically focusing on long horizon pick-and-place tasks. The action plan is then implemented in real-world and PyBullet simulation environments.\n\n![main](https://github.com/ai4ce/SeeDo/blob/main/media/main.jpg)\n\n## News\n- [2025/06] SeeDo is accepted by IROS 2025! We will update the camera-ready version soon.\n\n## Setup Instructions\n\nNote that SeeDo relies on GroundingDINO, SAM and SAM2. The code has only been tested on Ubuntu 20.04. The version of CUDA tested is 11.8, the Pytorch version is 2.3.1+cu118.\n\n- Install SeeDo and create a new environment\n\n```python\ngit clone https://github.com/ai4ce/SeeDo\nconda create --name seedo python=3.10.14\nconda activate seedo\ncd SeeDo\npip install -r requirements.txt\n```\n\n- Install Pytorch (Only for CUDA 11.8 user)\n\n```python\npip install torch==2.3.1+cu118 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118\n```\n\n- Install GroundingDINO, SAM and SAM2 in the same environment\n\n```python\ngit clone https://github.com/IDEA-Research/GroundingDINO\ngit clone https://github.com/facebookresearch/segment-anything.git\ngit clone https://github.com/facebookresearch/segment-anything-2.git\n```\n\n- Make sure these models are installed in editable packages\n\n```python\ncd GroundingDINO\npip install -e .\n```\nAnd do the same with segment-anything, segment-anything-2\n\n- We have slightly modified the GroundingDINO\n\nIn `GroundingDINO/groundingdino/util/inference.py`, we add a function to help inference on an array of images. Please paste the following function into `inference.py`.\n\n```python\ndef load_image_from_array(image_array: np.array) -\u003e Tuple[np.array, torch.Tensor]:\n    transform = T.Compose(\n        [\n            T.RandomResize([800], max_size=1333),\n            T.ToTensor(),\n            T.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),\n        ]\n    )\n    image_source = Image.fromarray(image_array)\n    image_transformed, _ = transform(image_source, None)\n    return image_array, image_transformed\n```\n\n- The code still uses one checkpoint from segment-anything.\n\nMake sure you download it in the SeeDo folder.\n**`default` or `vit_h`: [ViT-H SAM model.](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth)**\n\n- Obtain an OpenAI API key and create a `key.py` file under `VLM_CaP/src`\n\n```python\ncd VLM_CaP/src\ntouch key.py\necho 'projectkey = \"YOUR_OPENAI_API_KEY\"' \u003e key.py\n```\n\n## Pipeline\n\nThere are mainly four parts of SeeDo. To ensure the video is successfully processed in subsequent steps, use `convert_video.py` to convert the video to the appropriate encoding before inputting it. The `convert_video.py` script accepts two parameters: `--input` and `--output`, which specify the path of your original video and the path of the converted video, respectively.\n\n1. **Keyframe Selection Module**\n\n   `get_frame_by_hands.py`: The `get_frame_by_hands.py` script allows selecting key frames by tracking hand movements. It accepts two parameters.\n\n    `--video_path`, which specifies the path of the input video.\n\n   `--output_dir`, which designates the directory where the key frames will be saved. If `output_dir` is not specified, the keyframes will be saved to `./output` by default. For debugging purpose, the hand image and hand speed plot will also be saved in this directory.\n\n2. **Visual Perception Module**\n\n   `track_objects.py`: The `track_objects.py` script is used to track each object and add a visual prompt for the objects. It also returns a string containing the center coordinates of each object in the key frames. The script accepts three parameters.\n\n    `--input` is the video converted to the appropriate format.\n\n    `--output` specifies the output path for the video with the visual prompts. \n\n    `--key_frames` is the list of key frame indices obtained from `get_frames_by_hands.py`.\n\n   This module will return a `box_list` string stored for useage in  VLM Reasoning Module\n\n3. **VLM Reasoning Module**\n\n   `vlm.py`: The `vlm.py` script performs reasoning on the key frames and generates an action list for the video. It accepts three parameters.\n\n    `--input` is the video with visual prompts added by the Visual Perception Module.\n\n    `--list` is the keyframe index list obtained from the Keyframe Selection Module.\n\n   `--bbx_list` is the `box_list` string obtained from the Visual Perception Module.\n\n   This module will return two strings: `obj_list` representing for the objects in the environment; `action_list` representing for the action list performed on these objects.\n\n4. **Robot Manipulation Module**\n\n   `simulation.py`: The `simulation.py` script accepts three parameters: `obj_list`, `action_list`, `output`. It first initializes a random simulation scene based on the `obj_list`, and then executes pick-and-place tasks according to the `action_list`, and finally write the video to output.\n\n   Example usage: `python simulation.py --action_list \"put chili on bowl and then put eggplant on glass\" --obj_list chili carrot eggplant bowl glass --output demo2.mp4`\n\n   Note that this part uses a modified version of the Code as Policies framework, and its successful execution depends heavily on whether the objects are already modeled and whether the corresponding execution functions for actions are present in the prompt. We provide a series of new object models and prompts that are compatible with our defined action list. If you want to operate on unseen objects, you will need to provide the corresponding object modeling, and modify the LMP and prompt file accordingly.\n\n   We provide some simple object modelings of vegetables on hugging face. Download from https://huggingface.co/datasets/ai4ce/SeeDo/tree/main/SeeDo\n\n   There will be an `assets.zip` file, extract that file into `assets` and make sure this folder is under the path of VLM_CaP. `VLM_CaP/assets` will then be used by `simulation.py` for simulation.\n\n   It will write out a video of robot manipulation of a series of pick-and-place tasks in simulation.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai4ce%2Fseedo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fai4ce%2Fseedo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fai4ce%2Fseedo/lists"}