{"id":43222390,"url":"https://github.com/chrockey/visualreasoning_dataset","last_synced_at":"2026-02-01T09:09:21.497Z","repository":{"id":335232933,"uuid":"1126578108","full_name":"chrockey/visualreasoning_dataset","owner":"chrockey","description":null,"archived":false,"fork":false,"pushed_at":"2026-01-29T07:36:24.000Z","size":71683,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-01-29T21:43:28.053Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chrockey.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-02T07:27:07.000Z","updated_at":"2026-01-07T07:27:28.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/chrockey/visualreasoning_dataset","commit_stats":null,"previous_names":["chrockey/visualreasoning_dataset"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/chrockey/visualreasoning_dataset","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrockey%2Fvisualreasoning_dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrockey%2Fvisualreasoning_dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrockey%2Fvisualreasoning_dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrockey%2Fvisualreasoning_dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chrockey","download_url":"https://codeload.github.com/chrockey/visualreasoning_dataset/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chrockey%2Fvisualreasoning_dataset/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28974379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-01T08:16:14.655Z","status":"ssl_error","status_checked_at":"2026-02-01T08:06:51.373Z","response_time":56,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-02-01T09:09:17.249Z","updated_at":"2026-02-01T09:09:21.490Z","avatar_url":"https://github.com/chrockey.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Visual Reasoning Annotation\n\nPipeline system for visual reasoning dataset annotation using Molmo and SAM2.\n\n## Installation\n\n```bash\n./install.sh\n```\n\n## Supported Pipelines\n\n| Pipeline | Use Case | Command |\n|----------|----------|---------|\n| **GT Visual Trace** | With GT pose (3D→2D projection) | `python -m src.pipelines.gt_visual_trace` |\n| **Visual Trace** | Without GT pose (SAM3 video tracking) | `python -m src.pipelines.affordance_type1` |\n\n## Supported Datasets\n\n| Dataset | Status | Additional Metadata |\n|---------|--------|---------------------|\n| **EgoDex** | ✅ | Camera parameters (intrinsics/extrinsics), joint transforms (70+ joints), confidence scores, MANO hand poses |\n| **Open X-Embodiment** | ✅ | Robot states (joint angles), actions, 512-dim language embeddings |\n| **AgiBotWorld** | ✅ | Camera parameters (intrinsics/extrinsics), additional views, actions, proprio_stats |\n| **HoloAssist** | ✅ | Hand poses (left/right), depth |\n\nAll datasets inherit from `BaseDataset` and provide:\n- `video_name`: Video identifier string (for tracking annotations)\n- `frames`: Video frames as (N, H, W, 3) numpy array\n- `description`: Task description string\n- `metadata`: Dataset-specific annotations and additional data\n\n### Using Datasets\n\n```python\nfrom src.datasets.egodex import EgoDexDataset\nfrom src.datasets.oxe import OXEDataset\nfrom src.datasets.agibotworld import AgiBotWorldDataset\nfrom src.datasets.holoassist import HoloAssistDataset\n\n# EgoDex: Egocentric hand manipulation videos\ndataset = EgoDexDataset()  # Default: vla-dataset-samples/egodex\nprint(f\"Videos: {len(dataset)}\")\ndata = dataset[0]\n# data['video_name']: \"part1/add_remove_lid/0\"\n# data['frames']: (288, 1080, 1920, 3)\n# data['descriptions]: \n# [(0, 287, 'Add lids onto four cups placed on a wooden table with a red background.')]\n# data['metadata']: camera, MANO hand poses, transforms\n\n# Open X-Embodiment: Robot manipulation episodes\ndataset = OXEDataset()  # Default: vla-dataset-samples/open-x-embodiment\nprint(f\"Episodes: {len(dataset)}\")\ndata = dataset[0]\n# data['video_name']: \"asu_table_top_converted_externally_to_rlds/00003/0\"\n#                     {dataset_name}/{shard_id}/{episode_in_shard}\n# data['frames']: (3225, 256, 256, 3)\n# data['descriptions]: \n# [(0, 3224, 'Interact with the objects in diverse but meaningful ways.')]\n# data['metadata']['state']: robot joint states\n# data['metadata']['action']: robot actions\n# data['metadata']['language_embedding']: 512-dim embedding\n# data['metadata']['tfrecord_info']: shard tracking info for annotations\n\n# AgiBotWorld-Beta: Bimanual manipulation dataset\ndataset = AgiBotWorldDataset()\nprint(f\"Episodes: {len(dataset)}\")\ndata = dataset[0]\n# data['frames']: (1295, 480, 640, 3)\n# data['descriptions']:\n# [(36, 187, 'Retrieve cucumber from the shelf.'),\n#  (187, 426, 'Place the held cucumber into the plastic bag in the shopping cart.'),\n#  (426, 591, 'Retrieve tomato from the shelf.'),\n#  (591, 788, 'Place the held tomato into the plastic bag in the shopping cart.'),\n#  (788, 956, 'Retrieve corn from the shelf.'),\n#  (956, 1232, \"Place the held corn into the shopping cart's plastic bag.\")]\n# data['metadata']['hand_left_frames]: frames from hand-left cam\n# data['metadata']['hand_right_frames]: frames from hand-right cam\n# data['metadata']['action_config']: action text, skill(pick,place,..)\n# data['metadata']['proprio_stats]: effector (orientation, velocity, ..)\n\n# HoloAssist: Egocentric human interaction dataset\ndataset = HoloAssistDataset()\nprint(f\"Videos: {len(dataset)}\")\ndata = dataset[0]\n# data['frames]: (9933, 504, 896, 3)\n# data['descriptions']\n# [(268, 691, 'The student grabs the GoPro.'),\n#  (731, 1620, 'The student changes the battery for the GoPro.'),\n#  (1672, 7444, 'The student opens the GoPro.'),\n#  (7496, 7667, 'The student turns on their GoPro.'),\n#  (7685, 7957, 'The student turns off the gopro.'),\n#  (7988, 8604, 'The student assembles the mounting_peg.'),\n#  (8679, 8864, 'The student disassemble the mounting_peg.'),\n#  (8883, 9526, 'The student assemble handheld_grip.'),\n#  (9543, 9896, 'The students disassemble the handheld_grip.')]\n# data['metadata']['depth']: Depth\n# data['metadata']['hands_left']: Hand pose (left)\n# data['metadata']['hands_right']: Hand pose (right)\n# data['metadata']['pose_sync']: Camera pose\n\n```\n\n## Project Structure\n\n```\nsrc/\n├── datasets/            # Dataset loaders\n│   ├── base.py          # BaseDataset abstract class\n│   ├── agibotworld.py   # AgiBotWorld-Beta (Bimanual manipulation robot manipulation)\n│   ├── egodex.py        # EgoDex (egocentric hand manipulation)\n│   ├── holoassist.py    # HoloAssist (Egocentric human interaction)\n│   └── oxe.py           # Open X-Embodiment (robot manipulation)\n├── models/              # Model wrappers\n│   ├── molmo.py         # VLM for point extraction\n│   └── sam2.py          # Segmentation model\n├── pipelines/           # \u003c-- WORK HERE\n│   ├── base.py\n│   ├── affordance_type1.py\n│   ├── affordance_type2.py\n│   └── visual_trace.py\n└── job_server/          # Distributed job system\n    ├── server.py        # FastAPI REST server\n    ├── worker.py        # Base worker class\n    ├── pipeline_worker.py\n    └── client.py        # CLI client\n```\n\n## Testing Pipelines\n\n```bash\npython -m src.pipelines.affordance_type1\npython -m src.pipelines.affordance_type2\npython -m src.pipelines.visual_trace\n```\n\u003cdetails open\u003e\n\u003csummary\u003eVisualize GT robot gripper trajectories\u003c/summary\u003e\n\n- MP4 files are saved under viz/*\n- Runnable datasets\n    - [x] AgiBotWorld\n    - [ ] EgoDex\n    - [ ] HoloAssist\n    - [ ] Open X-Embodiment\n\u003c/details\u003e\n\n```bash\npython -m src.pipelines.gt_visual_trace\n``` \n\n## Creating a New Pipeline\n\n```python\n# src/pipelines/my_pipeline.py\nfrom typing import Any, Dict\nfrom src.models.molmo import Molmo\nfrom src.models.sam2 import SAM2\nfrom .base import BasePipeline\n\n\nclass MyPipeline(BasePipeline):\n    def __init__(self, threshold: float = 0.5):\n        super().__init__()\n        self.threshold = threshold\n        self.molmo = Molmo()\n        self.sam2 = SAM2()\n\n    def preprocess(self, data_dict: Dict[str, Any]):\n        return data_dict\n\n    def process(self, data_dict: Dict[str, Any]):\n        return {\"result\": \"...\"}\n\n\nif __name__ == \"__main__\":\n    pipeline = MyPipeline(threshold=0.7)\n    result = pipeline({\"video_path\": \"/path/to/test.mp4\"}, save_dir=\"/tmp/test\")\n    print(result)\n```\n\nThen register in `src/job_server/pipeline_worker.py`:\n\n```python\nPIPELINES = {\n    ...\n    \"my_pipeline\": \"src.pipelines.my_pipeline.MyPipeline\",\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrockey%2Fvisualreasoning_dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchrockey%2Fvisualreasoning_dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchrockey%2Fvisualreasoning_dataset/lists"}