{"id":29355896,"url":"https://github.com/zh-plus/interseg3d-studio","last_synced_at":"2025-07-09T04:08:06.203Z","repository":{"id":284529118,"uuid":"953884971","full_name":"zh-plus/interSeg3D-Studio","owner":"zh-plus","description":"A web-based interactive 3D point cloud segmentation and annotation tool, using the AGILE3D click‑based segmentation algorithm and Gemini for 3D object recognition.","archived":false,"fork":false,"pushed_at":"2025-05-14T08:20:02.000Z","size":48612,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-14T08:22:11.203Z","etag":null,"topics":["3d","annotation","interactive","interactive-segmentation","segmentation","semantic","tool","toolbox"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zh-plus.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-03-24T08:32:15.000Z","updated_at":"2025-05-14T07:02:17.000Z","dependencies_parsed_at":"2025-05-14T08:33:03.445Z","dependency_job_id":null,"html_url":"https://github.com/zh-plus/interSeg3D-Studio","commit_stats":null,"previous_names":["zh-plus/interseg3d-studio"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zh-plus/interSeg3D-Studio","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zh-plus%2FinterSeg3D-Studio","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zh-plus%2FinterSeg3D-Studio/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zh-plus%2FinterSeg3D-Studio/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zh-plus%2FinterSeg3D-Studio/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zh-plus","download_url":"https://codeload.github.com/zh-plus/interSeg3D-Studio/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zh-plus%2FinterSeg3D-Studio/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264390706,"owners_count":23600563,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["3d","annotation","interactive","interactive-segmentation","segmentation","semantic","tool","toolbox"],"created_at":"2025-07-09T04:08:04.788Z","updated_at":"2025-07-09T04:08:06.181Z","avatar_url":"https://github.com/zh-plus.png","language":"Python","readme":"# interSeg3D-Studio\n\nAn web-based interactive 3D point cloud annotation tool, using [AGILE3D](https://github.com/ywyue/AGILE3D) for\nclick-based segmentation algorithm and Gemini for 3D object recognition \u0026 description.\n\n\u003e [!NOTE]  \n\u003e Note that this project is still in development. The current version is a prototype and may contain bugs.\n\n\n![interSeg3D-Studio](assets/MainUI.png)\n\n## Features\n\n- **Interactive 3D Annotation**: Easily mark and annotate objects in 3D point clouds using positive and negative clicks\n- **AI-Powered Segmentation**: Click-based segmentation\n  with [AGILE3D model](https://github.com/ywyue/AGILE3D)\n  ![seg-demo.gif](assets/seg-demo.gif)\n- **Object Analysis**: Automatically identify and describe objects in your scene\n  ![Analyze](assets/Object%20Analyze.png)\n- **Metadata Loading**: Import previous annotations and object information from metadata files\n\n## Installation\n\n### Prerequisites\n\n- Node.js (v16 or newer)\n- Python 3.10 (recommended, but should work with 3.8+)\n- CUDA-compatible GPU (recommended)\n\n### Backend Setup\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/zh-plus/interSeg3D-Studio\n   cd interSeg3D-Studio\n   ```\n\n2. Install AGILE3D according to the instructions at:\n   https://github.com/ywyue/AGILE3D/blob/main/installation.md\n   (Dont clone the repository, just install the environment. And try to use gcc-9 \u0026 g++-9 to install MinkowskiEngine.)\n\n3. Download the pre-trained model weights from:\n   https://polybox.ethz.ch/index.php/s/RnB1o8X7g1jL0lM, and put it into the `src/backend/agile3d/weights` directory.\n\n4. Create a `.env` file in the `src/backend` directory with:\n\n   ```\n   GOOGLE_API_KEY='your_google_api_key'  # For object recognition\n   ```\n\n### Frontend Setup\n\n1. Install Node.js dependencies:\n   ```bash\n   cd src/frontend\n   npm install\n   ```\n\n2. Create a `.env` file in the `src/frontend` directory with:\n   ```\n   VITE_API_BASE_URL=http://localhost:9500\n   VITE_USE_PROXY=true\n   ```\n\n## Running the Application\n\n1. Start the backend server:\n   ```bash\n   python src/backend/app.py\n   ```\n\n2. In a new terminal, start the frontend development server:\n   ```bash\n   cd src/frontend\n   npm run dev\n   ```\n\n3. Open your browser and navigate to:\n   ```\n   http://localhost:3001\n   ```\n\n## Usage Guide\n\n### Interaction Modes\n\nThe application has three main interaction modes:\n\n1. **Navigate Mode**: Rotate, pan, and zoom the view\n2. **Annotate Mode**: Mark points on objects or background\n3. **Select Mode**: Select and edit segmented objects\n\n### Basic Workflow\n\n1. **Upload** a PLY file using the file upload panel\n2. **Load metadata** to import previous annotations and object information (optional)\n3. **Create objects** by entering names and clicking \"Create Object\"\n4. **Select an object** from the list and switch to \"Annotation Mode\"\n5. **Mark points** on the object by clicking on the point cloud\n6. **Mark background** by switching to \"Background\" mode and clicking non-object areas\n7. **Run segmentation** by clicking the \"RUN SEGMENTATION\" button\n8. **(Optional) Analyze objects** to automatically identify and describe them (remember to click `APPLY LABEL` or\n   `APPLY ALL RESULTS`)\n9. **Switch to Select Mode** to click on segmented objects and edit their labels and descriptions\n10. **Save results** to download the segmented point cloud and metadata\n\n### Keyboard Shortcuts\n\n- **A**: Toggle between navigation and annotation modes\n- **S**: Activate select mode\n- **Enter**: Run segmentation\n- **N**: Create a new object with default name \"new obj\"\n- **Ctrl+Z**: Undo the last click\n- **Shift+Ctrl+Z**: Redo an undone click\n- **Ctrl+S**: Save object information\n\n### Mouse Controls\n\n- **Left-click + drag** (Navigation Mode): Rotate the view\n- **Right-click + drag**: Pan the view\n- **Scroll wheel**: Zoom in/out\n- **Left-click** (Annotation Mode): Mark a point\n- **Left-click** (Select Mode): Select an object to edit its label and description\n\n## How it works\n\nAfter segmenting the project interactively using clicks, all the objects are scanned from multiple angles and analyzed\nusing the AI object recognition model. The model uses the LLM API to identify the objects and provide a description of\nthe object. The description includes the object's name, color, and size.\n\nFor example, the object are indicated using convex hulls and the cameras are positioned at green spheres.\n\n![Camera-Position](assets/Camera-Position.png)\n\n## Development\n\n### Project Structure\n\n```\n├── package.json\n├── src\n│   ├── backend\n│   │   ├── agile3d               # Interactive Segmentation Model\n│   │   ├── app.py                # FastAPI server\n│   │   ├── inference.py          # Segmentation model\n│   │   ├── view_rendering.py     # View generation\n│   │   └── visual_obj_recognition.py  # AI object recognition\n│   └── frontend\n│       ├── App.vue               # Main application component\n│       ├── components            # Vue components\n│       ├── composables           # Vue composition functions\n│       ├── services              # Service layer\n│       ├── types                 # TypeScript type definitions\n│       └── utils                 # Utility functions\n```\n\n## Contributing\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add some amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## Acknowledgments\n\n- [AGILE3D](https://github.com/ywyue/AGILE3D) for the segmentation models\n- [Three.js](https://threejs.org/) for 3D rendering capabilities\n- [Open3D](http://www.open3d.org/) for point cloud processing","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzh-plus%2Finterseg3d-studio","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzh-plus%2Finterseg3d-studio","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzh-plus%2Finterseg3d-studio/lists"}