{"id":15681368,"url":"https://github.com/lukas-blecher/blender-dextr","last_synced_at":"2025-08-11T04:03:03.131Z","repository":{"id":108669974,"uuid":"216407559","full_name":"lukas-blecher/Blender-DEXTR","owner":"lukas-blecher","description":"Use Deep Extreme Cut (DEXTR) to mask image sequences with motion tracking data from Blender","archived":false,"fork":false,"pushed_at":"2020-01-05T18:35:15.000Z","size":8002,"stargazers_count":11,"open_issues_count":0,"forks_count":4,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-07T12:13:10.488Z","etag":null,"topics":["blender","deep-learning","machine-learning","masking","tracking","video-editing"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lukas-blecher.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2019-10-20T18:25:03.000Z","updated_at":"2023-01-27T12:58:31.000Z","dependencies_parsed_at":"2023-03-24T01:19:01.139Z","dependency_job_id":null,"html_url":"https://github.com/lukas-blecher/Blender-DEXTR","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/lukas-blecher/Blender-DEXTR","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukas-blecher%2FBlender-DEXTR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukas-blecher%2FBlender-DEXTR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukas-blecher%2FBlender-DEXTR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukas-blecher%2FBlender-DEXTR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lukas-blecher","download_url":"https://codeload.github.com/lukas-blecher/Blender-DEXTR/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lukas-blecher%2FBlender-DEXTR/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":269827266,"owners_count":24481494,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-11T02:00:10.019Z","response_time":75,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["blender","deep-learning","machine-learning","masking","tracking","video-editing"],"created_at":"2024-10-03T16:53:18.343Z","updated_at":"2025-08-11T04:03:03.102Z","avatar_url":"https://github.com/lukas-blecher.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Blender DEXTR\n**Note:** Since I published this repository I have written a better and improved Addon with the same idea. It can be found here: [https://github.com/lukas-blecher/AutoMask](https://github.com/lukas-blecher/AutoMask)\n### Base Repository\nThis repository is an application of the [Deep Extreme Cut (DEXTR) repository](https://github.com/scaelles/DEXTR-PyTorch) by scaelles for [Blender](https://www.blender.org).\nThis enabels you to get automatic masks from only a few tracking points. A downside is that the poits are determining the bounding box of the mask. There is an option for padding implemented, however the performance of the pretrained model suffers greatly. This implementation is also unable to take previous frames into account and only works on a frame to frame basis. \n\n### Video to Image Sequence\nThe required input are the independent frames. For the extraction of the frames from the video, I've added the python script `framecutter.py` that saves the pictures in a compatible manner using [ffmpeg](https://www.ffmpeg.org).\n\n\n### Blender\nFinally, I've compiled an Add-on for Blender 2.7.x to extract the tracking data and saves it as a .csv file. To use it you just need to go to `File \u003e User Preferences \u003e Add-ons` and press `Install Add-on from File...`. Next choose the python file `export_tracks_blender.py`. \n\nAfter activation there is the new option `Movie clip Editor \u003e Tools Panel \u003e Solve \u003e Export tracks` in Blender. \n\nI recommend using an `Dilate/Erode`-node with `Feather` in the `Node Editor` with a negative distance.\n\n![NodeEditor](https://github.com/lukas-blecher/Blender-DEXTR/blob/master/doc/node_editor.png?raw=true)\n\n### Usage\n * Track the bounding box of the object\n * Export the tracking data\n * Get Image Sequence \n * Execute a command like `python demo.py --input PATH_FOLDER_OF_IMAGE_SEQUENCE --output OUTPUT_FOLDER --anchor-points PATH_TO_FOLDER_WITH_TRACKING_DATA`\n * Add `Image Sequence` in the Node Editor in Blender. (Optional: Add `Dilate/Erode`-node)\n\nThis is how an output could look like: \n![demo](https://github.com/lukas-blecher/Blender-DEXTR/blob/master/doc/anim.gif?raw=true)\nOriginal Video: https://www.youtube.com/watch?v=1Cir0J6jwBM\n\n---\n\n#### Original README:\n\n## Deep Extreme Cut (DEXTR)\nVisit our [project page](http://www.vision.ee.ethz.ch/~cvlsegmentation/dextr) for accessing the paper, and the pre-computed results.\n\n![DEXTR](doc/dextr.png)\n\nThis is the implementation of our work `Deep Extreme Cut (DEXTR)`, for object segmentation from extreme points.\n\n#### This code was ported to PyTorch 0.4.0! For the previous version of the code with Pytorch 0.3.1, please checkout [this branch](https://github.com/scaelles/DEXTR-PyTorch/tree/PyTorch-0.3.1).\n#### NEW: Keras with Tensorflow backend implementation also available: [DEXTR-KerasTensorflow](https://github.com/scaelles/DEXTR-KerasTensorflow )!\n\n### Abstract\nThis paper explores the use of extreme points in an object (left-most, right-most, top, bottom pixels) as input to obtain precise object segmentation for images and videos. We do so by adding an extra channel to the image in the input of a convolutional neural network (CNN), which contains a Gaussian centered in each of the extreme points. The CNN learns to transform this information into a segmentation of an object that matches those extreme points. We demonstrate the usefulness of this approach for guided segmentation (grabcut-style), interactive segmentation, video object segmentation, and dense segmentation annotation. We show that we obtain the most precise results to date, also with less user input, in an extensive and varied selection of benchmarks and datasets.\n\n### Installation\nThe code was tested with [Miniconda](https://conda.io/miniconda.html) and Python 3.6. After installing the Miniconda environment:\n\n\n0. Clone the repo:\n    ```Shell\n    git clone https://github.com/scaelles/DEXTR-PyTorch\n    cd DEXTR-PyTorch\n    ```\n \n1. Install dependencies:\n    ```Shell\n    conda install pytorch torchvision -c pytorch\n    conda install matplotlib opencv pillow scikit-learn scikit-image\n    ```\n  \n2. Download the model by running the script inside ```models/```:\n    ```Shell\n    cd models/\n    chmod +x download_dextr_model.sh\n    ./download_dextr_model.sh\n    cd ..\n    ```\n    The default model is trained on PASCAL VOC Segmentation train + SBD (10582 images). To download models trained on PASCAL VOC Segmentation train or COCO, please visit our [project page](http://www.vision.ee.ethz.ch/~cvlsegmentation/dextr/#downloads), or keep scrolling till the end of this README.\n\n3. To try the demo version of DEXTR, please run:\n    ```Shell\n    python demo.py\n    ```\nIf installed correctly, the result should look like this:\n\u003cp align=\"center\"\u003e\u003cimg src=\"doc/github_teaser.gif\" align=\"center\" width=480 height=auto/\u003e\u003c/p\u003e\n\nTo train and evaluate DEXTR on PASCAL (or PASCAL + SBD), please follow these additional steps:\n\n4. Install tensorboard (integrated with PyTorch). \n    ```Shell\n    pip install tensorboard tensorboardx\n    ```\n\n5. Download the pre-trained PSPNet model for semantic segmentation, taken from [this](https://github.com/isht7/pytorch-deeplab-resnet) repository.\n    ```Shell\n    cd models/\n    chmod +x download_pretrained_psp_model.sh\n    ./download_pretrained_psp_model.sh\n    cd ..\n    ```\n6. Set the paths in ```mypath.py```, so that they point to the location of PASCAL/SBD dataset.\n\n7. Run ```python train_pascal.py```, after changing the default parameters, if necessary (eg. gpu_id).\n\nEnjoy!!\n\n### Pre-trained models\nYou can use the following DEXTR models under MIT license as pre-trained on:\n  * [PASCAL + SBD](https://data.vision.ee.ethz.ch/kmaninis/share/DEXTR/Downloads/models/dextr_pascal-sbd.pth), trained on PASCAL VOC Segmentation train + SBD (10582 images). Achieves mIoU of 91.5% on PASCAL VOC Segmentation val.\n  * [PASCAL](https://data.vision.ee.ethz.ch/kmaninis/share/DEXTR/Downloads/models/dextr_pascal.pth), trained on PASCAL VOC Segmentation train (1464 images). Achieves mIoU of 90.5% on PASCAL VOC Segmentation val.\n  * [COCO](https://data.vision.ee.ethz.ch/kmaninis/share/DEXTR/Downloads/models/dextr_coco.pth), trained on COCO train 2014 (82783 images). Achieves mIoU of 87.8% on PASCAL VOC Segmentation val.\n\n### Citation\nIf you use this code, please consider citing the following papers:\n\n\t@Inproceedings{Man+18,\n\t  Title          = {Deep Extreme Cut: From Extreme Points to Object Segmentation},\n\t  Author         = {K.K. Maninis and S. Caelles and J. Pont-Tuset and L. {Van Gool}},\n\t  Booktitle      = {Computer Vision and Pattern Recognition (CVPR)},\n\t  Year           = {2018}\n\t}\n\n\t@InProceedings{Pap+17,\n\t  Title          = {Extreme clicking for efficient object annotation},\n\t  Author         = {D.P. Papadopoulos and J. Uijlings and F. Keller and V. Ferrari},\n\t  Booktitle      = {ICCV},\n\t  Year           = {2017}\n\t}\n\n\nWe thank the authors of [pytorch-deeplab-resnet](https://github.com/isht7/pytorch-deeplab-resnet) for making their PyTorch re-implementation of DeepLab-v2 available!\n\nIf you encounter any problems please contact us at {kmaninis, scaelles}@vision.ee.ethz.ch.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukas-blecher%2Fblender-dextr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flukas-blecher%2Fblender-dextr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flukas-blecher%2Fblender-dextr/lists"}