{"id":19386315,"url":"https://github.com/davidstutz/hierarchical-graph-based-video-segmentation","last_synced_at":"2025-04-23T22:32:32.248Z","repository":{"id":33621974,"uuid":"37274229","full_name":"davidstutz/hierarchical-graph-based-video-segmentation","owner":"davidstutz","description":"Implementation of the hierarchical graph-based video segmentation algorithm proposed by Grundmann et al. [1] based on the image segmentation algorithm by Felzenswalb and Huttenlocher [2].","archived":false,"fork":false,"pushed_at":"2018-11-28T13:56:51.000Z","size":57376,"stargazers_count":24,"open_issues_count":0,"forks_count":14,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-02T21:51:20.727Z","etag":null,"topics":["computer-vision","opencv","video-segmentation"],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/davidstutz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-06-11T17:00:34.000Z","updated_at":"2024-02-15T16:27:24.000Z","dependencies_parsed_at":"2022-08-07T22:16:07.313Z","dependency_job_id":null,"html_url":"https://github.com/davidstutz/hierarchical-graph-based-video-segmentation","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidstutz%2Fhierarchical-graph-based-video-segmentation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidstutz%2Fhierarchical-graph-based-video-segmentation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidstutz%2Fhierarchical-graph-based-video-segmentation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/davidstutz%2Fhierarchical-graph-based-video-segmentation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/davidstutz","download_url":"https://codeload.github.com/davidstutz/hierarchical-graph-based-video-segmentation/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250527319,"owners_count":21445352,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","opencv","video-segmentation"],"created_at":"2024-11-10T10:05:02.858Z","updated_at":"2025-04-23T22:32:27.238Z","avatar_url":"https://github.com/davidstutz.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Hierarchical Graph-Based Video Segmentation\n\n[![Build Status](https://travis-ci.org/davidstutz/hierarchical-graph-based-video-segmentation.svg?branch=master)](https://travis-ci.org/davidstutz/hierarchical-graph-based-video-segmentation)\n\nThis is an implementation of the hierarchical graph-based video segmentation algorithm proposed by Grundmann et al. [1] based on the graph-based image segmentation algorithm by Felzenswalb and Huttenlocher [2].\n\n    [1] M. Grundmann, V. Kwatra, M. Han, and I. A. Essa.\n        Efficient hierarchical graph-based video segmentation.\n        In Conference on Computer Vision and Pattern Recognition, pages 2141–2148,\n        San Francisco, 2010, June 2010.\n    [2] P. F. Felzenszwalb and D. P. Huttenlocher.\n        Efficient graph-based image segmentation.\n        International Journal of Computer Vision, 59(2):167–181, September 2004.\n\nFurther, evaluation metrics based on the Precision-Recall Framework for videos [3,4], Undersegmentation Error [4,5] and Achievable Segmentation Accuracy [3] are provided.\n\n    [3] C. Xu and J. J. Corso.\n        Evaluation of super-voxel methods for early video processing.\n        In Computer Vision and Pattern Recognition, Conference on,\n        pages 1202–1209, Providence, RI, June 2012.\n    [4] F. Galasso, N. S. Nagaraja, T. J. Cardenas, T. Brox and B.Schiele.\n        A Unified Video Segmentation Benchmark: Annotation, Metrics and Analysis.\n        In International Conference on Computer Vision, \n        pages 3527-3534, Sydney, Australia, December 2013.\n    [5] P. Neubert and P. Protzel.\n        Superpixel benchmark and comparison.\n        In Forum Bildverarbeitung, Regensburg, Germany, November 2012.\n\nThe algorithm is based on optical flow, therefore a command line tool to pre-compute optical flow using OpenCV is provided. Further, the `alley_1` sequence form the [Sintel dataset]() [6] is provided. Pre-computed flow (in the correct format, see `alley_1_flow` and `optical_flow_cli`) as well as ground truth is provided (see `alley_1_gt`).\n\n    [6] D. J. Butler, J. Wulff, G. B. Stanley and M. J. Black.\n        A naturalistic open source movie for optical flow evaluation.\n        In European Conference on Computer Vision, pages 611--625, October 2012.\n\n![Example: original sequence, initial oversegmentation and a selected hierarchy level.](results/alley_1_full_spaced_690.png?raw=true \"Example: original sequence, initial oversegmentation and a selected hierarchy level.\")\n\n## Compile\n\nThe implementation is based on [OpenCV](http://opencv.org/), see [here](http://docs.opencv.org/doc/tutorials/introduction/linux_install/linux_install.html#linux-installation) for installation instructions, [CMake](http://www.cmake.org/) and [Boost](http://www.boost.org/). The implementation has been tested on Ubuntu 14.04 and 14.10:\n\n    $ sudo apt-get install build-essential cmake libboost-all-dev\n    $ git clone https://github.com/davidstutz/hierarchical-graph-based-video-segmentation.git\n    $ cd hierarchical-graph-based-video-segmentation\n    $ mkdir build\n    $ cd build\n    $ cmake ..\n    $ make\n    Scanning dependencies of target segment\n    [ 16%] Building CXX object lib_segment/CMakeFiles/segment.dir/graph_segmentation.cpp.o\n    [ 33%] Building CXX object lib_segment/CMakeFiles/segment.dir/evaluation.cpp.o\n    [ 50%] Building CXX object lib_segment/CMakeFiles/segment.dir/io.cpp.o\n    [ 66%] Building CXX object lib_segment/CMakeFiles/segment.dir/io_util.cpp.o\n    Linking CXX static library libsegment.a\n    [ 66%] Built target segment\n    Scanning dependencies of target segment_cli\n    [ 83%] Building CXX object segment_cli/CMakeFiles/segment_cli.dir/main.cpp.o\n    Linking CXX executable segment_cli\n    [ 83%] Built target segment_cli\n    Scanning dependencies of target optical_flow_cli\n    [100%] Building CXX object optical_flow_cli/CMakeFiles/optical_flow_cli.dir/main.cpp.o\n    Linking CXX executable optical_flow_cli\n    [100%] Built target optical_flow_cli\n    # Run video segmentation on the alley_1 sequence and save visualization to build/output_vis:\n    $ ./segment_cli/segment_cli ../alley_1 ../alley_1_flow/ --vis-dir output_vis --input-gt ../alley_1_gt/\n    ----- Level 0\n    Built graph (1.43415).\n    Oversegmented graph (3.04033).\n    Enforced minimum region size (2.0374).\n    3D Boundary Recall: 0.990899\n    3D Undersegmentation Error: 0.0906389\n    3D Achievable Segmentation Accuracy: 0.954518\n    # ...\n\nThe two command line tools provide the following options:\n\n    $ ./segment_cli/segment_cli --help\n    Allowed options:\n      --help                           produce help message\n      --input-video arg                directory of input video (provided as \n                                       sequence of individual images)\n      --input-flow arg                 input flow directory (text files in \n                                       cv::Storage format, see io.h)\n      --input-gt arg                   input ground truth; if given, some metrics \n                                       will be computed and displayed\n      --length arg (=10)               length of video to oversegment (may be lower\n                                       than the actual sequence length)\n      --flow-weight arg (=0.200000003) weight on flow angle for edge weight \n                                       computation\n      --threshold arg (=0.0199999996)  threshold (is multiplied by 1.3 for each \n                                       additional hierarchy level)\n      --hierarchies arg (=40)          number of hierarchies\n      --vis-dir arg                    visualization directory (default will not \n                                       visualize the result!)\n      --output-dir arg (=output)       output directory\n\n    # Compute OpenCV optical flow and save in the correct format.\n    $ ./optical_flow_cli/optical_flow_cli --help\n    Allowed options:\n      --help                                produce help message\n      --input-dir arg                       input directory\n      --pyramid-scale arg (=0.5)            pyramid scale\n      --pyramid-levels arg (=1)             pyramid levels\n      --window-size arg (=7)                window size\n      --num-iterations arg (=10)            number of iterations at each pyramid \n                                            level\n      --polynomial-neighborhood-size arg (=5)\n                                            size of pixel neighborhood to find \n                                            polynomial expansion\n      --polynomial-sigma arg (=1.10000002)  standard deviation of Gaussian used to \n                                            smooth derivatives\n      --gaussian-filter                     use a Gaussian filter isntead of a box \n                                            filter\n      --output-dir arg (=output)            output directory\n\n\n## Usage\n\nAn usage example can be found in `segment_cli/main.cpp`. Example:\n\n    // The video is expected to be provided as sequence of images, see alley_1 as example.\n    // The same holds for the optical flow, which is provided using text files formatted\n    // according to cv::FileStorage, see io.h.\n    boost::filesystem::path in_dir(\"path/to/video\");\n    boost::filesystem::path flow_dir(\"path/to/flow\");\n    \n    // Length of the seuqnce to read (may be smaller than the actual length).\n    int length = 10;\n    Video video = IO::readVideo(in_dir, length);\n    FlowVideo flowVideo = IO::readFlowVideo(flow_dir, length);\n    \n    assert(video.getFrameNumber() \u003e 0);\n    assert(video.getFrameNumber() == flowVideo.getFrameNumber());\n    \n    // Parameters:\n    int M = 300; // Minimum segment size, enforced after segmentation.\n    int L = 40; // Number of hierarchy levels.\n    float c = 0.02; // Threshold used for the initial oversegmentation.\n    float beta = 0.25; // Importance of flow information for edge weight computation.\n    float alpha = 1 - beta; // ... importance of color.\n    \n    GraphSegmentationMagic* magic = new GraphSegmentationMagicThreshold(c);\n    GraphSegmentationDistance* distance = new GraphSegmentationEuclideanRGBFlowAngle(alpha, beta);\n    \n    GraphSegmentation segmenter(distance, magic);\n    \n    // Setup the video graph.\n    segmenter.buildGraph(video, flowVideo);\n    segmenter.buildEdges();    \n\n    // Oversegment the video.\n    segmenter.oversegmentGraph();\n    \n    // Enforce minimum segment size.\n    segmenter.enforceMinimumSegmentSize(M);\n    \n    // Get the corresponding segmentation video (that is containing the labels.\n    // The video contains length three channel images where the labels are encoded \n    // as 24 bit numbers, see io_util.h\n    SegmentationVideo sv_video = segmenter.deriveLabels();\n    IO::writeSegmentationVideo(out_dir / boost::filesystem::path(\"0\"), sv_video);\n    \n    // Visualize by randonly coloring segments.\n    IO::writeColoredSegmentationVideo(vis_dir / boost::filesystem::path(\"0\"), sv_video);\n    \n    // Each new hierarchy level, the threshold is raised by the factor 1.3.\n    GraphSegmentationHierarchyMagic* hmagic = new GraphSegmentationHierarchyMagicThreshold(c, 1.3);\n    GraphSegmentationHierarchyDistance* hdistance = \n            new GraphSegmentationHierarchyRGBChiSquareFlowAngle(alpha, beta);\n    \n    segmenter.setHierarchyMagic(hmagic);\n    segmenter.setHierarchyDistance(hdistance);\n    \n    for (int l = 0; l \u003c L; l++) {\n        // Build the region graph.\n        segmenter.buildRegionGraph();\n        \n        // Segment the region graph.\n        segmenter.addHierarchyLevel();\n        \n        // Enforce minimum segment size.\n        segmenter.enforceMinimumSegmentSize(l/2 * M);\n        \n        SegmentationVideo sv_video = segmenter.deriveLabels();\n        IO::writeSegmentationVideo(out_dir / boost::filesystem::path(std::to_string(l + 1)), \n                sv_video);\n        \n        IO::writeColoredSegmentationVideo(vis_dir / boost::filesystem::path(std::to_string(l + 1)), \n                sv_video);\n    }\n\nFurther documentation can be found in the corresponding header files: `graph_segmentation.h`, `evaluation.h`, `io.h`.\n\n## License\n\nCopyright (c) 2014-2018 David Stutz\n\n**Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use this software and associated documentation files (the \"Software\").**\n\nThe authors hereby grant you a non-exclusive, non-transferable, free of charge right to copy, modify, merge, publish, distribute, and sublicense the Software for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects.\n\nAny other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artefacts for commercial purposes.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\nYou understand and agree that the authors are under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Software. The authors nevertheless reserve the right to update, modify, or discontinue the Software at any time.\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. You agree to refer to this repository in documents and papers that report on research using the Software.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidstutz%2Fhierarchical-graph-based-video-segmentation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdavidstutz%2Fhierarchical-graph-based-video-segmentation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdavidstutz%2Fhierarchical-graph-based-video-segmentation/lists"}