{"id":13420181,"url":"https://github.com/torrvision/crfasrnn","last_synced_at":"2025-04-07T07:12:16.490Z","repository":{"id":41086352,"uuid":"43455137","full_name":"torrvision/crfasrnn","owner":"torrvision","description":"This repository contains the source code for the semantic image segmentation method described in the ICCV 2015 paper: Conditional Random Fields as Recurrent Neural Networks. http://crfasrnn.torr.vision/","archived":false,"fork":false,"pushed_at":"2019-11-30T13:17:39.000Z","size":3050,"stargazers_count":1344,"open_issues_count":47,"forks_count":459,"subscribers_count":81,"default_branch":"master","last_synced_at":"2025-03-31T06:02:57.284Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"MATLAB","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/torrvision.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-30T19:47:15.000Z","updated_at":"2025-03-02T07:09:12.000Z","dependencies_parsed_at":"2022-09-05T15:51:27.343Z","dependency_job_id":null,"html_url":"https://github.com/torrvision/crfasrnn","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/torrvision%2Fcrfasrnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/torrvision%2Fcrfasrnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/torrvision%2Fcrfasrnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/torrvision%2Fcrfasrnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/torrvision","download_url":"https://codeload.github.com/torrvision/crfasrnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247608153,"owners_count":20965952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-30T22:01:28.146Z","updated_at":"2025-04-07T07:12:16.447Z","avatar_url":"https://github.com/torrvision.png","language":"MATLAB","funding_links":[],"categories":["TODO scan for Android support in followings","MATLAB","Matlab","\u003ca name=\"Vision\"\u003e\u003c/a\u003e2. Vision"],"sub_categories":[],"readme":"# CRF-RNN for Semantic Image Segmentation\n\n\u003cb\u003eLive demo:\u003c/b\u003e \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; [http://crfasrnn.torr.vision](http://crfasrnn.torr.vision) \u003cbr/\u003e\n\u003cb\u003ePyTorch version:\u003c/b\u003e \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;[http://github.com/sadeepj/crfasrnn_pytorch](http://github.com/sadeepj/crfasrnn_pytorch)\u003cbr/\u003e\n\u003cb\u003eTensorflow/Keras version:\u003c/b\u003e [http://github.com/sadeepj/crfasrnn_keras](http://github.com/sadeepj/crfasrnn_keras)\u003cbr/\u003e\n\n![sample](sample.png)\n\n[![License (3-Clause BSD)](https://img.shields.io/badge/license-BSD%203--Clause-brightgreen.svg?style=flat-square)](https://github.com/torrvision/crfasrnn/blob/master/LICENSE)\n\n\n\nThis package contains code for the \"CRF-RNN\" semantic image segmentation method, published in the ICCV 2015 paper [Conditional Random Fields as Recurrent Neural Networks](http://www.robots.ox.ac.uk/~szheng/papers/CRFasRNN.pdf). This paper was initially described in an [arXiv tech report](http://arxiv.org/abs/1502.03240). The online demonstration based on this code won the Best Demo Prize at ICCV 2015. Our software is built on top of the [Caffe](http://caffe.berkeleyvision.org/) deep learning library. The current version was developed by:\n\n[Sadeep Jayasumana](http://www.robots.ox.ac.uk/~sadeep/),\n[Shuai Zheng](http://kylezheng.org/),\n[Bernardino Romera Paredes](http://romera-paredes.com/), \n[Anurag Arnab](http://www.robots.ox.ac.uk/~aarnab/),\nand\nZhizhong Su.\n\nSupervisor: [Philip Torr](http://www.robots.ox.ac.uk/~tvg/)\n\nOur work allows computers to recognize objects in images, what is distinctive about our work is that we also recover the 2D outline of objects. Currently we have trained this model to recognize 20 classes. This software allows you to test our algorithm on your own images – have a try and see if you can fool it, if you get some good examples you can send them to us.\n\nWhy are we doing this? This work is part of a project to build augmented reality glasses for the partially sighted. Please read about it here: [smart-specs](http://www.va-st.com/smart-specs/). \n\nFor demo and more information about CRF-RNN please visit the project website: \u003chttp://crfasrnn.torr.vision\u003e.\n\nIf you use this code/model for your research, please cite the following papers:\n```\n@inproceedings{crfasrnn_ICCV2015,\n    author = {Shuai Zheng and Sadeep Jayasumana and Bernardino Romera-Paredes and Vibhav Vineet and\n    Zhizhong Su and Dalong Du and Chang Huang and Philip H. S. Torr},\n    title  = {Conditional Random Fields as Recurrent Neural Networks},\n    booktitle = {International Conference on Computer Vision (ICCV)},\n    year   = {2015}\n}\n```\n```\n@inproceedings{higherordercrf_ECCV2016,\n\tauthor = {Anurag Arnab and Sadeep Jayasumana and Shuai Zheng and Philip H. S. Torr},\n\ttitle  = {Higher Order Conditional Random Fields in Deep Neural Networks},\n\tbooktitle = {European Conference on Computer Vision (ECCV)},\n\tyear   = {2016}\n}\n```\n\n\n## How to use the CRF-RNN layer\nCRF-RNN has been developed as a custom Caffe layer named MultiStageMeanfieldLayer. Usage of this layer in the model definition prototxt file looks the following. Check the `matlab-scripts` or the `python-scripts` folder for more detailed examples.\n```\n# This is part of FCN, coarse is a blob coming from FCN\nlayer { type: 'Crop' name: 'crop' bottom: 'bigscore' bottom: 'data' top: 'coarse' }\n\n# This layer is used to split the output of FCN into two. This is required by CRF-RNN.\nlayer { type: 'Split' name: 'splitting'\n  bottom: 'coarse' top: 'unary' top: 'Q0'\n}\n\nlayer {\n  name: \"inference1\" # Keep the name \"inference1\" to load the trained parameters from our caffemodel.\n  type: \"MultiStageMeanfield\" # Type of this layer\n  bottom: \"unary\" # Unary input from FCN\n  bottom: \"Q0\" # A copy of the unary input from FCN\n  bottom: \"data\" # Input image\n  top: \"pred\" # Output of CRF-RNN\n  param {\n    lr_mult: 10000 # learning rate for W_G\n  }\n  param {\n    lr_mult: 10000 # learning rate for W_B\n  }\n  param {\n    lr_mult: 1000 # learning rate for compatiblity transform matrix\n  }\n  multi_stage_meanfield_param {\n    num_iterations: 10 # Number of iterations for CRF-RNN\n    compatibility_mode: POTTS # Initialize the compatilibity transform matrix with a matrix whose diagonal is -1.\n    threshold: 2\n    theta_alpha: 160\n    theta_beta: 3\n    theta_gamma: 3\n    spatial_filter_weight: 3\n    bilateral_filter_weight: 5\n  }\n}\n```\n## Installation Guide\nFirst, clone the project by running:\n```\ngit clone --recursive https://github.com/torrvision/crfasrnn.git\n```\n\nYou need to compile the modified Caffe library in this repository. Instructions for Ubuntu 14.04 are included below. You can also consult the generic [Caffe installation guide](http://caffe.berkeleyvision.org/installation.html) for further help.\n\n\n### 1.1 Install dependencies\n##### General dependencies\n```\nsudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler\nsudo apt-get install --no-install-recommends libboost-all-dev\n```\n\n##### CUDA (optional - needed only if you are planning to use a GPU for faster processing)\nInstall the correct CUDA driver and its SDK. Download CUDA SDK from Nvidia website. \n\nYou might need to blacklist some modules so that they do not interfere with the driver installation. You also need to uninstall your default Nvidia Driver first.\n```\nsudo apt-get install freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa libglu1-mesa-dev\n``` \nopen `/etc/modprobe.d/blacklist.conf` and add:\n```\nblacklist amd76x_edac\nblacklist vga16fb\nblacklist nouveau\nblacklist rivafb\nblacklist nvidiafb\nblacklist rivatv\n```\n```\nsudo apt-get remove --purge nvidia*\n```\n\nWhen you restart your PC, before logging in, try \"Ctrl + Alt + F1\" to switch to a text-based login. Try:\n```\nsudo service lightdm stop\nchmod +x cuda*.run\nsudo ./cuda*.run\n```\n\n##### BLAS\nInstall a BLAS library such as ATLAS, OpenBLAS or MKL. To install BLAS:\n```\nsudo apt-get install libatlas-base-dev \n```\n\n##### Python \nInstall Anaconda Python distribution or install the default Python distribution with numpy, scipy, etc.\n\n##### MATLAB (optional - needed only if you are planning to use the MATLAB interface)\nInstall MATLAB using a standard distribution.\n\n### 1.2 Build the custom Caffe version\nSet the path correctly in the ``Makefile.config``. You can rename the ``Makefile.config.example`` to ``Makefile.config``, as most common parts are filled already. You may need to change it a bit according to your environment.\n\nAfter this, in Ubuntu 14.04, try:\n```\nmake\n```\n\nIf there are no error messages, you can then compile and install the Python and Matlab wrappers:\nTo install the MATLAB wrapper (optional):\n```\nmake matcaffe\n```\n\nTo install the Python wrapper (optional):\n```\nmake pycaffe\n```\n\nThat's it! Enjoy our software!\n\n\n### 1.3 Run the demo\nMATLAB and Python scripts for running the demo are available in the ``matlab-scripts`` and ``python-scripts`` directories, respectively. Both of these scripts do the same thing - you can choose either.\n\n#### Python users:\nChange the directory to ``python-scripts``. First download the model that includes the trained weights. In Linux, this can be done by:\n```\nsh download_trained_model.sh\n```\nAlternatively, you can also get the model by directly clicking the link in ``python-scripts/README.md``.\n\nTo run the demo, execute:\n```\npython crfasrnn_demo.py\n```\nYou will get an output.png image.\n\nTo use your own images, replace \"input.jpg\" in the ``crfasrnn_demo.py`` file.\n\n#### MATLAB users:\nChange the directory to ``matlab-scripts``. First download the model that includes the trained weights. In Linux, this can be done by:\n```\nsh download_trained_model.sh\n```\nAlternatively, you can also get the model by directly clicking the link in ``matlab-scripts/README.md``.\n\nLoad your MATLAB application and run ``crfrnn_demo.m``.\n\nTo use your own images, just replace \"input.jpg\" in the ``crfrnn_demo.m`` file.\n\nYou can also find a part of our model in [MatConvNet](http://www.vlfeat.org/matconvnet/pretrained/).\n\n\n#### Explanation about the CRF-RNN layer:\nIf you would like to try out the CRF-RNN model we trained, you should keep the layer name as it is (\"inference1\"), so that the code will correctly load the parameters from the caffemodel. Otherwise, it will reinitialize parameters.\n\nYou should find out that the end-to-end trained CRF-RNN model does better than the alternatives. If you set the CRF-RNN layer name to \"inference2\", you should observe lower performance since the parameters for both CNN and CRF are not jointly optimized.\n\n\n#### Training CRF-RNN on a new dataset:\nIf you would like to train CRF-RNN on other datasets, please follow the piecewise training described in our paper. In short, you should first train a strong pixel-wise CNN model. After this, you could plug our CRF-RNN layer into it by adding the MultiStageMeanfieldLayer to the prototxt file. You should then be able to train the CNN and CRF-RNN parts jointly end-to-end.\n\nNotice that the current deploy.prototxt file we have provided is tailored for PASCAL VOC Challenge. This dataset contains 21 class labels including background. You should change the num_output in the corresponding layer if you would like to finetune our model for other datasets. Also, the deconvolution layer in current code does not allow initializing the parameters through prototxt. If you change the num_output there, you should manually re-initialize the parameters in the caffemodel file.\n\nSee ``examples/segmentationcrfasrnn`` for more information.\n\n\n#### Why predictions are all black?\nThis could happen if you change layer names in the model definition prototxt, causing the weights not to load correctly. This could also happen if you change the number of outputs in deconvolution layer in the prototxt but not initialize the deconvolution layer properly. \n\n#### MultiStageMeanfield causes a segfault?\nThis error usually occurs when you do not place the ``spatial.par`` and ``bilateral.par`` files in the script path.\n\n#### Python training script from third parties\nWe would like to thank martinkersner and MasazI for providing Python training scripts for CRF-RNN. \n\n1. [martinkersner's scripts](https://github.com/martinkersner/train-CRF-RNN)\n2. [MasazI's scripts](https://github.com/MasazI/crfasrnn-training)\n\n#### Merge with the upstream caffe\nIt is possible to integrate the CRF-RNN code into upstream Caffe. However, due to the change of the crop layer, the caffemodel we provided might require extra training to provide the same accuracy. mtourne kindly provided a version that merged the code with upstream caffe. \n\n1. [mtourne upstream version with CRFRNN](https://github.com/mtourne/crfasrnn)\n\n#### GPU version of CRF-RNN\nhyenal kindly provided a purely GPU version of CRF-RNN. This would lead to considerably faster training and testing.\n\n1. [hyenal's GPU crf-rnn](https://github.com/hyenal/crfasrnn)\n\n#### CRF-as-RNN as a layer in Lasagne\n[Lasagne CRFasRNN layer](https://github.com/hapemask/crfrnn_layer)\n\n#### Latest Caffe with CPU/GPU CRF-RNN\n[crfasrnn-caffe](https://github.com/torrvision/caffe/tree/crfrnn)\n\n#### Keras/Tensorflow version of CRF-RNN\n[crfasrnn_keras](https://github.com/sadeepj/crfasrnn_keras)\n\nLet us know if we have missed any other works from third parties.\n\n\nFor more information about CRF-RNN please visit the project website http://crfasrnn.torr.vision. Contact: \u003ccrfasrnn@gmail.com\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftorrvision%2Fcrfasrnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftorrvision%2Fcrfasrnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftorrvision%2Fcrfasrnn/lists"}