{"id":13497370,"url":"https://github.com/jzbontar/mc-cnn","last_synced_at":"2026-01-19T07:07:45.940Z","repository":{"id":37733507,"uuid":"43210518","full_name":"jzbontar/mc-cnn","owner":"jzbontar","description":"Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches","archived":false,"fork":false,"pushed_at":"2018-01-25T21:47:39.000Z","size":1444,"stargazers_count":701,"open_issues_count":43,"forks_count":231,"subscribers_count":59,"default_branch":"master","last_synced_at":"2024-08-01T20:36:58.232Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Cuda","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jzbontar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-09-26T15:22:38.000Z","updated_at":"2024-07-23T08:05:19.000Z","dependencies_parsed_at":"2022-09-16T06:11:02.152Z","dependency_job_id":null,"html_url":"https://github.com/jzbontar/mc-cnn","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jzbontar%2Fmc-cnn","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jzbontar%2Fmc-cnn/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jzbontar%2Fmc-cnn/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jzbontar%2Fmc-cnn/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jzbontar","download_url":"https://codeload.github.com/jzbontar/mc-cnn/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":222418056,"owners_count":16981262,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T20:00:29.920Z","updated_at":"2026-01-19T07:07:45.878Z","avatar_url":"https://github.com/jzbontar.png","language":"Cuda","funding_links":[],"categories":["3DVision","Cuda","Multi-view Stereo","Multiple View Stereovision"],"sub_categories":["Depth/StereoMatching","paper","Machine Learning based MVS"],"readme":"Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches\n===================================================================================\n\nThe repository contains\n\n- procedures to compute the stereo matching cost with a convolutional neural network;\n- procedures to train a convolutional neural network on the stereo matching task; and\n- a basic stereo method (cross-based cost aggregation, semiglobal matching,\n  left-right consistency check, median filter, and bilateral filter); \n\nA NVIDIA GPU with at least 6 GB of memory is required to run on the KITTI\ndata set and 12 GB to run on the Middlebury data set. We tested the code on GTX\nTitan (KITTI only), K80, and GTX Titan X. \nThe code is released under the BSD 2-Clause license.\nPlease cite our [paper](http://arxiv.org/abs/1510.05970)\nif you use code from this repository in your work.\n\n\t@article{zbontar2016stereo,\n\t  title={Stereo matching by training a convolutional neural network to compare image patches},\n\t  author={Zbontar, Jure and LeCun, Yann},\n\t  journal={Journal of Machine Learning Research},\n\t  volume={17},\n\t  pages={1--32},\n\t  year={2016}\n\t}\n\n\nDownload trained networks\n-------------------------\n\n- [KITTI 2012 fast](https://s3.amazonaws.com/mc-cnn/net_kitti_fast_-a_train_all.t7)\n- [KITTI 2012 accurate](https://s3.amazonaws.com/mc-cnn/net_kitti_slow_-a_train_all.t7)\n- [KITTI 2015 fast](https://s3.amazonaws.com/mc-cnn/net_kitti2015_fast_-a_train_all.t7)\n- [KITTI 2015 accurate](https://s3.amazonaws.com/mc-cnn/net_kitti2015_slow_-a_train_all.t7)\n- [Middlebury fast](https://s3.amazonaws.com/mc-cnn/net_mb_fast_-a_train_all.t7)\n- [Middlebury accurate](https://s3.amazonaws.com/mc-cnn/net_mb_slow_-a_train_all.t7)\n\nCompute the Matching Cost\n-------------------------\n\nInstall [Torch](http://torch.ch/), [OpenCV 2.4](http://opencv.org/), and\n[png++](http://www.nongnu.org/pngpp/).\n\nRun the following commands in the same directory as this README file.\n\nCompile the shared libraries:\n\n\t$ cp Makefile.proto Makefile\n\t$ make\n\nThe command should produce two files: `libadcensus.so` and `libcv.so`.\n\nTo run the stereo algorithm on a stereo pair from the KITTI 2012 training set\u0026mdash;\n\n- Left input image  \n  \u003cimg src=\"samples/input/kittiL.png\" style=\"width: 25%;\"/\u003e\n- Right input image  \n  \u003cimg src=\"samples/input/kittiR.png\" style=\"width: 25%;\"/\u003e\n\n\u0026mdash;download the pretrained network and call `main.lua` with the following arguments:\n\n\t$ wget -P net/ https://s3.amazonaws.com/mc-cnn/net_kitti_fast_-a_train_all.t7\n\t$ ./main.lua kitti fast -a predict -net_fname net/net_kitti_fast_-a_train_all.t7 -left samples/input/kittiL.png -right samples/input/kittiR.png -disp_max 70\n\tWriting right.bin, 1 x 70 x 370 x 1226\n\tWriting left.bin, 1 x 70 x 370 x 1226\n\tWriting disp.bin, 1 x 1 x 370 x 1226\n\nThe first two arguments (`kitti fast`) are used to set the default\nhyperparameters of the stereo method. The outputs are stored as three binary\nfiles:\n\n- `left.bin`: The matching cost after semiglobal matching and cross-based\n  cost aggregation with the left image treated as the reference image.  \n  \u003cimg src=\"samples/output/left.png\" style=\"width: 25%;\"/\u003e\n- `right.bin`: Same as `left.bin`, but with the right image treated as the\n  reference image.  \n  \u003cimg src=\"samples/output/right.png\" style=\"width: 25%;\"/\u003e\n- `disp.bin`: The disparity map after the full stereo method.  \n  \u003cimg src=\"samples/output/disp.png\" style=\"width: 25%;\"/\u003e\n\nUse the `bin2png.lua` script to generate the `.png` images like the ones above:\n\n\t$ luajit samples/bin2png.lua \n\tWriting left.png\n\tWriting right.png\n\tWriting disp.png\n\nIf you wish to use the raw convolutional neural network outputs, that is,\nwithout applying cross-based cost aggregation and semiglobal matching, run\nthe following command:\n\n\t$ ./main.lua kitti fast -a predict -net_fname net/net_kitti_fast_-a_train_all.t7 -left samples/input/kittiL.png -right samples/input/kittiR.png -disp_max 70 -sm_terminate cnn\n\tWriting right.bin, 1 x 70 x 370 x 1226\n\tWriting left.bin, 1 x 70 x 370 x 1226\n\tWriting disp.bin, 1 x 1 x 370 x 1226\n\nThe resulting disparity maps should look like this:\n\n- `left.png`  \n  \u003cimg src=\"samples/output/left_cnn.png\" style=\"width: 25%;\"/\u003e\n- `right.png`  \n  \u003cimg src=\"samples/output/right_cnn.png\" style=\"width: 25%;\"/\u003e\n\nNote that `-disp_max 70` is used only as an example. To reproduce our\nresults on the KITTI data sets use `-disp_max 228`.\n\nSee the [predict_kitti.lua](predict_kitti.lua) script for how you might\ncall `main.lua` in a loop, for multiple image pairs.\n\n### Load the Output Binary Files ###\n\nYou can load the binary files (if, for example, you want to apply \ndifferent post-processing steps that you have written yourself) by memory \nmapping them.  We include examples of memory mapping for some of \nthe more popular programming languages.\n\n- **Lua**\n\n\t\trequire 'torch'\n\t\tleft = torch.FloatTensor(torch.FloatStorage('../left.bin')):view(1, 70, 370, 1226)\n\t\tright = torch.FloatTensor(torch.FloatStorage('../right.bin')):view(1, 70, 370, 1226)\n\t\tdisp = torch.FloatTensor(torch.FloatStorage('../disp.bin')):view(1, 1, 370, 1226)\n\n- **Python**\n\n\t\timport numpy as np\n\t\tleft = np.memmap('../left.bin', dtype=np.float32, shape=(1, 70, 370, 1226))\n\t\tright = np.memmap('../right.bin', dtype=np.float32, shape=(1, 70, 370, 1226))\n\t\tdisp = np.memmap('../disp.bin', dtype=np.float32, shape=(1, 1, 370, 1226))\n\n- **Matlab**\n\n\t\tleft = memmapfile('../left.bin', 'Format', 'single').Data;\n\t\tleft = permute(reshape(left, [1226 370 70]), [3 2 1]);\n\t\tright = memmapfile('../right.bin', 'Format', 'single').Data;\n\t\tright = permute(reshape(right, [1226 370 70]), [3 2 1]);\n\t\tdisparity = memmapfile('../disp.bin', 'Format', 'single').Data;\n\t\tdisparity = reshape(disparity, [1226 370])';\n\n- **C**\n\n\t\t#include \u003cfcntl.h\u003e\n\t\t#include \u003cstdio.h\u003e\n\t\t#include \u003csys/mman.h\u003e\n\t\t#include \u003csys/stat.h\u003e\n\t\t#include \u003csys/types.h\u003e\n\t\tint main(void)\n\t\t{\n\t\t\tint fd;\n\t\t\tfloat *left, *right, *disp;\n\t\t\tfd = open(\"../left.bin\", O_RDONLY);\n\t\t\tleft = mmap(NULL, 1 * 70 * 370 * 1226 * sizeof(float), PROT_READ, MAP_SHARED, fd, 0);\n\t\t\tclose(fd);\n\t\t\tfd = open(\"../right.bin\", O_RDONLY);\n\t\t\tright = mmap(NULL, 1 * 70 * 370 * 1226 * sizeof(float), PROT_READ, MAP_SHARED, fd, 0);\n\t\t\tclose(fd);\n\t\t\tfd = open(\"../disp.bin\", O_RDONLY);\n\t\t\tdisp = mmap(NULL, 1 * 1 * 370 * 1226 * sizeof(float), PROT_READ, MAP_SHARED, fd, 0);\n\t\t\tclose(fd);\n\t\t\treturn 0;\n\t\t}\n\nTrain\n-----\n\nThis section explains how to train the convolutional neural network on the\nKITTI and Middlebury data sets.\n\n### KITTI ###\n\nDownload both\n\n- the [KITTI 2012](http://www.cvlibs.net/download.php?file=data_stereo_flow.zip) data set and unzip it\ninto `data.kitti/unzip` (you should end up with a file `data.kitti/unzip/training/image_0/000000_10.png`) and \n- the [KITTI 2015](http://www.cvlibs.net/download.php?file=data_scene_flow.zip) data set and unzip it\ninto `data.kitti2015/unzip` (you should end up with a file `data.kitti2015/unzip/training/image_2/000000_10.png`).\n\nRun the preprocessing script:\n\n\t$ ./preprocess_kitti.lua\n\tdataset 2012\n\t1\n\t...\n\t389\n\tdataset 2015\n\t1\n\t...\n\t400\n\nRun `main.lua` to train the network:\n\n\t$ ./main.lua kitti slow -a train_tr\n\tkitti slow -a train_tr \n\tconv(in=1, out=112, k=3)\n\tcudnn.ReLU\n\tconv(in=112, out=112, k=3)\n\tcudnn.ReLU\n\tconv(in=112, out=112, k=3)\n\tcudnn.ReLU\n\tconv(in=112, out=112, k=3)\n\tcudnn.ReLU\n\tnn.Reshape(128x224)\n\tnn.Linear(224 -\u003e 384)\n\tcudnn.ReLU\n\tnn.Linear(384 -\u003e 384)\n\tcudnn.ReLU\n\tnn.Linear(384 -\u003e 384)\n\tcudnn.ReLU\n\tnn.Linear(384 -\u003e 384)\n\tcudnn.ReLU\n\tnn.Linear(384 -\u003e 1)\n\tcudnn.Sigmoid\n\t...\n\nThe network is trained on a subset of all training examples with the remaining\nexamples used for validation; to train on all examples use\n\n\t$ ./main.lua kitti slow -a train_all\n\nIn the previous command, the KITTI 2012 data set is used. If you wish to train\non the KITTI 2015 run\n\n\t$ ./main.lua kitti2015 slow -a train_tr\n\nTo train the fast architecture instead use\n\n\t$ ./main.lua kitti fast -a train_tr\n\nThe network is stored in the `net/` directory.\n\n\t$ ls net/\n\t...\n\tnet_kitti2012_fast_-action_train_tr.t7\n\t...\n\n### Middlebury ###\n\nRun `download_middlebury.sh` to download the training data\n(this can take a long time, depending on your internet connection).\n\n\t$ ./download_middlebury.sh\n\nThe data set is downloaded into the `data.mb/unzip` directory.\n\nCompile the [MiddEval3-SDK](http://vision.middlebury.edu/stereo/submit3/). You\nshould end up with the `computemask` binary in one of the directories listed in\nyour `PATH` enviromential variable.  \n\nInstall [ImageMagick](http://www.imagemagick.org/script/index.php); the\npreprocessing steps requires the `convert` binary to resize the images.\n\nRun the preprocessing script:\n\n\t$ mkdir data.mb.imperfect_gray\n\t$ ./preprocess_mb.py imperfect gray\n\tAdirondack\n\tBackpack\n\t...\n\ttestH/Staircase\n\nThe preprocessing is slow (it takes around 30 minutes) the first time it is\nrun, because the images have to be resized.\n\nUse `main.lua` to train the network:\n\n\t$ ./main.lua mb slow -a train_tr\n\nOther Useful Commands\n---------------------\n\nCompute the error rate on the validation set (useful for setting hyperparameters):\n\n\t$ ./main.lua kitti fast -a test_te -net_fname net/net_kitti_fast_-a_train_tr.t7 \n\tkitti fast -a test_te -net_fname net/net_kitti_fast_-a_train_tr.t7 \n\t0.86836290359497        0.0082842716717202\n\t...\n\t0.73244595527649        0.024202708004929\n\t0.72730183601379        0.023603160822285\n\t0.030291934952454\n\nThe validation error rate of the fast architecture on the KITTI 2012 data set is 3.029%.\n\n\\***\n\nCompute the error rate on the validation set of one dataset for a network that\nhas been trained on a different dataset.\n\n\t$ ./main.lua kitti fast -a test_te -net_fname net/net_mb_fast_-a_train_all.t7\n\tkitti fast -a test_te -net_fname net/net_mb_fast_-a_train_all.t7\n\t2.1474301815033\t0.0071447750148986\n\t...\n\t1.4276049137115\t0.024273838024622\n\t1.4282908439636\t0.01881285579564\n\t1.408842086792\t0.021741689597834\n\t0.031564540460366\n\nThe validation error rate of the fast architecture tested on KITTI 2012 but trained on\nMiddlebury is 3.156%.\n\n\\***\n\nPrepare files for submission to the KITTI and Middlebury evaluation server.\n\n\t$ ./main.lua kitti fast -a submit -net_fname net/net_kitti_fast_-a_train_all.t7 \n\tkitti fast -a submit -net_fname net/net_kitti_fast_-a_train_all.t7 \n\t  adding: 000038_10.png (deflated 0%)\n\t  adding: 000124_10.png (deflated 0%)\n\t  ...\n\t  adding: 000021_10.png (deflated 0%)\n\nThe output is stored in `out/submission.zip` and can be used to submit to the\n[KITTI evaluation\nserver](http://www.cvlibs.net/datasets/kitti/user_submit.php).\n\n\\***\n\nExperiment with different network architectures:\n\n\t$ ./main.lua kitti slow -a train_tr -l1 2 -fm 128 -l2 3 -nh2 512\n\tkitti slow -a train_tr -l1 2 -fm 128 -l2 3 -nh2 512 \n\tconv(in=1, out=128, k=3)\n\tcudnn.ReLU\n\tconv(in=128, out=128, k=3)\n\tcudnn.ReLU\n\tnn.Reshape(128x256)\n\tnn.Linear(256 -\u003e 512)\n\tcudnn.ReLU\n\tnn.Linear(512 -\u003e 512)\n\tcudnn.ReLU\n\tnn.Linear(512 -\u003e 512)\n\tcudnn.ReLU\n\tnn.Linear(512 -\u003e 1)\n\tcudnn.Sigmoid\n\t...\n\n\\***\n\nMeasure the runtime on a particular data set:\n\n\t$ ./main.lua kitti fast -a time\n\tkitti fast -a time \n\tconv(in=1, out=64, k=3)\n\tcudnn.ReLU\n\tconv(in=64, out=64, k=3)\n\tcudnn.ReLU\n\tconv(in=64, out=64, k=3)\n\tcudnn.ReLU\n\tconv(in=64, out=64, k=3)\n\tnn.Normalize2\n\tnn.StereoJoin1\n\t0.73469495773315\n\nIt take 0.73 seconds to run the fast architecure on the KITTI 2012 data set. If\nyou care only about the time spent in the neural network, you can terminate the\nstereo method early:\n\n\t$ ./main.lua kitti fast -a time -sm_terminate cnn\n\tkitti fast -a time -sm_terminate cnn \n\tconv(in=1, out=64, k=3)\n\tcudnn.ReLU\n\tconv(in=64, out=64, k=3)\n\tcudnn.ReLU\n\tconv(in=64, out=64, k=3)\n\tcudnn.ReLU\n\tconv(in=64, out=64, k=3)\n\tnn.Normalize2\n\tnn.StereoJoin1\n\t0.31126594543457\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjzbontar%2Fmc-cnn","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjzbontar%2Fmc-cnn","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjzbontar%2Fmc-cnn/lists"}