{"id":33084639,"url":"https://github.com/tynguyen/unsupervisedDeepHomographyRAL2018","last_synced_at":"2025-11-19T07:02:04.500Z","repository":{"id":72668202,"uuid":"121344641","full_name":"tynguyen/unsupervisedDeepHomographyRAL2018","owner":"tynguyen","description":"Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model","archived":false,"fork":false,"pushed_at":"2018-04-23T22:58:47.000Z","size":240,"stargazers_count":271,"open_issues_count":3,"forks_count":81,"subscribers_count":12,"default_branch":"master","last_synced_at":"2024-06-15T01:49:49.911Z","etag":null,"topics":["deep","homography","robotics","tensorflow","training","unsupervised"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/1709.03966","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tynguyen.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2018-02-13T05:55:40.000Z","updated_at":"2024-06-08T16:13:41.000Z","dependencies_parsed_at":"2023-09-20T03:15:28.683Z","dependency_job_id":null,"html_url":"https://github.com/tynguyen/unsupervisedDeepHomographyRAL2018","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tynguyen/unsupervisedDeepHomographyRAL2018","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tynguyen%2FunsupervisedDeepHomographyRAL2018","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tynguyen%2FunsupervisedDeepHomographyRAL2018/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tynguyen%2FunsupervisedDeepHomographyRAL2018/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tynguyen%2FunsupervisedDeepHomographyRAL2018/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tynguyen","download_url":"https://codeload.github.com/tynguyen/unsupervisedDeepHomographyRAL2018/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tynguyen%2FunsupervisedDeepHomographyRAL2018/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":285200709,"owners_count":27131411,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-19T02:00:05.673Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep","homography","robotics","tensorflow","training","unsupervised"],"created_at":"2025-11-14T13:00:37.197Z","updated_at":"2025-11-19T07:02:04.488Z","avatar_url":"https://github.com/tynguyen.png","language":"Python","funding_links":[],"categories":["Image Registration"],"sub_categories":["Homography"],"readme":"#  Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model \n[This paper](https://arxiv.org/abs/1709.03966) will be presented in International Conference on Robotics and Automation (ICRA) 2018 (Brisbane, Australia) and appear in proceedings of IEEE Robotics and Automation Letters.\n\nWe devise an unsupervised learning algorithm that trains a Deep Convolutional Neural Network to estimate planar homographies. We compare the proposed algorithm to traditional feature-based and direct methods, as well as a corresponding supervised learning algorithm. Our empirical results demonstrate that compared to traditional approaches, the unsupervised algorithm achieves faster inference speed, while maintaining comparable or better accuracy and robustness to illumination variation. In addition, on both a synthetic dataset and representative real-world aerial dataset, our unsupervised method has superior adaptability and performance compared to the supervised deep learning method.\n\n\u003c!-- ![Teaser](teaser.JPG) --\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"400\" height=\"300\" src=\"teaser.JPG\"\u003e\n\u003c/p\u003e\n\n## Citation\nIf you use this code for research please cite:\n   \n    @InProceedings{nguyen2017unsupervised,\n      title={Unsupervised Deep Homography: A Fast and Robust Homography Estimation Model},\n      author={Nguyen, Ty and Chen, Steven W and Shivakumar, Shreyas S and Taylor, Camillo J and Kumar, Vijay},\n      booktitle={RA-L},\n      pages={},\n      year={2018},\n      organization={IEEE}\n      month        = \" \",\n      year         = \"2018\",\n      url          = \"https://arxiv.org/abs/1709.03966\"\n    }\n\n \n## Installation  \nBuilding and using requires the following libraries and programs\\\n    cuda 8.0.61 (required for gpu support) \\\n    python 2.7.12 \\\n    tensorflow 1.2.1 (or higher)\\\n    opencv 3.4.0 (can be installed using: pip install opencv-python )\\\n\nWe built our system on ubuntu 16.04. Tensorflow (CPU) and Tensorflow (GPU) can both work well; they are installed in virtualenv. Other methods to install tensorflow have not been tested. \n \n\nInstall required python packages (pip is required)\n```bash \nsource virtualenv_name/bin/activate \npip install -r requirements.txt \n```  \n\n## Build instructions\n### Clone repo \n```bash\ngit clone https://github.com/tynguyen/unsupervisedDeepHomographyRAL2018.git\n```\n\n## Trained Models \n### Model trained on Synthetic Data\nDownload at \n```bash \nhttps://drive.google.com/drive/folders/1Y9oNgbJTrAdkgf5-T1xONtU9n2ZqwDta?usp=sharing\n```\nThen, store the synthetic_models to folder models \n\n### Model trained on Aerial Image Data\n```bash \nhttps://drive.google.com/drive/folders/16RI7R0EVayiXfYoP2Ahhl4yN2sWhG76Z?usp=sharing\n```\nNote: you need to format your image data in a correct size in order to make use of this trained model. Please refer to the next sections to get how to format the raw images \n\n\n\n\n## Preparing training dataset (synthetic)\n#### Download MS-COCO dataset http://cocodataset.org/#download\nWe use 2014/Train to generate training data and 2014/Testing to generate test set. \nStore them into RAW_DATA_PATH and TEST_RAW_DATA_PATH which are repositories declared in generating synthetic data.  \n\n### Generate synthetic dataset\nIn the file code/utils/gen_synthetic_data.py, set important parameters as follows\n```bash\n  RHO = 45 # The maximum value of pertubation. The higher it is, the larger displacement between \n  # two generated images is. \n\n  DATA_NUMBER = 100000  # number of pair of synthetic images in training dataset \n  TEST_DATA_NUMBER = 5000 # number of pair of synthetic images in test dataset\n\n  IM_PER_REAL = 2 # Generate 2 different synthetic images from one single real image\n\n  # Size of synthetic image\n  HEIGHT = 240  \n  WIDTH = 320\n  # Size of crop \n  PATCH_SIZE = 128\n\n  # Directories to MS-COCO images \n  RAW_DATA_PATH = \"/Earthbyte/tynguyen/rawdata/train/\" # Real images used for generating synthetic data\n  TEST_RAW_DATA_PATH = \"/Earthbyte/tynguyen/rawdata/test/\" # Real images used for generating test synthetic data\n\n  # Synthetic data directories\n  DATA_PATH = \"/home/tynguyen/pose_estimation/data/synthetic/\" + str(RHO) + '/'\n\n  I_DIR = DATA_PATH + 'I/' # First large image in one pair \n  I_PRIME_DIR = DATA_PATH + 'I_prime/' # Second large image in one pair \n\n  # Since all generated images will be stored at the same location, we need .txt files to \n  # maintain training images and test images \n  FILENAMES_FILE = os.path.join(DATA_PATH,'train_synthetic.txt') # List of training images \n  TEST_FILENAMES_FILE = os.path.join(DATA_PATH,'test_synthetic.txt') # List of test images \n\n  GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'gt.txt') # (In training set): ground truth of homography parameters (delta movement of 4 corners)\n  PTS1_FILE = os.path.join(DATA_PATH,'pts1.txt') # (in training set): path to 4 corners on the first image \n\n  TEST_PTS1_FILE = os.path.join(DATA_PATH,'test_pts1.txt') # Test set: ground truth of homography parameters (delta movement of 4 corners)\n  TEST_GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'test_gt.txt') # Test: path to 4 corners on the first image \n\n```\n### Generate training dataset\nIt will take a few hours to generate 100.000 data samples. You can choose a smaller number of data for debugging. \n```bash\npython utils/gen_synthetic_data.py --mode train --num_data [number of data] \n```\n\n### Generate test dataset\n```bash \npython utils/gen_synthetic_data.py --mode test \n```\n\n## Debugging \nIn all training and testing processes, you can visualize images using either Tensorboard or just set --visual True in calling python functions. Tensorboard is highly recommended since it does not reduce the running speed as much as plotting using --visual flag. \nFor example\n```bash \npython homography_CNN_synthetic.py --mode train --lr 5e-4 --loss_type h_loss --visual True \n``` \n\n## Train model with synthetic dataset\nIn the file code/homography_CNN_synthetic.py, set important parameters as follows\n```bash\n  # Size of synthetic image and the pertubation range (RH0)\n  HEIGHT = 240 #\n  WIDTH = 320\n  RHO = 45 # The maximum value of pertubation. The higher it is, the larger displacement between \n  # two generated images is. Change this value to evaluate different levels of displacements \n  PATCH_SIZE = 128\n\n  # Synthetic data directories\n  DATA_PATH = \"/home/tynguyen/pose_estimation/data/synthetic/\" + str(RHO) + '/'\n\n  I_DIR = DATA_PATH + 'I/' # First large image in one pair \n  I_PRIME_DIR = DATA_PATH + 'I_prime/' # Second large image in one pair \n\n  # Since all generated images will be stored at the same location, we need .txt files to \n  # maintain training images and test images \n  FILENAMES_FILE = os.path.join(DATA_PATH,'train_synthetic.txt') # List of training images \n  TEST_FILENAMES_FILE = os.path.join(DATA_PATH,'test_synthetic.txt') # List of test images \n\n  GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'gt.txt') # (In training set): ground truth of homography parameters (delta movement of 4 corners)\n  PTS1_FILE = os.path.join(DATA_PATH,'pts1.txt') # (in training set): path to 4 corners on the first image \n\n  TEST_PTS1_FILE = os.path.join(DATA_PATH,'test_pts1.txt') # Test set: ground truth of homography parameters (delta movement of 4 corners)\n  TEST_GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'test_gt.txt') # Test: path to 4 corners on the first image \n\n  # Log and model directories\n  MAIN_LOG_PATH = '/media/tynguyen/'\n  LOG_DIR       = MAIN_LOG_PATH + \"docker_folder/pose_estimation/logs/\"\n  MODEL_DIR     = MAIN_LOG_PATH + \"docker_folder/pose_estimation/models/\"\n\n  # Where to save visualization images (for report)\n  RESULTS_DIR   = MAIN_LOG_PATH + \"docker_folder/pose_estimation/results/synthetic/report/\"\n\n  # List of augmentations to the data.\n  AUGMENT_LIST = ['normalize'] # 'normalize': standardize images\n\n```\n\n### Supervised\n```bash \npython homography_CNN_synthetic.py --mode train --lr 5e-4 --loss_type h_loss\n``` \n### Unsupervised\n```bash \npython homography_CNN_synthetic.py --mode train --lr 1e-4 --loss_type l1_loss \n``` \n\n## Test model with synthetic dataset\n### Supervised\n```bash \npython homography_CNN_synthetic.py --mode test --lr 5e-4 --loss_type h_loss \n``` \n\n### Unsupervised\n```bash \npython homography_CNN_synthetic.py --mode test --lr 1e-4 --loss_type l1_loss  \n``` \n\n# Generate aerial dataset\nDue to the company's privacy, we cannot make our aerial dataset publically available. However, there is an alternative which readers might be interested in, from: \nhttps://github.com/OpenDroneMap/OpenDroneMap/tree/master/tests/test_data\n\nThese datasets are quite similar to ours. \n\n\n### Supervised \nFor the supervised method, everything should be as same as in synthetic dataset. We use aerial images to generate synthetic images to train the model. \n\n### Unsupervised \nIn our aerial dataset, images are recorded in time sequence. Thus, we consider two consecutive images as a pair and generate some new pair of training samples (by randomly cropping). Each training sample consists of: a pair of (HEIGHT x WIDTH) images and a pair of corresponding crops. \n\nAs mentioned in the paper, from these original pair of images, we first resize from (FULL_HEIGHT x FULL_WIDTH) to (HEIGHT x WIDTH) then crop each pair of resized images at the same location (y,x). From each pair of original images, we generate IM_PER_REAL training samples by keeping y constant and romdomizing x (with max pertubation = RHO).\n\nIt is recommended that the resizing and cropping are highly dependent on the level of displacement between a pair of original images. Our aerial dataset features a large displacement so we have to keep y constant and make a large crop (PATCH_SIZE/WIDTH). However, there are still border effect during the warping: the warped crop of the second image has a black area near its edge. \nFor a better performance, ones can think of moving the crop window to the largest overlapping areas in the images other than just center-cropping.  \n\u003cp align=\"center\"\u003e\n  \u003cimg width=\"300\" height=\"200\" src=\"border_effect.JPG\"\u003e\n\u003c/p\u003e\n\n```bash\n   RHO = 24 # Maximum range of pertubation\n   DATA_NUMBER = 10000\n   TEST_DATA_NUMBER = 1000\n   IM_PER_REAL = 20 # Generate 20 different pairs of images from one single real image\n\n   # Size of synthetic image\n   HEIGHT = 142 #\n   WIDTH = 190\n   PATCH_SIZE = 128\n\n   FULL_HEIGHT = 480 #\n   FULL_WIDTH  =  640\n   # Directories to files\n   RAW_DATA_PATH = \"/Earthbyte/tynguyen/real_rawdata/joe_data/train/\" # Real images used for generating real dataset\n   TEST_RAW_DATA_PATH = \"/Earthbyte/tynguyen/real_rawdata/joe_data/test/\" # Real images used for generating real test dataset\n\n   # Data directories\n   DATA_PATH = \"/Earthbyte/tynguyen/docker_folder/pose_estimation/data/synthetic/\" + str(RHO) + '/'\n\n   I_DIR = DATA_PATH + 'I/' # Large image 240 x 320\n   I_PRIME_DIR = DATA_PATH + 'I_prime/' # Large image 240 x 320\n\n   FULL_I_DIR = DATA_PATH + 'FULL_I/' # Full image size 480 x 640\n   FULL_I_PRIME_DIR = DATA_PATH + 'FULL_I_prime/' # Full image size 480 x 640\n\n   PTS1_FILE = os.path.join(DATA_PATH,'pts1.txt')\n   FILENAMES_FILE = os.path.join(DATA_PATH,'train_real.txt')\n   GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'gt.txt')\n   TEST_PTS1_FILE = os.path.join(DATA_PATH,'test_pts1.txt')\n   TEST_FILENAMES_FILE = os.path.join(DATA_PATH,'test_real.txt')\n   # In real dataset, ground truth file consists of correspondences\n   # Each row in the file contains 8 numbers:[corr1, corr2]\n   TEST_GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'test_gt.txt')\n\n```\n\n```bash\npython utils/gen_real_data.py --mode train --num_data [number of training data] \n```\n```bash\npython utils/gen_real_data.py --mode test --num_data [number of test data] \n```\n\n## Train model with aerial dataset\nIn the file homography_CNN_real.py, set parameters as follows \n```bash\n   # Size of synthetic image and the pertubation range (RH0)\n   HEIGHT = 142 #\n   WIDTH = 190\n   RHO = 24\n   PATCH_SIZE = 128\n   # Full image size (used for displaying)\n   FULL_HEIGHT = 240 #\n   FULL_WIDTH =  320\n\n   # Data directories\n   DATA_PATH = \"/home/tynguyen/pose_estimation/data/real/\" + str(RHO) + '/'\n\n   I_DIR = DATA_PATH + 'I/' # Large image\n   I_PRIME_DIR = DATA_PATH + 'I_prime/' # Large image\n   PTS1_FILE = os.path.join(DATA_PATH,'pts1.txt')\n   FILENAMES_FILE = os.path.join(DATA_PATH,'train_real.txt')\n   GROUND_TRUTH_FILE = None # There is no ground truth during training \n\n   FULL_I_DIR = DATA_PATH + 'FULL_I/' # Large image\n   FULL_I_PRIME_DIR = DATA_PATH + 'FULL_I_prime/' # Large image\n   TEST_PTS1_FILE = os.path.join(DATA_PATH,'test_pts1.txt')\n   TEST_FILENAMES_FILE = os.path.join(DATA_PATH,'test_real.txt')\n   # Correspondences in test set\n   TEST_GROUND_TRUTH_FILE = os.path.join(DATA_PATH,'test_gt.txt')\n\n   # Log and model directories\n   MAIN_LOG_PATH = '/media/tynguyen/DATA/'\n   LOG_DIR       = MAIN_LOG_PATH + \"docker_folder/pose_estimation/logs/\"\n\n   # Where to load model. This could be the location of the model trained on synthetic data\n   # or any other dataset\n   LOAD_MODEL_DIR = MAIN_LOG_PATH + \"docker_folder/pose_estimation/models/\"\n   # Where to save new model. This is the location of the fine-tuned model\n   SAVE_MODEL_DIR = MAIN_LOG_PATH + \"docker_folder/pose_estimation/models/real_models/\"\n\n   # Where to save visualization images (for report)\n   RESULTS_DIR   = MAIN_LOG_PATH + \"docker_folder/pose_estimation/results/synthetic/report/\"\n\n   # list of augmentations to the data\n   AUGMENT_LIST = ['normalize']\n\n```\n### Supervised\nFor supervised method, after generating a new set of synthetic images using the aerial dataset, \nchange DATA_PATH in homography_CNN_synthetic.py accordingly and run \n```bash \npython homography_CNN_synthetic.py --mode train --lr 5e-4 --loss_type h_loss \n``` \n\n### Unsupervised\n```bash \npython homography_CNN_real.py --mode train --lr 1e-4 --loss_type l1_loss \n``` \nThere are a couple of options during the training. \n#### Training model from scratch \n```bash \npython homography_CNN_real.py --mode train --lr 1e-4 --loss_type l1_loss --finetune False\n``` \n#### Finetune model (after training on synthetic dataset or other datasets) \n```bash \npython homography_CNN_real.py --mode train --lr 1e-4 --loss_type l1_loss --finetune True\n``` \n#### Resume training the model (after training on aerial dataset for a while) \n```bash \npython homography_CNN_real.py --mode train --lr 1e-4 --loss_type l1_loss --resume True\n``` \n#### Resume training the model (after training on aerial dataset for a while) but reset iteration number\n```bash \npython homography_CNN_real.py --mode train --lr 1e-4 --loss_type l1_loss --resume True --retrain True \n``` \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftynguyen%2FunsupervisedDeepHomographyRAL2018","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftynguyen%2FunsupervisedDeepHomographyRAL2018","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftynguyen%2FunsupervisedDeepHomographyRAL2018/lists"}