{"id":21693527,"url":"https://github.com/kleinyuan/licas3","last_synced_at":"2025-09-05T20:44:17.808Z","repository":{"id":51225550,"uuid":"439988945","full_name":"KleinYuan/LiCaS3","owner":"KleinYuan","description":"[T-RO 2022] Official Implementation for \"LiCaS3: A Simple LiDAR–Camera Self-Supervised Synchronization Method,\" in IEEE Transactions on Robotics, doi: 10.1109/TRO.2022.3167455.","archived":false,"fork":false,"pushed_at":"2024-07-29T03:43:52.000Z","size":729,"stargazers_count":88,"open_issues_count":0,"forks_count":5,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-04-06T11:11:23.582Z","etag":null,"topics":["autonomous-driving","kalibr","kitti","lidar","lidar-camera-calibration","lidar-camera-fusion","lidar-camera-synchronization","monocular-depth-estimation","newer-college","ptp","self-supervised","self-supervised-learning","ssim-loss"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/KleinYuan.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-19T23:16:27.000Z","updated_at":"2025-03-16T12:22:46.000Z","dependencies_parsed_at":"2024-11-25T18:20:57.746Z","dependency_job_id":"a673b999-42eb-4598-bdc5-63d1964dc7d6","html_url":"https://github.com/KleinYuan/LiCaS3","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/KleinYuan/LiCaS3","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KleinYuan%2FLiCaS3","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KleinYuan%2FLiCaS3/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KleinYuan%2FLiCaS3/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KleinYuan%2FLiCaS3/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/KleinYuan","download_url":"https://codeload.github.com/KleinYuan/LiCaS3/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/KleinYuan%2FLiCaS3/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273818415,"owners_count":25173825,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-05T02:00:09.113Z","response_time":402,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["autonomous-driving","kalibr","kitti","lidar","lidar-camera-calibration","lidar-camera-fusion","lidar-camera-synchronization","monocular-depth-estimation","newer-college","ptp","self-supervised","self-supervised-learning","ssim-loss"],"created_at":"2024-11-25T18:20:39.426Z","updated_at":"2025-09-05T20:44:17.744Z","avatar_url":"https://github.com/KleinYuan.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LiCaS3\n\nHere's the official implementation of the Paper:\n[K. Yuan, L. Ding, M. Abdelfattah and Z. J. Wang, \"LiCaS3: A Simple LiDAR–Camera Self-Supervised Synchronization Method,\" in IEEE Transactions on Robotics, doi: 10.1109/TRO.2022.3167455.](https://ieeexplore.ieee.org/document/9770125)\n\n![demo-8x mov](https://user-images.githubusercontent.com/8921629/167262935-107c7665-66d5-4ae0-8cef-39dde75451d1.gif)\n\n(speed x 8)\n\n## Project Walk-Thru\n\nThis project consists of four folders/parts:\n\n- [X] [training data generation / serialization](training_data_serialization)\n- [X] [training](training)\n- [X] [evaluation](evaluation)\n  - [evaluation over datasets](evaluation/evaluation_over_datasets)\n  - [evaluation with limo](evaluation/evaluation_with_limo)\n- [X] [complementary newer college datasets](complementary_newer_college_datasets)\n\nEach folder is independent so that you can run them separately. You can find the details of training data generation\nand training in this README while the evaluation has its own READMEs, since some evaluation requires quite a lot of setups.\nDemonstrative pre-trained weights are provided, and please refer to the [README.md](evaluation/README.md) in evaluation folder.\nRaw results of evaluation with limo are also provided for whoever would like to compare my results and plot it on the same \nfigure, referring to the [evaluation with limo](evaluation/evaluation_with_limo) sub-folder.\n\n\n## Hardware Requirements\n\nI conducted my experiments with the following hardware setups, which is highly recommended but not necessarily a hard\nrequirement.\n\n| Item | Version|\n|---|---|\n| System | 18.04 (x86), referring to [MACHINE-SETUP.md](https://github.com/KleinYuan/RGGNet/blob/master/MACHINE-SETUP.md) to install the system|\n| SSD | 2T |\n| GPU Memory | \u003e= 11 GB |\n\n## Main Software Requirements\n\n| Item | Version        |\n|---|----------------|\n| Python | 3.7.x          |\n| NVIDIA-Driver | 440.36         |\n| CUDA  | 10.2           |\n| cuDNN| v7.4.2         |\n| Tensorflow-GPU | \u003e=1.14  \u0026  \u003c 2 |\n| Opencv | 4.2            |\n\nI use anaconda to manage virtual environment and except for NVIDIA-Driver and CUDA, all other \ndependencies are mostly used for training data generation. All training happens inside docker\nso that the environment is transferable.\n\n## Prepare Raw Datasets\n\n\n### KITTI\n\nOK, first of all, you shall download the KITTI datasets:\n\n- [X] Navigate to training_data_serialization\n- [X] Copy [bash script](training_data_serialization/download.sh) to where you wanna download the data (1-2 TB SSD recommended)\n- [X] Run the bash `bash download.sh` to download the `sequence 26_09` to your file dir (and this may take a while, like, a really long while depending on your network speeeeeed)\n- [X] Manually split the training and testing datasets, and here's how my folder setup look like (`tree $(YOUR_FOLDER}/kitti/raw/ -L 3`):\n\nThe overall size of the raw data is fairly small, 62.3 GB.\n\n```\n├── test (8.3 GB)\n│   └── 2011_09_26\n│       ├── 2011_09_26_drive_0005_sync\n│       ├── 2011_09_26_drive_0070_sync\n│       ├── calib_cam_to_cam.txt\n│       ├── calib_imu_to_velo.txt\n│       └── calib_velo_to_cam.txt\n└── train (53 GB)\n    ├── 2011_09_26\n    │   ├── 2011_09_26_drive_0001_sync\n    │   ├── 2011_09_26_drive_0002_sync\n    │   ├── .......\n    │   ├── 2011_09_26_drive_0117_sync\n    │   ├── calib_cam_to_cam.txt\n    │   ├── calib_imu_to_velo.txt\n    │   └── calib_velo_to_cam.txt\n    └── download.sh\n```\n\n### Newer College\n\nYou will need to request the download from this link:https://ori-drs.github.io/newer-college-dataset/\n\nAnd I organize my folder like below, which is 38.3 GB in total.\n\n```\n├── raw_data\n│   └── infra1 (45, 880 items, 9.2 GB)\n    │   ├── infra1_1583836591_152386717.png\n    │   ├── ......\n│   └── ouster_scan (15, 302 items, 29.1 GB)\n    │   ├── cloud_1583836591_182590976.pcd\n    │   ├── ......\n│   └── timeoffset\n    │   ├── nc-short-time-offsets.csv\n```\n\n## Generate/Serialize Training Data\n\nNote: all below happens on your host machine instead of the docker container, mainly due to visualization requirements.\n\nDepending on the stride, window, down-sample ratio, the volume of the datasets will really vary. To be safe, I just \nrecommend you get a 1 TB SSD, which is what I used.\n\n### Code Walk-thru\n\nSince we stick to tensorflow 1.x, all training data are created before-hand as a separate process\nin tfrecord format.\n\nBelow is the tree of the subfolder `training_data_serialization`, which is all you need to create the \ntfrecords:\n\n```\ntraining_data_serialization/\n├── configs\n│   ├── kitti.yaml\n│   └── newer_college.yaml\n├── data_generation_base.py\n├── download.sh\n├── __init__.py\n├── kitti_data_generator.py\n├── Makefile\n├── newer_college_data_generator.py\n├── newer_college_labelling_data_genearator.py\n└── utils\n    ├── __init__.py\n    ├── kitti_loader.py\n    ├── projection.py\n    └── reduce_lidar_lines.py\n```\n\n\nIt simply contains 3 main parts: generator scripts, config yamls and utils. [utils](training_data_serialization/utils) is\nsimply a folder containing helper functions of projections, data loader or reduce lidar lines. We only have two configs for\nkitti and newer college respectively. Beside training data configs, a redundant sensor information is also included for \nreferences. Finally, we have two data generator, which all are inherited from the \n[Generator class in data_generator_base.py](training_data_serialization/data_generator_base.py). As you can see \nfrom the class, regardless of the logics, the main flow is just three steps: load config, generate training data, serialize\ninto tfrecords on disk. I also provided the script ([newer_college_labelling_data_generator.py](training_data_serialization/newer_college_labelling_data_genearator.py)))\nthat I used to create the annotation datasets (which I used for benchmark only). You pbbly don't need this but I still \nput it here for references.\n\n\n### KITTI\n\nFirst, take a look at the [kitti.yaml](training_data_serialization/configs/kitti.yaml):\n\nBelow are all fields that you will need to update before creating the tfrecords:\n\n```\nraw_data:\n  ...\n  root_dir: \"Replace this with your training / validation / testing raw data sub-folder\"\n\ntraining_data:\n  ...\n  output_dir: \"Replace this with where you would like to save the tfrecords\"\n  name:  \"training\" #  [\"training\", \"testing\", \"validation\"]\n  sampling_window: 5 # by default, we use a single lidar frame\n  sampling_stride: 1 # skip every every two frames\n  features:\n    X:\n      ...\n      C: 24 # (sampling_window + 1) * 4\n\n```\n\nIt shall be noted that `sampling_window` here corresponds to `l-1` in the paper. This is because computer\nstarts from 0.\n\nThe comments shall explain itself very well. After this is done, just with one command as below, you shall\nbe able to see the data generation running on multi-processors:\n\n```\n# navigate to the subfolder training_data_serialization\nmake create-kitti-data\n```\n\nUnfortunately, you will need to run training/validation/testing data generation separately. However, you can simply\nopen three terminals and update the yaml on-the-fly to do the job. \n\nYou can tweak the following based on your compute power.\n\n```\ntraining_data:\n  chunk_size: 5000\n  downsample_ratio: 0.5 # TODO: This depends on how big the machine is\n```\n\nIt will take a while to finish the training data generation. Taking sample window as 5, sampling stride as 1, \ndownsample_ratio as 0.1 as an example, on my Intel Core i9-9900K CPU (16 cores, 32 GB Memory), it will take around 15 mins\nto serialize a 6.4 GB (2 files in total) training data. \n\n### Newer College\n\nLike-wise, you will need to update similar fields in the [newer_college.yaml](training_data_serialization/configs/newer_college.yaml):\n\n```\nraw_data:\n  ...\n  root_dir: \"Replace this with your raw data folder\"\n  ...\n  time_offsets_csv_fp: \"Replace this with your timeoffset csv path\" # timeoffset/nc-short-time-offsets.csv\n  generated_fp:\n    synced_raw_data_info: \"Replace this with where you would like to save the synced raw data info\" # xxxx/synced_raw_data_info.pkl\ntraining_data:\n  output_dir: \"Replace this with where you would like to save the tfrecords\"\n  sampling_stride: 1\n  sampling_window: 5\n  features:\n    X:\n      ...\n      C: 24  # (sampling_window + 1) * 4\n```\n\nThe main difference is that you don't have to run the scripts multiple times manually for training/validation/testing since\nthe split ratio does the trick. Note: the `time_offsets_csv_fp` is only used for metrics and the baseline (not really\nground truth since it's quite poor) offset is not used as supervising signal.\n\n```\n# navigate to the subfolder training_data_serialization\nmake create-newer-college-data\n```\n\n## Training\n\nNote: all below happens inside the docker container, so that you can pbbly run it everywhere.\n\n### Build the Image and Enter a Shell\n\nBuild the docker image: \n\n```\nmake build\n```\n\nUpdate the volume mount of your training data in the [Makefile](Makefile):\n\n```\nrun:\n\tsudo docker run -it \\\n        --runtime=nvidia \\\n        --name=\"licas3-experiment\" \\\n        --net=host \\\n        --privileged=true \\\n        --ipc=host \\\n        --memory=\"10g\" \\\n        --memory-swap=\"10g\" \\\n        -v ${PWD}:/root/licas3 \\\n        -v ${PWD}/newer_college:/root/newer_college \\\n        -v ${PWD}/kitti:/root/kitti \\\n      \tlicas3-docker bash\n```\n\nI by default believe that you put newer_college and kitti data under `${PWD}`. However, if that's not true, update it.\n\nAt last, enter a shell with simply doing the following:\n\n```\nmake run\n```\n\nAnd then you shall enter a shell with exact same environment of what I was using!\n\n\n### Experimental Data Management Suggestions\n\nIn you are working on a paper based on my works, which also needs to do ablation studies with various stride/window settings,\nI highly recommend you organize your training/validation/testing datasets as follows (taking kitti as an example):\n\n```\nkitti\n├── raw\n├── training_datasets\n│   ├── sample_01_stride_05_beam_64_downsample_05/\n│                   ├── training\n│                         ├── 0_e496049da47a41ea9ebaabdbc9a2ee6f.tfrecord\n│                         ├── 1_e5fc9b31f1864279b5e2bc5ddc241347.tfrecord\n│                         ├── .....tfrecord\n│                   ├── validation\n│                   ├── testing # technically you shouldn't include this in the training process but benchmark\n│   └── sample_02_stride_05_beam_64_downsample_05/\n│   └── ..../\n```\n\n### Run Training\n\n#### Train LiCaS3 against KITTI\n\nFirstly, you shall navigate to the [training/configs](training/configs) folder \nand update the following fields:\n\n```\nname: 'licas3_kitti_sample_01_stride_05_beam_64_downsample_05' # 0. use a good name to make your experiment life easier\n\ntensors:\n  placeholders:\n    X:\n      shape: [32, 256, 24]  # 1. make sure this is consistent with your training data\n\ndata:\n  num_parallel_reads: 16\n  inputs:\n    X:\n      C: 24   # 2. make sure this is consistent with your training data and same as your placeholders\n  tfrecords_train_dirs:\n    - \"/root/kitti/training_datasets/sample_01_stride_05_beam_64_downsample_05/training\" # 3. training data location\n\n  tfrecords_test_dirs:  # Test in training means \"validation\", which you shouldn't use the real testing datasets\n    - \"/root/kitti/training_datasets/sample_01_stride_05_beam_64_downsample_05/validation\" # 4. validation data location\n\ninference:\n  included_tensor_names:\n  - 'licas3_kitti_sample_01_stride_05_beam_64_downsample_05'  # 5*. this is only used when you try to create inference graph\n\n  freeze:\n    output_node_name: 'licas3_kitti_sample_01_stride_05_beam_64_downsample_05/prediction_from_classifier' # 6*. this is only used when you try to create inference graph\n```\n\nAfter you have the correct configs, and inside the shell, you can simply do \n\n```\nmake train-licas3-kitti\n```\n\nwhich is equal to \n\n```\nexport CUDA_VISIBLE_DEVICES=0 \u0026\u0026 python commander.py train \\\n\t--model_name \"licas3\"  \\\n\t--datasets_name \"kitti\"\n```\n\nAnd then you shall see your training starts and terminal printing logs:\n\n```\nFO:tensorflow:  [Stage 1 - 1 th iteration] Train loss: 0.37371593713760376\nINFO:tensorflow:  [Stage 1 - 2 th iteration] Train loss: 0.39040398597717285\nINFO:tensorflow:  [Stage 1 - 3 th iteration] Train loss: 0.38538438081741333\n....\n```\n\nThe checkpoints will be saved to [results/save/](results/save) and log will be saved to [results/log/](results/log). You\ncan use a separate terminal to launch tensorboard to check the training!\n\nIt shall be noted that to better manage the experiments, I also append a unix-epoch timestamp at the end of the experiment\nname. So you will have everything saved as something like `licas3_kitti_sample_01_stride_05_beam_64_downsample_05_16xxxxx.xxx`.\n\n#### Train LiCaS3 against Newer College\n\nThis is nothing too different from KITTI training. The [config](training/configs/licas3_newer_college.yaml) \nis the only thing you need to update.\n\nOnce it's done, inside the shell, simply do \n\n```\nmake train-licas3-newer-college\n```\n\n#### Train Supervised Learning Models\n\nSame thing, same thing! Update the [configs/sl_newer_college.yaml](training/configs/sl_newer_college.yaml)  or \n[configs/sl_kitti.yaml](training/configs/sl_kitti.yaml). Then, inside the shell, simply do \n\n```\nmake train-sl-newer-college\n```\n\nor \n\n```\nmake train-sl-kitti\n```\n\n\n## Evaluations\n\nPlease check the evaluation folder [README.md](evaluation/README.md).\n\n\n## Citation\n\nhttps://ieeexplore.ieee.org/document/9770125\n\n```\n@ARTICLE{9770125,\n  author={Yuan, Kaiwen and Ding, Li and Abdelfattah, Mazen and Wang, Z. Jane},\n  journal={IEEE Transactions on Robotics}, \n  title={LiCaS3: A Simple LiDAR–Camera Self-Supervised Synchronization Method}, \n  year={2022},\n  volume={38},\n  number={5},\n  pages={3203-3218},\n  doi={10.1109/TRO.2022.3167455}}\n```\n\n\n## Clarification\n\nThis repo is a largely refactored open-source version based on my internal experimental repository (which is really messy) for my publications.\nIf you see potential issues/bugs or have questions regarding my works, please feel free to email me (kaiwen dot yuan1992 at gmail dot com). As I graduated, UBC widthdrew my school email kaiwen@ece.ubc.ca, which is not valid any more.\n\nThis code is largely based (not dependent but borrowed many codes) on my previous projects: [RGGNet](https://github.com/KleinYuan/RGGNet) and\n [tf-blocks](https://github.com/KleinYuan/tf-blocks), which further depends on tensorflow 1.x. Every software has a limited life. If one day you find this project cannot be pulled-and-run like 2022, it's\nnot the end of the world. The idea of this work is super simple, and here's the one-liner: `Train a mono depth estimator with mis-synched pairs within latency bound,\nto generate self-labels to train another classifier for integral offset estimation, all in a self-supervised manner`.\nI believe that who are interested can implement it in a short time.\n\nIf you are interested in collaborations with me on related topics, don't hesitate to reach out to me :)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkleinyuan%2Flicas3","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkleinyuan%2Flicas3","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkleinyuan%2Flicas3/lists"}