{"id":24446253,"url":"https://github.com/fabio-sim/dedode-onnx-tensorrt","last_synced_at":"2025-05-07T17:07:40.678Z","repository":{"id":189036689,"uuid":"679894377","full_name":"fabio-sim/DeDoDe-ONNX-TensorRT","owner":"fabio-sim","description":"ONNX-compatible DeDoDe 🎶 Detect, Don't Describe - Describe, Don't Detect, for Local Feature Matching. Supports TensorRT 🚀","archived":false,"fork":false,"pushed_at":"2023-08-21T16:14:00.000Z","size":3041,"stargazers_count":69,"open_issues_count":4,"forks_count":4,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-01-20T23:52:32.870Z","etag":null,"topics":["dedode","deep-learning","feature-extraction","feature-matching","homography","homography-estimation","local-feature-matching","local-features","machine-learning","onnx","onnxruntime","openvino","pose-estimation","pytorch","tensorrt","visual-localization"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/fabio-sim.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-08-17T21:34:54.000Z","updated_at":"2025-01-15T09:12:32.000Z","dependencies_parsed_at":"2023-08-18T00:56:15.522Z","dependency_job_id":"e5121f31-fcbd-4a03-96ad-6c4a3559dad5","html_url":"https://github.com/fabio-sim/DeDoDe-ONNX-TensorRT","commit_stats":null,"previous_names":["fabio-sim/dedode-onnx-tensorrt"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabio-sim%2FDeDoDe-ONNX-TensorRT","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabio-sim%2FDeDoDe-ONNX-TensorRT/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabio-sim%2FDeDoDe-ONNX-TensorRT/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/fabio-sim%2FDeDoDe-ONNX-TensorRT/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/fabio-sim","download_url":"https://codeload.github.com/fabio-sim/DeDoDe-ONNX-TensorRT/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":235532531,"owners_count":19005140,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dedode","deep-learning","feature-extraction","feature-matching","homography","homography-estimation","local-feature-matching","local-features","machine-learning","onnx","onnxruntime","openvino","pose-estimation","pytorch","tensorrt","visual-localization"],"created_at":"2025-01-20T23:50:15.072Z","updated_at":"2025-01-25T03:21:42.749Z","avatar_url":"https://github.com/fabio-sim.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![GitHub](https://img.shields.io/github/license/fabio-sim/DeDoDe-ONNX-TensorRT)](/LICENSE)\n[![ONNX](https://img.shields.io/badge/ONNX-grey)](https://onnx.ai/)\n[![TensorRT](https://img.shields.io/badge/TensorRT-76B900)](https://developer.nvidia.com/tensorrt)\n[![GitHub Repo stars](https://img.shields.io/github/stars/fabio-sim/DeDoDe-ONNX-TensorRT)](https://github.com/fabio-sim/DeDoDe-ONNX-TensorRT/stargazers)\n[![GitHub all releases](https://img.shields.io/github/downloads/fabio-sim/DeDoDe-ONNX-TensorRT/total)](https://github.com/fabio-sim/DeDoDe-ONNX-TensorRT/releases)\n\n# DeDoDe-ONNX-TensorRT\nOpen Neural Network Exchange (ONNX) compatible implementation of [DeDoDe 🎶 Detect, Don't Describe - Describe, Don't Detect, for Local Feature Matching](https://github.com/Parskatt/DeDoDe). Supports TensorRT 🚀.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"assets/matches.jpg\" alt=\"DeDoDe figure\" width=80%\u003e\u003cbr\u003e\u003cem\u003eThe DeDoDe detector learns to detect 3D consistent repeatable keypoints, which the DeDoDe descriptor learns to match. The result is a powerful decoupled local feature matcher.\u003c/em\u003e\u003c/p\u003e\n\n\u003cp align=\"center\"\u003e\u003ca href=\"#%EF%B8%8F-inference-time-comparison\"\u003e\u003cimg src=\"assets/latency.png\" alt=\"Latency figure\" width=80%\u003e\u003c/a\u003e\u003cbr\u003e\u003cem\u003eDeDoDe ONNX TensorRT provides a 2x speedup over PyTorch.\u003c/em\u003e\u003c/p\u003e\n\n## 🔥 ONNX Export\n\nPrior to exporting the ONNX models, please install the [requirements](/requirements.txt).\n\nTo convert the DeDoDe models to ONNX, run [`export.py`](/export.py). We provide two types of ONNX exports: individual standalone models, and a combined end-to-end pipeline (recommended for convenience) with the `--end2end` flag.\n\n\u003cdetails\u003e\n\u003csummary\u003eExport Example\u003c/summary\u003e\n\u003cpre\u003e\npython export.py \\\n    --img_size 256 256 \\\n    --end2end \\\n    --dynamic_img_size --dynamic_batch \\\n    --fp16\n\u003c/pre\u003e\n\u003c/details\u003e\n\nIf you would like to try out inference right away, you can download ONNX models that have already been exported [here](https://github.com/fabio-sim/DeDoDe-ONNX-TensorRT/releases) or run `./weights/download.sh`.\n\n## ⚡ ONNX Inference\n\nWith ONNX models in hand, one can perform inference on Python using ONNX Runtime (see [requirements-onnx.txt](/requirements-onnx.txt)).\n\nThe DeDoDe inference pipeline has been encapsulated into a runner class:\n\n```python\nfrom onnx_runner import DeDoDeRunner\n\nimages = DeDoDeRunner.preprocess(image_array)\n# images.shape == (2B, 3, H, W)\n\n# Create ONNXRuntime runner\nrunner = DeDoDeRunner(\n    end2end_path=\"weights/dedode_end2end_1024.onnx\",\n    providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"],\n    # TensorrtExecutionProvider\n)\n\n# Run inference\nmatches_A, matches_B, batch_ids = runner.run(images)\n\nmatches_A = DeDoDeRunner.postprocess(matches_A, H_A, W_A)\nmatches_B = DeDoDeRunner.postprocess(matches_B, H_B, W_B)\n```\nAlternatively, you can also run [`infer.py`](/infer.py).\n\n\u003cdetails\u003e\n\u003csummary\u003eInference Example\u003c/summary\u003e\n\u003cpre\u003e\npython infer.py \\\n    --img_paths assets/im_A.jpg assets/im_B.jpg \\\n    --img_size 256 256 \\\n    --end2end \\\n    --end2end_path weights/dedode_end2end_1024_fp16.onnx \\\n    --fp16 \\\n    --viz\n\u003c/pre\u003e\n\u003c/details\u003e\n\n## 🚀 TensorRT Support\n\nTensorRT offers the best performance and greatest memory efficiency.\n\nTensorRT inference is supported for the end-to-end model via the TensorRT Execution Provider in ONNXRuntime. Please follow the [official documentation](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html) to install TensorRT. The exported ONNX models must undergo [shape inference](/tools/symbolic_shape_infer.py) for compatibility with TensorRT.\n\n\u003cdetails\u003e\n\u003csummary\u003eTensorRT Example\u003c/summary\u003e\n\u003cpre\u003e\npython tools/symbolic_shape_infer.py \\\n  --input weights/dedode_end2end_1024.onnx \\\n  --output weights/dedode_end2end_1024_trt.onnx \\\n  --auto_merge\u003cbr\u003e\nCUDA_MODULE_LOADING=LAZY \u0026\u0026 python infer.py \\\n  --img_paths assets/DSC_0410.JPG assets/DSC_0411.JPG \\\n  --img_size 256 256 \\\n  --end2end \\\n  --end2end_path weights/dedode_end2end_1024_trt.onnx \\\n  --trt \\\n  --viz\n\u003c/pre\u003e\n\u003c/details\u003e\n\nThe first run will take longer because TensorRT needs to initialise the `.engine` and `.profile` files. Subsequent runs should use the cached files. Only static input shapes are supported. Note that TensorRT will rebuild the cache if it encounters a different input shape.\n\n## ⏱️ Inference Time Comparison\n\nThe inference times of the end-to-end DeDoDe pipelines are shown below.\n\n\u003ctable align=\"center\"\u003e\u003cthead\u003e\u003ctr\u003e\u003cth\u003e# Keypoints\u003c/th\u003e\u003cth\u003e1024\u003c/th\u003e\u003cth\u003e2048\u003c/th\u003e\u003cth\u003e3840\u003c/th\u003e\u003cth\u003e4096\u003c/th\u003e\u003cth\u003e8192\u003c/th\u003e\u003c/tr\u003e\u003ctr\u003e\u003cth\u003e\u003c/th\u003e\u003cth colspan=\"5\"\u003eLatency (ms) (RTX 4080 12GB)\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd\u003ePyTorch\u003c/td\u003e\u003ctd\u003e169.72\u003c/td\u003e\u003ctd\u003e170.42\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003ctd\u003e176.18\u003c/td\u003e\u003ctd\u003e189.53\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003ePyTorch-MP\u003c/td\u003e\u003ctd\u003e79.42\u003c/td\u003e\u003ctd\u003e80.09\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003ctd\u003e83.8\u003c/td\u003e\u003ctd\u003e96.93\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eONNX\u003c/td\u003e\u003ctd\u003e170.84\u003c/td\u003e\u003ctd\u003e171.83\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003ctd\u003e180.18\u003c/td\u003e\u003ctd\u003e203.37\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eTensorRT\u003c/td\u003e\u003ctd\u003e78.12\u003c/td\u003e\u003ctd\u003e79.59\u003c/td\u003e\u003ctd\u003e94.88\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eTensorRT-FP16\u003c/td\u003e\u003ctd\u003e33.9\u003c/td\u003e\u003ctd\u003e35.45\u003c/td\u003e\u003ctd\u003e42.35\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003ctd\u003eN/A\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003eEvaluation Details\u003c/summary\u003e\nThe inference time, or latency, of only the end-to-end DeDoDe pipeline is reported; that is, the time taken for image preprocessing, postprocessing, copying data between the host \u0026 device, or finding inliers (e.g., CONSAC/MAGSAC) is not measured. The inference time is defined as the median over all samples in the \u003ca href=\"https://arxiv.org/abs/1804.00607\"\u003eMegaDepth\u003c/a\u003e test dataset. We use the data provided by \u003ca href=\"https://arxiv.org/abs/2104.00680\"\u003eLoFTR\u003c/a\u003e \u003ca href=\"https://github.com/zju3dv/LoFTR/blob/master/docs/TRAINING.md\"\u003ehere\u003c/a\u003e - a total of 403 image pairs.\n\n\u003cbr\u003eEach image is resized such that its dimensions are `512x512` before being fed into the pipeline. The inference time of the DeDoDe pipeline is then measured for different values of the detector's `num_keypoints` parameter: 1024, 2048, 4096, and 8192. Note that TensorRT has a \u003ca href=\"https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/infer/Graph/Layers.html?highlight=resizecoordinatetransformation#tensorrt.ITopKLayer\"\u003ehard limit\u003c/a\u003e of 3840 keypoints.\n\nFor reproducibility, the evaluation script \u003ca href=\"eval.py\"\u003e`eval.py`\u003c/a\u003e is provided.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"assets/latency.png\" alt=\"Latency figure\" width=80%\u003e\u003c/p\u003e\n\u003c/details\u003e\n\n## Credits\nIf you use any ideas from the papers or code in this repo, please consider citing the authors of [DeDoDe](https://arxiv.org/abs/2308.08479). Lastly, if the ONNX or TensorRT versions helped you in any way, please also consider starring this repository.\n\n```txt\n@article{edstedt2023dedode,\n      title={DeDoDe: Detect, Don't Describe -- Describe, Don't Detect for Local Feature Matching}, \n      author={Johan Edstedt and Georg Bökman and Mårten Wadenbäck and Michael Felsberg},\n      year={2023},\n      eprint={2308.08479},\n      archivePrefix={arXiv},\n      primaryClass={cs.CV}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabio-sim%2Fdedode-onnx-tensorrt","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffabio-sim%2Fdedode-onnx-tensorrt","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffabio-sim%2Fdedode-onnx-tensorrt/lists"}