{"id":17717417,"url":"https://github.com/wgcban/hypertransformer","last_synced_at":"2025-04-16T14:26:38.642Z","repository":{"id":43344014,"uuid":"431189304","full_name":"wgcban/HyperTransformer","owner":"wgcban","description":"[CVPR'22] HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening","archived":false,"fork":false,"pushed_at":"2023-10-19T13:52:29.000Z","size":20218,"stargazers_count":102,"open_issues_count":10,"forks_count":12,"subscribers_count":1,"default_branch":"main","last_synced_at":"2023-10-19T14:46:50.063Z","etag":null,"topics":["attention-mechanism","deep-learning","hyperspectral-imaging","image-fusion","multispectral-images","pansharpening","super-resolution","transformers"],"latest_commit_sha":null,"homepage":"https://www.wgcban.com/research#h.ar24vwqlm021","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wgcban.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-23T17:15:18.000Z","updated_at":"2023-10-19T02:54:35.000Z","dependencies_parsed_at":"2024-10-25T23:00:01.336Z","dependency_job_id":null,"html_url":"https://github.com/wgcban/HyperTransformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2FHyperTransformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2FHyperTransformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2FHyperTransformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wgcban%2FHyperTransformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wgcban","download_url":"https://codeload.github.com/wgcban/HyperTransformer/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249250356,"owners_count":21237884,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["attention-mechanism","deep-learning","hyperspectral-imaging","image-fusion","multispectral-images","pansharpening","super-resolution","transformers"],"created_at":"2024-10-25T14:20:40.077Z","updated_at":"2025-04-16T14:26:38.624Z","avatar_url":"https://github.com/wgcban.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening (CVPR'22)\n\n[`Wele Gedara Chaminda Bandara`](https://www.wgcban.com/), and [`Vishal M. Patel`](https://engineering.jhu.edu/vpatel36/sciencex_teams/vishalpatel/)\n\n\nFor more information, please see our \n- **Paper**: [`CVPR-2022-Open-Access`](https://openaccess.thecvf.com/content/CVPR2022/html/Bandara_HyperTransformer_A_Textural_and_Spectral_Feature_Fusion_Transformer_for_Pansharpening_CVPR_2022_paper.html) or [`arxiv`](https://arxiv.org/abs/2203.02503).\n- **Poster**: [`view here`](https://www.dropbox.com/s/6gw64uo2j327yp1/poster.pdf?dl=0)\n- **Video Presentation**: [`view here`](https://youtu.be/x-jLi7ki0Sw)\n- **Presentation Slides**: [`download here`](https://www.dropbox.com/s/odki2ikymkoh85r/Presentation.pptx?dl=0)\n\n## Summary\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"/imgs/poster.jpg\" /\u003e\n\u003c/p\u003e\n\n\n# Setting up a virtual conda environment\nSetup a virtual conda environment using the provided ``environment.yml`` file or ``requirements.txt``.\n```\nconda env create --name HyperTransformer --file environment.yaml\nconda activate HyperTransformer\n```\nor\n```\nconda create --name HyperTransformer --file requirements.txt\nconda activate HyperTransformer\n```\n\n# Download datasets\n\nWe use three publically available HSI datasets for experiments, namely\n\n1) **Pavia Center scene** [`Download the .mat file here`](https://www.dropbox.com/s/znykgpipttircdr/Pavia_centre.mat?dl=0), and save it in \"./datasets/pavia_centre/Pavia_centre.mat\".\n2) **Botswana dataset**[`Download the .mat file here`](https://www.dropbox.com/s/w5bie03gaeuu6t9/Botswana.mat?dl=0), and save it in \"./datasets/botswana4/Botswana.mat\".\n3) **Chikusei dataset** [`Download the .mat file here`](https://naotoyokoya.com/Download.html), and save it in \"./datasets/chikusei/chikusei.mat\".\n\n# Processing the datasets to generate LR-HSI, PAN, and Reference-HR-HSI using Wald's protocol\n We use Wald's protocol to generate LR-HSI and PAN image. To generate those cubic patches,\n  1) Run `process_pavia.m` in `./datasets/pavia_centre/` to generate cubic patches. \n  2) Run `process_botswana.m` in `./datasets/botswana4/` to generate cubic patches.\n  3) Run `process_chikusei.m` in `./datasets/chikusei/` to generate cubic patches.\n \n# Training HyperTransformer \nWe use two stage procedure to train our HyperTransformer. \n\nWe first train the backbone of HyperTrasnformer and then fine-tune the MHFA modules. This way we get better results and faster convergence instead of training whole network at once.\n\n## Training the Backbone of HyperTrasnformer\nUse the following codes to pre-train HyperTransformer on the three datasets.\n 1) Pre-training on Pavia Center Dataset: \n    \n    Change \"train_dataset\" to \"pavia_dataset\" in config_HSIT_PRE.json. \n    \n    Then use following commad to pre-train on Pavia Center dataset.\n    `python train.py --config configs/config_HSIT_PRE.json`.\n    \n 4) Pre-training on Botswana Dataset:\n     Change \"train_dataset\" to \"botswana4_dataset\" in config_HSIT_PRE.json. \n     \n     Then use following commad to pre-train on Pavia Center dataset. \n     `python train.py --config configs/config_HSIT_PRE.json`.\n     \n 6) Pre-training on Chikusei Dataset: \n     \n     Change \"train_dataset\" to \"chikusei_dataset\" in config_HSIT_PRE.json. \n     \n     Then use following commad to pre-train on Pavia Center dataset. \n     `python train.py --config configs/config_HSIT_PRE.json`.\n     \n\n## Fine-tuning the MHFA modules in HyperTrasnformer\nNext, we fine-tune the MHFA modules in HyperTransformer starting from pre-trained backbone from the previous step.\n 1) Fine-tuning MHFA on Pavia Center Dataset: \n\n    Change \"train_dataset\" to \"pavia_dataset\" in config_HSIT.json. \n    \n    Then use the following commad to train HyperTransformer on Pavia Center dataset. \n    \n    Please specify path to best model obtained from previous step using --resume.\n    `python train.py --config configs/config_HSIT.json --resume ./Experiments/HSIT_PRE/pavia_dataset/N_modules\\(4\\)/best_model.pth`.\n   \n 3) Fine-tuning on Botswana Dataset: \n\n    Change \"train_dataset\" to \"botswana4_dataset\" in config_HSIT.json. \n    \n    Then use following commad to pre-train on Pavia Center dataset. \n    \n    `python train.py --config configs/config_HSIT.json --resume ./Experiments/HSIT_PRE/botswana4/N_modules\\(4\\)/best_model.pth`.\n\n 5) Fine-tuning on Chikusei Dataset: \n\n    Change \"train_dataset\" to \"chikusei_dataset\" in config_HSIT.json.\n    \n    Then use following commad to pre-train on Pavia Center dataset. \n    \n    `python train.py --config configs/config_HSIT.json --resume ./Experiments/HSIT_PRE/chikusei_dataset/N_modules\\(4\\)/best_model.pth`.\n    \n# Trained models and pansharpened results on test-set\nYou can download trained models and final prediction outputs through the follwing links for each dataset.\n  1) Pavia Center: [`Download here`](https://www.dropbox.com/sh/9zg0wrbq6fzx1wa/AACH3mnRlqkVFmo6BF4wcDdaa?dl=0)\n  2) Botswana: [`Download here`](https://www.dropbox.com/sh/e7og46hkn3wuaxr/AACrFOpOSFF2u0hG1CzNYVRxa?dl=0)\n  3) Chikusei: [`Download here`](https://www.dropbox.com/sh/l6gaf723cb6asq4/AABPBUleyZ7aFX8POh_d5jC9a?dl=0)\n\n# Citation\nIf you find our work useful, please consider citing our paper.\n```\n@InProceedings{Bandara_2022_CVPR,\n    author    = {Bandara, Wele Gedara Chaminda and Patel, Vishal M.},\n    title     = {HyperTransformer: A Textural and Spectral Feature Fusion Transformer for Pansharpening},\n    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},\n    month     = {June},\n    year      = {2022},\n    pages     = {1767-1777}\n}\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwgcban%2Fhypertransformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwgcban%2Fhypertransformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwgcban%2Fhypertransformer/lists"}