{"id":15968176,"url":"https://github.com/Arhosseini77/SUM","last_synced_at":"2025-10-20T12:30:49.623Z","repository":{"id":246308403,"uuid":"818557823","full_name":"Arhosseini77/SUM","owner":"Arhosseini77","description":"[WACV2025] SUM: Saliency Unification through Mamba for Visual Attention Modeling","archived":false,"fork":false,"pushed_at":"2024-12-08T07:15:31.000Z","size":8598,"stargazers_count":46,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-12-08T07:24:57.665Z","etag":null,"topics":["mamba","mamba-image","mamba-model","mamba-state-space-models","marketing","saliency","saliency-analysis","saliency-map","saliency-methods","saliency-model","saliency-prediction","vision-mamba","visual-attention","wacv","wacv2025"],"latest_commit_sha":null,"homepage":"https://arhosseini77.github.io/sum_page/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Arhosseini77.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-06-22T07:00:49.000Z","updated_at":"2024-12-08T07:15:34.000Z","dependencies_parsed_at":"2024-12-08T07:33:00.540Z","dependency_job_id":null,"html_url":"https://github.com/Arhosseini77/SUM","commit_stats":null,"previous_names":["arhosseini77/sum"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arhosseini77%2FSUM","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arhosseini77%2FSUM/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arhosseini77%2FSUM/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Arhosseini77%2FSUM/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Arhosseini77","download_url":"https://codeload.github.com/Arhosseini77/SUM/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237332624,"owners_count":19292781,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mamba","mamba-image","mamba-model","mamba-state-space-models","marketing","saliency","saliency-analysis","saliency-map","saliency-methods","saliency-model","saliency-prediction","vision-mamba","visual-attention","wacv","wacv2025"],"created_at":"2024-10-07T19:00:47.860Z","updated_at":"2025-10-20T12:30:49.617Z","avatar_url":"https://github.com/Arhosseini77.png","language":"Python","funding_links":[],"categories":["On the replacement of transformer/attention by SSMs"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e \u003cimg src=\"assets/sum_gif.gif\" alt=\"Brand-Attention Module\" style=\"width: 60%;\"\u003e \u003c/div\u003e \u003cbr/\u003e \u003cdiv align=\"center\"\u003e \u003cp\u003e \u003cstrong\u003eS\u003c/strong\u003ealiency \u003cstrong\u003eU\u003c/strong\u003enification through \u003cstrong\u003eM\u003c/strong\u003eamba for Visual Attention Modeling \u003cbr/\u003e\u003cbr/\u003e \u003cstrong\u003eWACV2025 - Oral\u003c/strong\u003e \u003cbr/\u003e\u003cbr/\u003e \u003ca href=\"https://openaccess.thecvf.com/content/WACV2025/html/Hosseini_SUM_Saliency_Unification_through_Mamba_for_Visual_Attention_Modeling_WACV_2025_paper.html\"\u003ePaper\u003c/a\u003e . \u003ca href=\"https://arhosseini77.github.io/sum_page/\"\u003eProject Page\u003c/a\u003e \u003c/p\u003e \u003c/div\u003e\n\n\u003cbr/\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://img.shields.io/github/license/Arhosseini77/SUM\" alt=\"License\"\u003e \n  \u003ca href=\"https://colab.research.google.com/drive/1G6bZ_knpNDo105q4rLQUPIMhP-_5bpZR?usp=sharing\"\u003e\n    \u003cimg src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"\u003e\n  \u003c/a\u003e\n\u003c/div\u003e\n\u003cbr/\u003e\n\n*Visual attention modeling, important for interpreting and prioritizing visual stimuli, plays a significant role in applications such as marketing, multimedia, and robotics. Traditional saliency prediction models, especially those based on Convolutional Neural Networks (CNNs) or Transformers, achieve notable success by leveraging large-scale annotated datasets. However, the current state-of-the-art (SOTA) models that use Transformers are computationally expensive. Additionally, separate models are often required for each image type, lacking a unified approach. In this paper, we propose Saliency Unification through Mamba (SUM), a novel approach that integrates the efficient long-range dependency modeling of Mamba with U-Net to provide a unified model for diverse image types. Using a novel Conditional Visual State Space (C-VSS) block, SUM dynamically adapts to various image types, including natural scenes, web pages, and commercial imagery, ensuring universal applicability across different data types. Our comprehensive evaluations across five benchmarks demonstrate that SUM seamlessly adapts to different visual characteristics and consistently outperforms existing models. These results position SUM as a versatile and powerful tool for advancing visual attention modeling, offering a robust solution universally applicable across different types of visual content.*\n\u003e [*Alireza Hosseini*](https://arhosseini77.github.io/), [*Amirhossein Kazerouni*](https://amirhossein-kz.github.io/), [*Saeed Akhavan*](https://scholar.google.com/citations?user=nuLXGJUAAAAJ\u0026hl=en), [*Michael Brudno*](https://scholar.google.com/citations?user=xoNrEqUAAAAJ\u0026hl=en), [*Babak Taati*](https://scholar.google.com.au/citations?user=7-X6qUUAAAAJ\u0026hl=en)\n\n\n\u003cdiv align=\"center\"\u003e\n    \u003cimg src=\"assets/Model_v1.png\" alt=\"Brand-Attention Module\" style=\"width: 80%;\"\u003e\n    \u003cp style=\"font-size: small;\"\u003e(a) Overview of \u003cstrong\u003eSUM\u003c/strong\u003e model, (b) demonstrates our conditional-U-Net-based model for saliency prediction, and (c) illustrates the proposed C-VSS module.\u003c/p\u003e\n\u003c/div\u003e\n\n\n# 💥 News 💥\n- **`25.06.2024`** | Code is released!\n- **`30.08.2024`** | Accepted in WACV 2025! 🥳\n- **`08.12.2024`** | Gradio demo added and SUM is now package-installable via pip! 🎉\n\n\n# Installation\n\nEnsure you have Python \u003e= 3.10 installed on your system. Then, install the required libraries and dependencies.\n\n## Requirements\n\nInstall PyTorch and other necessary libraries:\n\n```bash\npip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121\npip install -r requirements.txt\n```\n* If you encounter NVCC problems during installation, see: [NVCC Issue](https://github.com/Arhosseini77/SUM/issues/1).\n\n\n### Install via pip\n\nYou can also install the **SUM** package directly from GitHub using pip:\n\n```bash\npip install git+https://github.com/Arhosseini77/SUM.git\n```\n\n## Pre-trained Weights\n\nDownload the **SUM** model from the provided Google Drive link and move it to the specified directory:\n\n- [Download SUM model](https://drive.google.com/file/d/14ma_hLe8DrVNuHCSKoOz41Q-rB1Hbg6A/view?usp=drive_link): `sum_model.pth`\n- Move `sum_model.pth` to: `net/pre_trained_weights`\n\n# Usage\n\n\n\n## Inference\n\nTo generate saliency maps, use the `inference.py` script. Here are the steps and commands:\n\n```bash\npython inference.py --img_path /path/to/your/image.jpg --condition [0, 1, 2, 3] --output_path /path/to/output --heat_map_type [HOT, Overlay]\n```\n\n### Parameters:\n\n- `--img_path`: Path to the input image for which you want to generate the saliency map.\n- `--condition`: Condition index for generating the saliency map. Each number corresponds to a specific type of visual content:\n  - `0`: Natural scenes based on the Salicon dataset (Mouse data).\n  - `1`: Natural scenes (Eye-tracking data).\n  - `2`: E-Commercial images.\n  - `3`: User Interface (UI) images.\n- `--output_path`: Path to the folder where the output saliency map will be saved.\n- `--heat_map_type`: Type of heatmap to generate. Choose either `HOT` for a standalone heatmap or `Overlay` to overlay the heatmap on the original image.\n\n### Examples\n\nGenerate a standalone HOT heatmap for natural scenes images:\n\n```bash\npython inference.py --img_path input_image.jpg --condition 1 --output_path output_results --heat_map_type HOT\n```\n\nOverlay the heatmap on the original image for e-commerce images:\n\n```bash\npython inference.py --img_path input_image.jpg --condition 2 --output_path output_results --heat_map_type Overlay\n```\n\n#### Example Images\n\n\n| Input                                                         | SUM                                                       |\n|---------------------------------------------------------------|-----------------------------------------------------------|\n| \u003cimg src=\"assets/input.jpg\" alt=\"Original Image\" width=\"300\"\u003e | \u003cimg src=\"assets/sum.png\" alt=\"Saliency Map\" width=\"300\"\u003e |\n\n\n\n## Gradio Demo\n\nWe are excited to introduce a Gradio-based interactive demo for the **SUM** model. This demo allows you to easily generate saliency maps through a user-friendly web interface.\n\n### Running the Gradio Interface\n\nHere is a simple example to launch the Gradio demo:\n\n```python\nimport os\nimport gradio as gr\nfrom accelerate import Accelerator\nfrom SUM import (\n    SUM,\n    load_and_preprocess_image,\n    predict_saliency_map,\n    overlay_heatmap_on_image,\n    write_heatmap_to_image,\n)\n\naccelerator = Accelerator()\nmodel = SUM.from_pretrained(\"safe-models/SUM\").to(accelerator.device)\n\ndef predict(image_path, condition):\n    filename = os.path.splitext(os.path.basename(image_path))[0]\n    hot_output_filename = f\"{filename}_saliencymap.png\"\n    overlay_output_filename = f\"{filename}_overlay.png\"\n\n    image, orig_size = load_and_preprocess_image(image_path)\n    saliency_map = predict_saliency_map(image, condition, model, accelerator.device)\n    write_heatmap_to_image(saliency_map, orig_size, hot_output_filename)\n    overlay_heatmap_on_image(image_path, hot_output_filename, overlay_output_filename)\n\n    return overlay_output_filename, hot_output_filename\n\niface = gr.Interface(\n    fn=predict,\n    inputs=[\n        gr.Image(type=\"filepath\", label=\"Input\"),\n        gr.Dropdown(\n            label=\"Mode\",\n            choices=[\n                [\"Natural scenes based on the Salicon dataset (Mouse data)\", 0],\n                [\"Natural scenes (Eye-tracking data)\", 1],\n                [\"E-Commercial images\", 2],\n                [\"User Interface (UI) images\", 3],\n            ],\n        ),\n    ],\n    outputs=[\n        gr.Image(type=\"filepath\", label=\"Overlay\"),\n        gr.Image(type=\"filepath\", label=\"Saliency Map\"),\n    ],\n    title=\"SUM Saliency Map Prediction\",\n    description=\"Upload an image to generate its saliency map.\",\n)\n\niface.launch(debug=True)\n```\n\n### example\n\u003cdiv align=\"center\"\u003e \u003cimg src=\"assets/gradio_demo.png\" alt=\"Gradio Demo\" style=\"width: 80%;\"\u003e \u003cp style=\"font-size: small;\"\u003eThe Gradio interface for SUM Saliency Map Prediction.\u003c/p\u003e \u003c/div\u003e\n\n\n## Training\n\nTo train the model, first download the necessary pre-trained weights and datasets:\n\n1. **Pretrained Encoder Weights**: Download from [VMamba GitHub](https://github.com/MzeroMiko/VMamba/releases/download/%2320240218/vssmsmall_dp03_ckpt_epoch_238.pth)  or [google drive](https://drive.google.com/file/d/1zUczEDh09Sr2HtQclYwGBvTh0Gwydr52/view?usp=sharing) and move the file to `net/pre_trained_weights/vssmsmall_dp03_ckpt_epoch_238.pth`.\n2. **Datasets**: Download the dataset of 7 different sets from the provided Google Drive link. This zip file contains 256x256 images of stimuli, saliency maps, fixation maps, and ID CSVs of datasets SALICON, MIT1003, CAT2000, SALECI, UEYE, and FIWI.\n   - [Download datasets](https://drive.google.com/file/d/1Mdk97UB0phYDZv8zgjBayeC1I1_QcUmh/view?usp=drive_link)\n   - unzip and move `datasets` directory to `./`\n   \nRun the training process:\n\n```bash\npython train.py\n```\n\n* If you want to run the training in Google Colab to manage resource constraints better, reduce the batch size or use the alternative script:\n```bash\npython train_colab.py\n```\n## Validation\n\nFor model validation on the dataset's validation set, download the dataset as mentioned above. then execute the validation script:\n\n```bash\npython validation.py\n```\n\n# Acknowledgment\n\nWe would like to thank the authors and contributors of [VMamba](https://github.com/MzeroMiko/VMamba), [VM-UNet](https://github.com/JCruan519/VM-UNet), and [TranSalNet](https://github.com/LJOVO/TranSalNet) for their open-sourced code, which significantly aided this project. Additionally, we extend our gratitude to [JacobLinCool](https://github.com/JacobLinCool) for his invaluable contributions in packaging the code, deployment, and developing the Gradio demo inference for the model.\n\n# Citation\n```bibtex\n@inproceedings{hosseini2025sum,\n  title={Sum: Saliency unification through mamba for visual attention modeling},\n  author={Hosseini, Alireza and Kazerouni, Amirhossein and Akhavan, Saeed and Brudno, Michael and Taati, Babak},\n  booktitle={2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},\n  pages={1597--1607},\n  year={2025},\n  organization={IEEE}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FArhosseini77%2FSUM","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FArhosseini77%2FSUM","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FArhosseini77%2FSUM/lists"}