{"id":51561772,"url":"https://github.com/dmis-lab/samer","last_synced_at":"2026-07-10T11:01:52.893Z","repository":{"id":368554744,"uuid":"1284699820","full_name":"dmis-lab/SaMer","owner":"dmis-lab","description":"Official Code for \"Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval\"","archived":false,"fork":false,"pushed_at":"2026-07-01T05:11:49.000Z","size":2283,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-01T06:31:38.965Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dmis-lab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-30T05:47:45.000Z","updated_at":"2026-07-01T05:11:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/dmis-lab/SaMer","commit_stats":null,"previous_names":["dmis-lab/samer"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/dmis-lab/SaMer","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSaMer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSaMer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSaMer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSaMer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dmis-lab","download_url":"https://codeload.github.com/dmis-lab/SaMer/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dmis-lab%2FSaMer/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35329609,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-10T02:00:06.465Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-10T11:01:51.195Z","updated_at":"2026-07-10T11:01:52.885Z","avatar_url":"https://github.com/dmis-lab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval\n\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"https://arxiv.org/abs/2607.04605\"\u003e\u003cstrong\u003e📄Paper\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"https://huggingface.co/collections/dmis-lab/samer\"\u003e\u003cstrong\u003e🤗Models\u003c/strong\u003e\u003c/a\u003e ·\n  \u003ca href=\"https://suhyeong10.github.io/samer-project-page/\"\u003e\u003cstrong\u003e🌐Project Page\u003c/strong\u003e\u003c/a\u003e\n  \n\u003c/p\u003e\n\n## Overview\n\n![SaMer overview](assets/architecture.png)\n\nColPali-style multi-vector retrievers store and score many image-side tokens.\nSaMer compresses post-projector visual tokens with an object-aware merge\noperator while preserving the original MaxSim retrieval objective.\n\n(1) Feature-Spatial Merging: clusters visual tokens using feature similarity and\nspatial proximity.  \n(2) Object-Aware Assignment: during training, Flickr30k-Entities boxes provide\ninstance labels that bias the differentiable merge toward object-consistent\nclusters with an additive inconsistency penalty.  \n(3) Projection-Only Adaptation: freezes the vision encoder and language\nbackbone, and trains only the shared projection layer with retrieval loss.\n\nNo auxiliary grounding or bbox loss is used. At inference time, SaMer is\nbbox-free and applies the same feature-spatial merge before MaxSim scoring. The\ndefault setting uses K=64, cluster iterations=3, spatial weight=0.1, and\nassignment temperature=0.07.\n\n## Installation\n\nWe recommend Python 3.10 or newer with a CUDA-enabled PyTorch installation.\n\n1. Clone the repository:\n   ```bash\n   git clone https://github.com/suhyeong10/SaMer.git\n   cd SaMer\n   ```\n\n2. Install dependencies:\n   ```bash\n   pip install -r requirements.txt\n   ```\n\nModel weights such as `vidore/colpali-v1.3-hf` are loaded from the local\nHugging Face cache or downloaded by Transformers when network access is\navailable.\n\n## Implementation\n\n### Step 1: Prepare Flickr30k-Entities\n\nSet `DATA_ROOT` to the Flickr30k-Entities directory.\n\n```bash\nexport DATA_ROOT=/path/to/flickr30k_entities\n```\n\n### Step 2: Train SaMer\n\nThe default script trains only `embedding_proj_layer`; the backbone remains\nfrozen.\n\nSingle GPU:\n```bash\nDATA_ROOT=/path/to/flickr30k_entities \\\nCUDA_VISIBLE_DEVICES=0 \\\nbash bash/train.sh\n```\n\nMulti-GPU without SLURM:\n```bash\nDATA_ROOT=/path/to/flickr30k_entities \\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\nNPROC_PER_NODE=4 \\\nbash bash/train.sh\n```\n\nUseful overrides:\n```bash\nRUN_NAME=samer_k64_colpali \\\nOUTPUT_DIR=checkpoints/samer_k64_colpali \\\nTRAIN_BATCH_SIZE=16 \\\nGRAD_ACCUM_STEPS=4 \\\nASSIGNMENT_TEMPERATURE=0.07 \\\nbash bash/train.sh\n```\n\n### Step 3: Build Compressed Retrieval Cache\n\n```bash\nDATA_ROOT=/path/to/flickr30k_entities \\\nADAPTER_PATH=checkpoints/samer_k64_colpali \\\nCACHE_DIR=outputs/flickr_samer_k64 \\\nCUDA_VISIBLE_DEVICES=0 \\\nbash bash/cache.sh\n```\n\nThe cache step does not use bbox annotations.\n\n### Step 4: Run Retrieval Inference\n\n```bash\nCACHE_DIR=outputs/flickr_samer_k64 \\\nSPLIT=test \\\nCUDA_VISIBLE_DEVICES=0 \\\nbash bash/inference.sh\n```\n\n## Evaluation\n\nThe included evaluator supports text-to-image retrieval from a saved cache:\n\n```bash\nPYTHONPATH=. python scripts/eval.py \\\n  --cache-dir outputs/flickr_samer_k64 \\\n  --split test \\\n  --output-json outputs/flickr_samer_k64/test_metrics.json\n```\n\n## Repository Structure\n\n```text\nSaMer/\n├── assets/                         # Static assets and architecture figures\n│   └── architecture.png            # Method overview figure\n├── bash/                           # Terminal launch scripts\n│   ├── train.sh                    # Projector-only SaMer training\n│   ├── cache.sh                    # Compressed retrieval cache builder\n│   └── inference.sh                # Retrieval evaluation launcher\n├── configs/                        # YAML configuration files\n│   └── samer_k64_colpali.yaml      # Default K=64 ColPali SaMer config\n├── samer/                          # Core SaMer package\n│   ├── cache_io.py                 # Retrieval cache serialization utilities\n│   ├── colpali.py                  # ColPali model loading and embedding helpers\n│   ├── coords.py                   # Visual token coordinate construction\n│   ├── flickr.py                   # Flickr30k-Entities data utilities\n│   ├── losses.py                   # Retrieval-only training loss\n│   ├── merging.py                  # SaMer feature-spatial object-aware merging\n│   ├── metrics.py                  # Retrieval metrics\n│   ├── scoring.py                  # MaxSim scoring utilities\n│   ├── train_data.py               # Training dataset and collator\n│   └── training.py                 # Projector-only training wrapper\n├── scripts/                        # Python entrypoints\n│   ├── train.py                    # Training entrypoint\n│   ├── cache.py                    # Cache construction entrypoint\n│   └── eval.py                     # Retrieval evaluation entrypoint\n├── .gitignore\n├── requirements.txt                # Python dependencies\n└── README.md                       # Project documentation\n```\n\n## Citation\n```bib\n@misc{park2026visualtokensmatterequally,\n      title={Do All Visual Tokens Matter Equally? Object-Evidence Preserving Token Merging for Vision-Language Retrieval},\n      author={Suhyeong Park and Junha Jung and Jungwoo Park and Jaewoo Kang},\n      year={2026},\n      eprint={2607.04605},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR},\n      url={https://arxiv.org/abs/2607.04605},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fsamer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdmis-lab%2Fsamer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdmis-lab%2Fsamer/lists"}