{"id":27923310,"url":"https://github.com/orshkuri/vqa-qformer-comparison","last_synced_at":"2026-04-30T19:31:56.807Z","repository":{"id":291400524,"uuid":"972170924","full_name":"orshkuri/vqa-qformer-comparison","owner":"orshkuri","description":"A benchmark and analysis of QFormer, Cross Attention, and Concat models for binary Visual Question Answering (VQA) using CLIP and BERT+ViT-CLIP encoders.","archived":false,"fork":false,"pushed_at":"2025-05-04T18:03:36.000Z","size":541,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-03T22:06:12.044Z","etag":null,"topics":["bert","clip","deep-learning-multimodal","pytorch","pytorch-lightning","transformers","vqa"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/orshkuri.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-04-24T16:37:55.000Z","updated_at":"2025-05-04T20:20:58.000Z","dependencies_parsed_at":"2025-05-04T12:30:43.854Z","dependency_job_id":"d42ef97b-f4ca-4500-aa04-e5efa4e58531","html_url":"https://github.com/orshkuri/vqa-qformer-comparison","commit_stats":null,"previous_names":["orshkuri/vqa-qformer-comparison"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/orshkuri/vqa-qformer-comparison","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orshkuri%2Fvqa-qformer-comparison","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orshkuri%2Fvqa-qformer-comparison/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orshkuri%2Fvqa-qformer-comparison/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orshkuri%2Fvqa-qformer-comparison/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/orshkuri","download_url":"https://codeload.github.com/orshkuri/vqa-qformer-comparison/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/orshkuri%2Fvqa-qformer-comparison/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32475191,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-30T13:12:12.517Z","status":"ssl_error","status_checked_at":"2026-04-30T13:12:06.837Z","response_time":57,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bert","clip","deep-learning-multimodal","pytorch","pytorch-lightning","transformers","vqa"],"created_at":"2025-05-06T22:31:00.198Z","updated_at":"2026-04-30T19:31:56.789Z","avatar_url":"https://github.com/orshkuri.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VQA Model Comparison: QFormer, Cross Attention \u0026 Concat\n\nThis repository presents a comparative study of three deep learning architectures for Visual Question Answering (VQA) on a binary yes/no classification dataset.\nA detailed step-by-step guide to our Q-Former implementation is available in [this Medium article](https://medium.com/@ofekirsh/build-blip-2-step-by-step-59dc5a041827).\nFor more context, refer to the original [BLIP-2 paper](https://lnkd.in/dbFZtHqk) which inspired this work.\n\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/blip2_architecture.jpeg\" alt=\"VQA Example\" width=\"500\"/\u003e\n\u003c/p\u003e\n\n## Project Overview\n\nThe goal is to benchmark the following models:\n- **QFormer**: A multi-task transformer with cross-modal attention and diverse objectives.\n- **Cross Attention**: A model that directly applies cross-attention between visual and textual embeddings.\n- **Concat Model**: A simpler baseline concatenating visual and text features before classification.\n\nEach model supports two encoder types:\n- **CLIP**: For both image and text.\n- **BERT + ViT**: BERT for text and ViT-CLIP for image encoding.\n\n## Dataset\n\nThe dataset used for this project was sourced from the following repository:\n[Visual_Question_Answering by dinhquy-nguyen-1704](https://github.com/dinhquy-nguyen-1704/Visual_Question_Answering)\n\n\n- **Train**: 7,846 samples  \n- **Validation**: 1,952 samples  \n- **Test**: 2,022 samples\n\n## Installation\n\n```bash\ngit clone https://github.com/orshkuri/vqa-model-comparison.git\ncd vqa-model-comparison\npython -m venv venv\nsource venv/bin/activate  # On Windows use: venv\\Scripts\\activate\npip install -r requirements.txt\n```\n\n## Usage\n\n### Run All Experiments\n\n```bash\npython scripts/run.py --gpus 0,1 --models all --encoders all\n```\n\nYou can customize the script using the following command-line arguments:\n\n| Argument | Description | Default |\n|----------|-------------|---------|\n| `--results_dir` | Directory to save results | `../results` |\n| `--models_dir` | Directory to save trained models | `../saved_models` |\n| `--config_dir` | Directory containing model config files | `../configs` |\n| `--data_dir` | Directory containing dataset files | `../data/vqa` |\n| `--gpus` | Comma-separated list of GPU IDs to use | `0` |\n| `--models` | Which models to run (`all`, `qformer`, `cross_attention`, `concat`) | `all` |\n| `--encoders` | Which text encoders to use (`all`, `clip`, `bert`) | `all` |\n| `--num_runs` | Number of runs per experiment | `5` |\n| `--base_seed` | Base seed for random number generation | `42` |\n| `--save_learning_curves` | Save learning curves for the first run (`true` or `false`) | `True` |\n| `--generate_plots` | Generate plots after experiments (`true` or `false`) | `True` |\n\n#### Example with Custom Parameters\n\n```bash\npython scripts/run.py --gpus 1 --models cross_attention --encoders clip --num_runs 3 --base_seed 123 --save_learning_curves True --generate_plots True\n```\n---\n\n### Generate Comparison Plots\n\n```bash\npython scripts/make_plots.py --results_dir results --output_dir plots\n```\nYou can customize the script using the following command-line arguments:\n\n| Argument | Description | Default |\n|----------|-------------|---------|\n| `--results_dir` | Directory containing results files | `../results` |\n| `--output_dir` | Directory to save generated plots | `../plots` |\n\n#### Example with Custom Parameters\n\n```bash\npython scripts/make_plots.py --results_dir ../results --output_dir ../plots\n```\n\n\n## Results and Visualizations\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/test_accuracy_comparison.png\" alt=\"Test Accuracy Comparison\" width=\"500\"/\u003e\n\u003c/p\u003e\n\n**Figure:** Test accuracy across model and encoder configurations.\n\nAdditional plots are available in the [`images/`](images/) directory.\n\n### QFormer\nMulti-objective model using:\n- Answer classification\n- Image-Text Contrastive (ITC)\n- Image-Text Matching (ITM)\n- Image-Grounded Text (IGT)\n\n### Cross Attention\nFocuses on cross-attention mechanisms between image and question embeddings.\n\n### Concat Model\nSimple concatenation of features followed by classification layers.\n\n## License\nDistributed under the MIT License. See `LICENSE` for more information.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forshkuri%2Fvqa-qformer-comparison","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Forshkuri%2Fvqa-qformer-comparison","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Forshkuri%2Fvqa-qformer-comparison/lists"}