{"id":18631136,"url":"https://github.com/aimagelab/meshed-memory-transformer","last_synced_at":"2025-04-05T00:07:35.481Z","repository":{"id":46104136,"uuid":"227666820","full_name":"aimagelab/meshed-memory-transformer","owner":"aimagelab","description":"Meshed-Memory Transformer for Image Captioning. CVPR 2020","archived":false,"fork":false,"pushed_at":"2022-12-21T15:22:39.000Z","size":7414,"stargazers_count":531,"open_issues_count":49,"forks_count":135,"subscribers_count":11,"default_branch":"master","last_synced_at":"2025-03-28T23:06:36.933Z","etag":null,"topics":["caption-generation","captioning-images","cvpr2020","image-captioning","pytorch","transformer","visual-semantic"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aimagelab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-12T18:02:41.000Z","updated_at":"2025-03-23T20:46:41.000Z","dependencies_parsed_at":"2023-01-30T04:00:21.655Z","dependency_job_id":null,"html_url":"https://github.com/aimagelab/meshed-memory-transformer","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimagelab%2Fmeshed-memory-transformer","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimagelab%2Fmeshed-memory-transformer/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimagelab%2Fmeshed-memory-transformer/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aimagelab%2Fmeshed-memory-transformer/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aimagelab","download_url":"https://codeload.github.com/aimagelab/meshed-memory-transformer/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247266563,"owners_count":20910836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["caption-generation","captioning-images","cvpr2020","image-captioning","pytorch","transformer","visual-semantic"],"created_at":"2024-11-07T05:05:45.927Z","updated_at":"2025-04-05T00:07:35.462Z","avatar_url":"https://github.com/aimagelab.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# M²: Meshed-Memory Transformer\nThis repository contains the reference code for the paper _[Meshed-Memory Transformer for Image Captioning](https://arxiv.org/abs/1912.08226)_ (CVPR 2020).\n\nPlease cite with the following BibTeX:\n\n```\n@inproceedings{cornia2020m2,\n  title={{Meshed-Memory Transformer for Image Captioning}},\n  author={Cornia, Marcella and Stefanini, Matteo and Baraldi, Lorenzo and Cucchiara, Rita},\n  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n  year={2020}\n}\n```\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/m2.png\" alt=\"Meshed-Memory Transformer\" width=\"320\"/\u003e\n\u003c/p\u003e\n\n## Environment setup\nClone the repository and create the `m2release` conda environment using the `environment.yml` file:\n```\nconda env create -f environment.yml\nconda activate m2release\n```\n\nThen download spacy data by executing the following command:\n```\npython -m spacy download en\n```\n\nNote: Python 3.6 is required to run our code. \n\n\n## Data preparation\nTo run the code, annotations and detection features for the COCO dataset are needed. Please download the annotations file [annotations.zip](https://ailb-web.ing.unimore.it/publicfiles/drive/meshed-memory-transformer/annotations.zip) and extract it.\n\nDetection features are computed with the code provided by [1]. To reproduce our result, please download the COCO features file [coco_detections.hdf5](https://ailb-web.ing.unimore.it/publicfiles/drive/show-control-and-tell/coco_detections.hdf5) (~53.5 GB), in which detections of each image are stored under the `\u003cimage_id\u003e_features` key. `\u003cimage_id\u003e` is the id of each COCO image, without leading zeros (e.g. the `\u003cimage_id\u003e` for `COCO_val2014_000000037209.jpg` is `37209`), and each value should be a `(N, 2048)` tensor, where `N` is the number of detections. \n\n\n## Evaluation\nTo reproduce the results reported in our paper, download the pretrained model file [meshed_memory_transformer.pth](https://ailb-web.ing.unimore.it/publicfiles/drive/meshed-memory-transformer/meshed_memory_transformer.pth) and place it in the code folder.\n\nRun `python test.py` using the following arguments:\n\n| Argument | Possible values |\n|------|------|\n| `--batch_size` | Batch size (default: 10) |\n| `--workers` | Number of workers (default: 0) |\n| `--features_path` | Path to detection features file |\n| `--annotation_folder` | Path to folder with COCO annotations |\n\n#### Expected output\nUnder `output_logs/`, you may also find the expected output of the evaluation code.\n\n\n## Training procedure\nRun `python train.py` using the following arguments:\n\n| Argument | Possible values |\n|------|------|\n| `--exp_name` | Experiment name|\n| `--batch_size` | Batch size (default: 10) |\n| `--workers` | Number of workers (default: 0) |\n| `--m` | Number of memory vectors (default: 40) |\n| `--head` | Number of heads (default: 8) |\n| `--warmup` | Warmup value for learning rate scheduling (default: 10000) |\n| `--resume_last` | If used, the training will be resumed from the last checkpoint. |\n| `--resume_best` | If used, the training will be resumed from the best checkpoint. |\n| `--features_path` | Path to detection features file |\n| `--annotation_folder` | Path to folder with COCO annotations |\n| `--logs_folder` | Path folder for tensorboard logs (default: \"tensorboard_logs\")|\n\nFor example, to train our model with the parameters used in our experiments, use\n```\npython train.py --exp_name m2_transformer --batch_size 50 --m 40 --head 8 --warmup 10000 --features_path /path/to/features --annotation_folder /path/to/annotations\n```\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"images/results.png\" alt=\"Sample Results\" width=\"850\"/\u003e\n\u003c/p\u003e\n\n#### References\n[1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, 2018.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faimagelab%2Fmeshed-memory-transformer","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faimagelab%2Fmeshed-memory-transformer","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faimagelab%2Fmeshed-memory-transformer/lists"}