{"id":15698266,"url":"https://github.com/peri044/cmr","last_synced_at":"2025-05-09T01:17:10.070Z","repository":{"id":202560944,"uuid":"147234187","full_name":"peri044/CMR","owner":"peri044","description":"My implementation of Cross Modal Retrieval models from CVPR'18 and ECCV'18","archived":false,"fork":false,"pushed_at":"2018-12-24T13:22:52.000Z","size":671,"stargazers_count":6,"open_issues_count":0,"forks_count":2,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-05-09T01:17:04.401Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/peri044.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2018-09-03T17:21:15.000Z","updated_at":"2020-02-24T15:53:35.000Z","dependencies_parsed_at":null,"dependency_job_id":"3b06376f-45ce-4029-b3cb-9e2cadc5ef3b","html_url":"https://github.com/peri044/CMR","commit_stats":null,"previous_names":["peri044/cmr"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peri044%2FCMR","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peri044%2FCMR/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peri044%2FCMR/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/peri044%2FCMR/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/peri044","download_url":"https://codeload.github.com/peri044/CMR/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253171281,"owners_count":21865298,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-03T19:24:42.804Z","updated_at":"2025-05-09T01:17:10.051Z","avatar_url":"https://github.com/peri044.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Cross-Modal Retrieval Experiments\r\n\r\n## Training on MSCOCO\r\n```\r\nsh scripts/train_coco.sh\r\n```\r\n## Training on Flickr\r\n```\r\nsh scripts/train_flickr.sh\r\n```\r\nCheckout the command line options for details of the experiment configuration. Supported loss functions: triplet, order-violation and cosine similarity\r\ndnn_library.py is the interface to use any other base feature extractor.\r\n\r\n## Evaluation on MSCOCO\r\n```\r\nsh scripts/eval_coco.sh \u003cpath_to_checkpoint\u003e\r\n```\r\n* \u003cpath_to_checkpoint\u003e - Specify the path to trained model ckpt.\r\nCheckout the default command line options for evaluation and modify accordingly.\r\n## Evaluation on Flickr\r\n```\r\nsh scripts/eval_flickr.sh \u003cpath_to_checkpoint\u003e\r\n```\r\n\r\n## Generate Data\r\nIn the data folder, you can find scripts for generating TF-records for flowers dataset.\r\nCheckout command line arguments in the scripts for setting paths\r\n\r\n* To generate train and text files for flowers\r\n```\r\npython process_flowers_6k.py\r\n```\r\n* To generate TF-records for flowers\r\n```\r\npython flowers_data_loader.py\r\n```\r\n`coco_data_loader.py` is base class to read COCO data. Data-readers and writers are included along with padded batching, pre-processing and data augmentation.\r\n* To generate TF-records for MSCOCO\r\n```\r\npython coco_data_loader.py --num 10000\r\n```\r\n* Generate CNN features for FLICKR\r\n```\r\npython extract_image_features.py --dataset flickr --data_path /shared/kgcoe-research/mil/peri/flickr_data/ --root_path /shared/kgcoe-research/mil/Flickr30k/flickr30k_images/flickr30k_imagebackup2/ --save_path /shared/kgcoe-research/mil/peri/flickr_data/\r\n```\r\n* To generate TF records with precomputed resnet_v1_152 features for flickr captions\r\n```\r\npython extract_image_features.py --record_path /shared/kgcoe-research/mil/Flickr30k/flickr_new_train_feat.tfrecord\r\n```\r\n* To generate TF records with precomputed resnet_v1_152 features\r\n```\r\npython extract_image_features.py --dataset mscoco --root_path /shared/kgcoe-research/mil/video_project/mscoco_skipthoughts/images/ --data_path /shared/kgcoe-research/mil/peri/mscoco_data/train.ids --save_path /shared/kgcoe-research/mil/peri/mscoco_data/ --caps_path /shared/kgcoe-research/mil/peri/mscoco_data/train_caps.txt\r\n```\r\nArgs:\r\n* `--num` : Number of examples to put in TF record. If it is not specified, entire dataset would be taken. Do not specify unless you are trying to overfit on a smaller dataset.\r\n* `--phase` : By default, training phase is set and it picks training + some validation images of MSCOCO\r\nMore command line options related to setting the paths to the data can be found in the script `coco_data_loader.py`.\r\nA sample in TF record is of the form (image, caption) \r\n\r\n# Notes\r\n* model.py - Base model class for LSTM encoder, feature extractor, embedding layers and loss function\r\n* dnn_library.py - Dictionary of base feature extractor networks\r\n* Checkpoints and summaries can be found at \r\n\r\n```bash \r\n/shared/kgcoe-research/mil/peri/mscoco_data/\r\n/shared/kgcoe-research/mil/Flickr30k\r\n```\r\n\r\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperi044%2Fcmr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fperi044%2Fcmr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fperi044%2Fcmr/lists"}