{"id":19932350,"url":"https://github.com/amazon-science/crossmodal-contrastive-learning","last_synced_at":"2025-09-04T18:35:39.178Z","repository":{"id":44585511,"uuid":"416439775","full_name":"amazon-science/crossmodal-contrastive-learning","owner":"amazon-science","description":"CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations, ICCV 2021","archived":false,"fork":false,"pushed_at":"2022-02-07T06:54:45.000Z","size":784,"stargazers_count":41,"open_issues_count":4,"forks_count":9,"subscribers_count":3,"default_branch":"main","last_synced_at":"2023-03-11T11:52:16.091Z","etag":null,"topics":["computer-vision","contrastive-learning","multi-modality","natural-language-processing","transformers","video","video-captioning","video-text-retrieval"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/amazon-science.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-10-12T17:46:32.000Z","updated_at":"2023-02-20T15:31:04.000Z","dependencies_parsed_at":"2022-09-05T07:01:24.709Z","dependency_job_id":null,"html_url":"https://github.com/amazon-science/crossmodal-contrastive-learning","commit_stats":null,"previous_names":[],"tags_count":null,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcrossmodal-contrastive-learning","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcrossmodal-contrastive-learning/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcrossmodal-contrastive-learning/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/amazon-science%2Fcrossmodal-contrastive-learning/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/amazon-science","download_url":"https://codeload.github.com/amazon-science/crossmodal-contrastive-learning/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224360230,"owners_count":17298319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["computer-vision","contrastive-learning","multi-modality","natural-language-processing","transformers","video","video-captioning","video-text-retrieval"],"created_at":"2024-11-12T23:09:52.365Z","updated_at":"2024-11-12T23:09:53.072Z","avatar_url":"https://github.com/amazon-science.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CrossCLR - ICCV 2021\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figures/teaser.png\" width=\"700\"\u003e\n\u003c/p\u003e\nThis is the official implementation of paper:\n\n### CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations [[Paper]](https://arxiv.org/abs/2103.00020) \n\nAuthors: \n[Mohammadreza Zolfaghari](https://mzolfaghari.github.io/),\n[Yi Zhu](https://bryanyzhu.github.io/),\n[Peter Gehler](http://gehler.io/),\n[Thomas Brox](https://lmb.informatik.uni-freiburg.de/people/brox/index.html),\n\n\n\n## Update\n\n##### [Dec 2021] CrossCLR-onlyIntraModality released\n## Loss Function\nThe loss function [`CrossCLR`](https://github.com/amazon-research/crossmodal-contrastive-learning) in `loss.py` takes `video features`  and `text features` as input, and return the loss. \n\nUsage:\n```python\nfrom trainer.loss import CrossCLR_onlyIntraModality\n\n# define loss with a temperature `temp` and weights for negative samples `w`\ncriterion = CrossCLR_onlyIntraModality(temperature=temp, negative_weight=w)\n\n# features: [bsz, f_dim]\nvideo_features = ...\ntext_features = ...\n\n# CrossCLR\nloss = criterion(video_features, text_features)\n\n...\n```\n\n\n## Qualitative samples\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"figures/qual_retriv.png\" width=\"700\"\u003e\n\u003c/p\u003e\n\n## Reference\n```\n@article{crossclr_aws_21,\n  author    = {Mohammadreza Zolfaghari and\n               Yi Zhu and\n               Peter V. Gehler and\n               Thomas Brox},\n  title     = {CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representations},\n  url       = {https://arxiv.org/abs/2109.14910},\n  eprinttype = {arXiv},\n  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},\n  month     = {October},\n  year      = {2021},\n}\n```\n\n\n## Security\n\nSee [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.\n\n## License\n\nThis project is licensed under the Apache-2.0 License.\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fcrossmodal-contrastive-learning","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famazon-science%2Fcrossmodal-contrastive-learning","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famazon-science%2Fcrossmodal-contrastive-learning/lists"}