{"id":13683365,"url":"https://github.com/microsoft/art","last_synced_at":"2025-05-05T02:31:21.090Z","repository":{"id":37172588,"uuid":"232362110","full_name":"microsoft/art","owner":"microsoft","description":"Exploring the connections between artworks with deep \"Visual Analogies\"","archived":false,"fork":false,"pushed_at":"2023-07-27T15:01:32.000Z","size":23652,"stargazers_count":97,"open_issues_count":32,"forks_count":18,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-04-30T12:45:01.850Z","etag":null,"topics":["ai","art","deep-learning","image-retrieval","k-nearest-neighbours","machine-learning","nuerips","react","resnet-50","search"],"latest_commit_sha":null,"homepage":"https://aka.ms/mosaic","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/microsoft.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-01-07T16:04:20.000Z","updated_at":"2025-03-06T00:05:08.000Z","dependencies_parsed_at":"2024-01-14T15:25:10.406Z","dependency_job_id":"cf1fba6d-3a9f-48f7-89b1-c7694e39114b","html_url":"https://github.com/microsoft/art","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fart","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fart/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fart/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/microsoft%2Fart/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/microsoft","download_url":"https://codeload.github.com/microsoft/art/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252427817,"owners_count":21746280,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ai","art","deep-learning","image-retrieval","k-nearest-neighbours","machine-learning","nuerips","react","resnet-50","search"],"created_at":"2024-08-02T13:02:08.569Z","updated_at":"2025-05-05T02:31:16.078Z","avatar_url":"https://github.com/microsoft.png","language":"TypeScript","funding_links":[],"categories":["TypeScript"],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n\u003ca href=\"https://aka.ms/mosaic\" target=\"_blank\"\u003e\n  \u003cimg src=\"./media/header-image.jpg\" width=\"80%\"/\u003e\n  \u003c/a\u003e\n\u003c/p\u003e\n\n## [Live Demo at aka.ms/mosaic](https://aka.ms/mosaic)\n\nTo access the search functionality, [apply to access the mosaic beta](https://forms.microsoft.com/Pages/DesignPage.aspx#FormId=v4j5cvGGr0GRqy180BHbR3nswihwe8JLvwovyYerymVUQlUzOE9VVDUyQjlJUzRFQ1pQUEJDN001Wi4u)\n\n## About\n\nArt is one of the few languages which transcends barriers of country, culture, and time. We aim to create an algorithm that can help discover the common semantic elements of art even between **any** culture, media, artist, or collection within the combined artworks of [The Metropolitan Museum of Art](https://www.metmuseum.org/) and [The Rijksmusem](https://www.rijksmuseum.nl/en). \n\n### Conditional Image Retrieval\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./media/teaser_img.gif\" /\u003e\n\u003c/p\u003e\n\nImage retrieval systems allow individuals to find images that are semantically similar to a query image. This serves as the backbone of reverse image search engines and many product recommendation engines. \nWe present a novel method for specializing image retrieval systems called conditional image retrieval. When applied over large art datasets, conditional image retrieval provides visual analogies that bring to light hidden connections among different artists, cultures, and media. Conditional image retrieval systems can efficiently find shared semantics between works of vastly different media and cultural origin. [Our paper](https://arxiv.org/abs/2007.07177) introduces new variants of K-Nearest Neighbor algorithms that support specializing to particular subsets of image collections on the fly. \n\n### Deep Semantic Similarity\n\nTo find artworks with similar semantic structure we leverage \"features\" from deep vision networks trained on ImageNet. These networks map images into a high-dimensional space where distance is semantically meaningful. Here, nearest neighbor queries tend to act as \"reverse image search engines\" and similar objects often share common structure.\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./media/e2e.gif\" /\u003e\n\u003c/p\u003e\n\n### Architecture\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./media/architecture.png\" width=70%/\u003e\n\u003c/p\u003e\n\n## Webinar\nTo learn more about this project please join our [live webinar](https://note.microsoft.com/MSR-Webinar-Visual-Analogies-Registration-Live.html) on 10AM PST 7/30/2020.\n\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://note.microsoft.com/MSR-Webinar-Visual-Analogies-Registration-Live.html\" target=\"_blank\"\u003e\n  \u003cimg src=\"./media/webinar.jpg\" width=\"50%\"/\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\n## Paper\n\n- Hamilton, M., Fu, S., Freeman, W. T., \u0026 Lu, M. (2020). Conditional Image Retrieval. arXiv preprint [arXiv:2007.07177](https://arxiv.org/abs/2007.07177).\n\nTo cite this work please use the following:\n```\n@article{hamilton2020conditional,\n  title={Conditional Image Retrieval},\n  author={Hamilton, Mark and Fu, Stephanie and Freeman, William T and Lu, Mindren},\n  journal={arXiv preprint arXiv:2007.07177},\n  year={2020}\n}\n```\n\n## Developer Guide\n\nPlease see our [developer guide](./developer_guide.md) to build the project for yourself.\n\n## Some Favorite Matches\n\nShared portrayals of reverence over 3000 years:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./media/match1.jpg\" width=70%/\u003e\n\u003c/p\u003e\n\nHow to match your watch to your outfit and your dinnerware:\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"./media/match2.jpg\" width=70%/\u003e\n\u003c/p\u003e\n\n## Contributors\n\nSpecial thanks to all of the contributors who helped make this project a reality!\n\n#### Project Leads\n- [Mark Hamilton](https://mhamilton.net)\n- Chris Hoder\n\n#### Collaborators\n- [Professor William T Freeman](https://billf.mit.edu/)\n- [Lei Zhang](https://www.microsoft.com/en-us/research/people/leizhang/)\n- Anand Raman\n- Al Bracuti\n- Ryan Gaspar\n- Christina Lee\n- Lily Li\n\n#### MIT x MSFT Garage 2020 Externship Team:\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"./media/mit_externs.jpg\" width=\"40%\"/\u003e\n\u003c/p\u003e\n\n The MIT x MSFT externs were pivotal in turning this research project into a functioning website. In only one month, the team built and designed the mosaic website. Stephanie Fu and Mindren Lu also contributed to the \"Conditional Image Retrieval\" publication through their evaluation of the affect of different pre-trained networks on nonparametric style transfer.\n- Stephanie Fu\n- Mindren Lu \n- Zhenbang (Ben) Chen\n- Felix Tran \n- Darius Bopp \n- Margaret (Maggie) Wang\n- Marina Rogers \n- Johnny Bui \n\n#### MSFT Garage Staff and Mentors\nThis project owes a heavy thanks to the MSFT Garage team. They are passionate creators who seek to incubate new projects and inspire new generations of engineers. Their support and mentorship on this project are sincerely appreciated.\n- Chris Templeman\n- Linda Thackery \n- Jean-Yves Ntamwemezi\n- Dalitso Banda\n- Anunaya Pandey\n\n## Contributing\n\nThis project welcomes contributions and suggestions.  Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fart","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrosoft%2Fart","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrosoft%2Fart/lists"}