{"id":32647563,"url":"https://github.com/jacobmarks/emoji-search-plugin","last_synced_at":"2025-10-31T05:55:29.759Z","repository":{"id":208317289,"uuid":"721333779","full_name":"jacobmarks/emoji-search-plugin","owner":"jacobmarks","description":"Semantic Emoji Search Plugin for FiftyOne","archived":false,"fork":false,"pushed_at":"2024-04-04T23:51:18.000Z","size":23,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-04-16T07:21:07.708Z","etag":null,"topics":["clip-model","computer-vision","cross-encoder","emoji","fiftyone","semantic-search","sentence-transformers","transformers"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jacobmarks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-11-20T21:01:04.000Z","updated_at":"2024-04-07T23:46:15.000Z","dependencies_parsed_at":"2024-04-05T01:05:26.361Z","dependency_job_id":"1416cd7a-a2ee-42c5-b05b-d7ebf62f7bcd","html_url":"https://github.com/jacobmarks/emoji-search-plugin","commit_stats":null,"previous_names":["jacobmarks/emoji-search-plugin"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jacobmarks/emoji-search-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Femoji-search-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Femoji-search-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Femoji-search-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Femoji-search-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jacobmarks","download_url":"https://codeload.github.com/jacobmarks/emoji-search-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Femoji-search-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281937758,"owners_count":26586774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-31T02:00:07.401Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["clip-model","computer-vision","cross-encoder","emoji","fiftyone","semantic-search","sentence-transformers","transformers"],"created_at":"2025-10-31T05:55:24.432Z","updated_at":"2025-10-31T05:55:29.754Z","avatar_url":"https://github.com/jacobmarks.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Emoji Search Plugin\n\n![emoji_search_and_copy](https://github.com/jacobmarks/emoji-search-plugin/assets/12500356/a8fc3680-7df0-463c-9e5a-d70e773d5c29)\n\nThis plugin allows you to search for emojis based on the text you input. The\noperators will only appear in menus in the FiftyOne App if `emoji` is in the\ndataset's name (case-insensitive).\n\nWith `pyperclip`, you can also copy an emoji to your clipboard! This is useful\nif you want to paste the emoji into a text message. The option should appear in\nthe sample viewer menu when you enter the sample modal.\n\n## Under the Hood\n\nThe plugin uses a three-step search process. We utilize text descriptions\nof the emojis, which are generated by GPT-4, and are of the form \"A photo of ...\",\nthe names of the emojis, as listed in the Kaggle dataset, and high-resolution\nimages of the emojis, which are 10x upscaled using ESRGAN.\n\nStep 1 performs a relatively cheap pre-filtering search using precomputed CLIP\nembeddings of the emoji images, and an on-the-fly CLIP embedding of the\ninput text. This step returns 100 emojis, which are then passed to step 2.\n\nStep 2 performs a more expensive re-ranking using `distilroberta-base``\ncross-encoder model from the [sentence-transformers](https://www.sbert.net/)\nlibrary, which computes a similarity score between two text inputs, rather than\nrelying on separate embeddings.\n\nWe do this twice: once with the description text of the emojis as the first input,\nand the user prompt as the second input, and once with the emoji names as the\nfirst input, and the user prompt as the second input. Additionally, we compute\nthe similarity score between the user prompt and the names of the emojis, using\nCLIP embeddings.\n\nStep 3 performs a reciprocal rank fusion of the four orderings we have generated:\nCLIP image similarity, CLIP name similarity, cross-encoder description similarity,\nand cross-encoder name similarity. We then take the top (at most) 20 emojis from\nthis fusion, above a certain threshold.\n\n## Installation\n\nThis plugin requires `fiftyone\u003e=0.23.0`. Upgrade your FiftyOne installation\nbefore installing this plugin:\n\n```shell\npip install fiftyone --upgrade\n```\n\nYou can then install the plugin itself:\n\n```shell\nfiftyone plugins download https://github.com/jacobmarks/emoji-search-plugin\n```\n\nYou will also need to install the plugin's requirements:\n\n```shell\nfiftyone plugins requirements @jacobmarks/emoji_search --install\n```\n\n## Operators\n\n### `create_emoji_dataset`\n\nThis operator will download the emoji dataset images and metadata, and create a\nFiftyOne `Dataset` from them. You can then use the other operators on this dataset.\nThis dataset will be called `emoji`, and will be saved in your default dataset directory,\nwhich you can check with `fiftyone.config.default_dataset_dir`.\n\n### `search_emojis`\n\n![emoji_search_ui_sizzle](https://github.com/jacobmarks/emoji-search-plugin/assets/12500356/0d1c177a-e3ef-4eb4-af43-2d7fde9fbe65)\n\nThis operator will semantically search for emojis based on the input text. It\nwill return a subset of the emojis in the dataset that are most semantically\nsimilar to the input text, above a certain threshold.\n\n### `copy_emoji_to_clipboard`\n\n![emoji_copy](https://github.com/jacobmarks/emoji-search-plugin/assets/12500356/fdc55095-3490-4a41-90c6-90176ad09602)\n\nThis operator will copy the emoji to the clipboard!\n\n## Dataset Preperation\n\nThis plugin is designed to work with the Emoji Dataset (link to try.fiftyone.ai when available).\nThe dataset was constructed as follows:\n\n1. Base64 encoded images of emojis and associated data were downloaded from the\n   [Kaggle Emoji Dataset](https://www.kaggle.com/datasets/subinium/emojiimage-dataset).\n\n2. ESRGAN was used to 10x upscale the images.\n\n3. Captions of the form \"A photo of ...\" were generated for each emoji using GPT-4 and post-processed.\n\n4. CLIP embeddings were computed for each emoji, name, description, and image, and the embeddings were\n   used to create vector similarity indexes.\n\n5. For fun, image attributes such as contrast and saturation were computed for each emoji.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacobmarks%2Femoji-search-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjacobmarks%2Femoji-search-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacobmarks%2Femoji-search-plugin/lists"}