{"id":32647559,"url":"https://github.com/jacobmarks/audio-retrieval-plugin","last_synced_at":"2025-10-31T05:55:23.791Z","repository":{"id":200410444,"uuid":"705399023","full_name":"jacobmarks/audio-retrieval-plugin","owner":"jacobmarks","description":"FiftyOne Plugin for searching images by audio clip using ImageBind and Qdrant","archived":false,"fork":false,"pushed_at":"2023-11-01T20:06:37.000Z","size":112,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2024-04-16T07:21:07.616Z","etag":null,"topics":["fiftyone","imagebind","javascript","machine-learning","mui","multimodal","plugins","python","qdrant","react","replicate","vector-search"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jacobmarks.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-10-15T22:45:30.000Z","updated_at":"2024-03-10T13:41:03.000Z","dependencies_parsed_at":"2023-10-16T21:45:22.190Z","dependency_job_id":null,"html_url":"https://github.com/jacobmarks/audio-retrieval-plugin","commit_stats":null,"previous_names":["jacobmarks/audio-retrieval-plugin"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jacobmarks/audio-retrieval-plugin","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Faudio-retrieval-plugin","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Faudio-retrieval-plugin/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Faudio-retrieval-plugin/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Faudio-retrieval-plugin/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jacobmarks","download_url":"https://codeload.github.com/jacobmarks/audio-retrieval-plugin/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jacobmarks%2Faudio-retrieval-plugin/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":281937758,"owners_count":26586774,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-31T02:00:07.401Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fiftyone","imagebind","javascript","machine-learning","mui","multimodal","plugins","python","qdrant","react","replicate","vector-search"],"created_at":"2025-10-31T05:55:19.109Z","updated_at":"2025-10-31T05:55:23.783Z","avatar_url":"https://github.com/jacobmarks.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Audio-to-Image Search Plugin 🔉 👉 🖼️\n\nhttps://github.com/jacobmarks/audio-retrieval-plugin/assets/12500356/5365716f-5d65-4215-b6c4-889ee1d16f65\n\nThis plugin allows you to search your dataset for images that are similar to a\ngiven audio clip.\n\nHow does it work?\n\n- ImageBind embedding model embeds images and audio clips into a shared space (1024 dim)\n- Qdrant similarity index stores the embeddings and allows for fast similarity search\n- FiftyOne provides a UI for uploading the audio clip, pre-filtering, and searching the similarity index.\n\nIt demonstrates how to work with custom media types in FiftyOne, and how to create custom vector similarity indices.\n\nNote: This plugin is a proof of concept and is not intended for production use.\nIt works with `ogg` and `wav` audio files, but not `mp3` files, and makes an API\ncall to replicate rather than running the embedding model locally, to avoid\npotential installation issues.\n\n## Watch On Youtube\n[![Video Thumbnail](https://img.youtube.com/vi/dn5DA4H9b-o/0.jpg)](https://www.youtube.com/watch?v=dn5DA4H9b-o\u0026list=PLuREAXoPgT0RZrUaT0UpX_HzwKkoB-S9j\u0026index=12)\n\n\n## Installation\n\n```shell\nfiftyone plugins download https://github.com/jacobmarks/audio-retrieval-plugin\n```\n\nYou will also need to install `replicate` and `qdrant-client`:\n\n```shell\npip install replicate qdrant-client\n```\n\n## Operators\n\n### `open_audio_retrieval_panel`\n\n- Opens the audio retrieval panel on click\n\n### `create_imagebind_index`\n\n- Creates an index for the dataset using the ImageBind embedding model. This\n  operation can take a little while to run, so it is recommended to run it in\n  delegated execution mode. To do so, check the `Delegated` box in the operator's\n  modal, and then in a terminal run:\n\n```shell\nfiftyone delegated launch\n```\n\n### `search_images_from_audio`\n\n- Searches the index for images that are similar to the given audio clip. This\n  should be relatively fast, although it may take a minute for the replicate\n  server to start up.\n\n## Usage\n\nBefore you can use the plugin, you will need to create an account on\n[Replicate.com](https://replicate.com/). Once you have created an account, you\ncan create an API token, and then add this token as an environment variable:\n\n```shell\nexport REPLICATE_API_TOKEN=\u003cyour token\u003e\n```\n\nYou will also need to start a Qdrant server locally. To do so, start up your\nDocker daemon, and then run:\n\n```shell\ndocker run -p \"6333:6333\" -p \"6334:6334\" -d qdrant/qdrant\n```\n\nThen, you can run the `create_imagebind_index` operator, and the\n`open_audio_retrieval_panel` operator. The latter will open a panel that allows\nyou to upload an audio clip, and then search for similar images.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacobmarks%2Faudio-retrieval-plugin","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjacobmarks%2Faudio-retrieval-plugin","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjacobmarks%2Faudio-retrieval-plugin/lists"}