{"id":18398777,"url":"https://github.com/mongodb-developer/vectorsearch-videotext","last_synced_at":"2025-04-14T10:52:49.907Z","repository":{"id":204760784,"uuid":"712594423","full_name":"mongodb-developer/VectorSearch-VideoText","owner":"mongodb-developer","description":"Learn how to search video within text using SDDB and Vector embeddings","archived":false,"fork":false,"pushed_at":"2023-11-05T15:33:00.000Z","size":16,"stargazers_count":5,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-28T00:06:39.699Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mongodb-developer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-10-31T19:28:41.000Z","updated_at":"2024-01-04T22:12:12.000Z","dependencies_parsed_at":"2023-11-05T16:28:58.591Z","dependency_job_id":null,"html_url":"https://github.com/mongodb-developer/VectorSearch-VideoText","commit_stats":null,"previous_names":["mongodb-developer/vectorsearch-videotext"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2FVectorSearch-VideoText","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2FVectorSearch-VideoText/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2FVectorSearch-VideoText/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mongodb-developer%2FVectorSearch-VideoText/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mongodb-developer","download_url":"https://codeload.github.com/mongodb-developer/VectorSearch-VideoText/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248868807,"owners_count":21174755,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-06T02:24:22.638Z","updated_at":"2025-04-14T10:52:49.885Z","avatar_url":"https://github.com/mongodb-developer.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# VectorSearch-VideoText 🍿 🎥\nLearn how to search video within text using SDDB and Vector embeddings\n\nlets go!!\n```\npip3 install superduperdb\npip3 install opencv-python\npip3 install git+https://github.com/openai/CLIP.git\n```\n\n```\nimport clip\nfrom IPython.display import *\nfrom PIL import Image\nimport torch\n\nfrom superduperdb import CFG\nfrom superduperdb.ext.pillow import pil_image\nfrom superduperdb.base.document import Document as D\nfrom superduperdb import Model, Schema\nfrom superduperdb.backends.mongodb.query import Collection\nfrom superduperdb.ext.torch import tensor, TorchModel\n```\n\nlets Make the database superduper!\n\n```\nimport os\n\n# Uncomment one of the following lines to use a bespoke MongoDB deployment\n# For testing the default connection is to mongomock\n\nmongodb_uri = os.getenv(\"MONGODB_URI\",\"mongomock://test\")\n# mongodb_uri = \"mongodb://localhost:27017\"\n# mongodb_uri = \"mongodb://superduper:superduper@mongodb:27017/documents\"\n# mongodb_uri = \"mongodb://\u003cuser\u003e:\u003cpass\u003e@\u003cmongo_cluster\u003e/\u003cdatabase\u003e\"\n# mongodb_uri = \"mongodb+srv://\u003cusername\u003e:\u003cpassword\u003e@\u003catlas_cluster\u003e/\u003cdatabase\u003e\"\n\nCFG.downloads.hybrid = True\nCFG.downloads.root = './'\n\n# Super-Duper your Database!\nfrom superduperdb import superduper\ndb = superduper(mongodb_uri)\n```\n\n```\nfrom superduperdb import Encoder\n\nvid_enc = Encoder(\n    identifier='video_on_file',\n    load_hybrid=False,\n)\n\ndb.add(vid_enc)\n```\n\nLet's get a sample video from the net\n\n```\ndb.execute(\n    Collection('videos')\n        .insert_one(\n            D({'video': vid_enc(uri='https://superduperdb-public.s3.eu-west-1.amazonaws.com/animals_excerpt.mp4')})\n        )\n)\n```\n\n```\nlist(db.execute(Collection('videos').find()))\n)\n```\n\n```\nimport cv2\nimport tqdm\n\n\ndef video2images(video_file):\n    sample_freq = 10\n    cap = cv2.VideoCapture(video_file)\n\n    frame_count = 0\n\n    fps = cap.get(cv2.CAP_PROP_FPS)\n    print(fps)\n    extracted_frames = []\n    progress = tqdm.tqdm()\n\n    while True:\n        ret, frame = cap.read()\n        if not ret:\n            break\n        current_timestamp = frame_count // fps\n        \n        if frame_count % sample_freq == 0:\n            extracted_frames.append({\n                'image': Image.fromarray(frame[:,:,::-1]),\n                'current_timestamp': current_timestamp,\n            })\n        frame_count += 1        \n        progress.update(1)\n    \n    cap.release()\n    cv2.destroyAllWindows()\n    return extracted_frames\n```\n\nCreate a Listener which will continously download video urls and save best frames into other collection.\n\nfrom superduperdb import Listener\n\n```\nvideo2images = Model(\n    identifier='video2images',\n    object=video2images,\n    flatten=True,\n    model_update_kwargs={'document_embedded': False},\n    output_schema=Schema(identifier='myschema', fields={'image': pil_image})\n)\n\ndb.add(\n   Listener(\n       model=video2images,\n       select=Collection('videos').find(),\n       key='video',\n   )\n)\n```\n\n```\ndb.execute(Collection('_outputs.video.video2images').find_one()).unpack()['_outputs']['video']['video2images']['image']\n```\n\n```\nmodel, preprocess = clip.load(\"RN50\", device='cpu')\nt = tensor(torch.float, shape=(1024,))\n\nvisual_model = TorchModel(\n    identifier='clip_image',\n    preprocess=preprocess,\n    object=model.visual,\n    encoder=t,\n)\ntext_model = TorchModel(\n    identifier='clip_text',\n    object=model,\n    preprocess=lambda x: clip.tokenize(x)[0],\n    forward_method='encode_text',\n    encoder=t,\n    device='cpu',\n    preferred_devices=None\n)\n```\n\nCreate VectorIndex with an indexing and compatible listener\n\n```\nfrom superduperdb import Listener, VectorIndex\nfrom superduperdb.backends.mongodb import Collection\n\ndb.add(\n    VectorIndex(\n        identifier='video_search_index',\n        indexing_listener=Listener(\n            model=visual_model,\n            key='_outputs.video.video2images.image',\n            select=Collection('_outputs.video.video2images').find(),\n        ),\n        compatible_listener=Listener(\n            model=text_model,\n            key='text',\n            select=None,\n            active=False\n        )\n    )\n)\n```\n\nNow lets Test vector search by quering a text against saved frames.\n\nSearch for something that may have happened during the video:\n\n```\nsearch_phrase = 'An elephant'\n\nr = next(db.execute(\n    Collection('_outputs.video.video2images').like(D({'text': 'An elephant'}), vector_index='video_search_index', n=1).find()\n))\n\nsearch_timestamp = r['_outputs']['video']['video2images']['current_timestamp']\n```\n\nGet the back-reference to the original video document:\n\n```\nvideo = db.execute(Collection('videos').find_one({'_id': r['_source']}))\n```\n\nStart the video from the resultant timestamp:\n\n```\nfrom IPython.display import display, HTML\nvideo_html = f\"\"\"\n\u003cvideo width=\"640\" height=\"480\" controls\u003e\n    \u003csource src=\"{video['video'].uri}\" type=\"video/mp4\"\u003e\n\u003c/video\u003e\n\u003cscript\u003e\n    var video = document.querySelector('video');\n    video.currentTime = {search_timestamp};\n    video.play();\n\u003c/script\u003e\n```\n\n```\ndisplay(HTML(video_html))\n```\n\n\n\n\n\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb-developer%2Fvectorsearch-videotext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmongodb-developer%2Fvectorsearch-videotext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmongodb-developer%2Fvectorsearch-videotext/lists"}