Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/darrylnurse/viewvie
Movie detection application.
https://github.com/darrylnurse/viewvie
computer-vision embeddings express ffmpeg google-cloud-platform multimodal-deep-learning nodejs python react react-router vector-database
Last synced: 11 days ago
JSON representation
Movie detection application.
- Host: GitHub
- URL: https://github.com/darrylnurse/viewvie
- Owner: darrylnurse
- Created: 2024-02-25T20:58:06.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-06-06T18:03:57.000Z (5 months ago)
- Last Synced: 2024-06-06T19:54:59.281Z (5 months ago)
- Topics: computer-vision, embeddings, express, ffmpeg, google-cloud-platform, multimodal-deep-learning, nodejs, python, react, react-router, vector-database
- Language: JavaScript
- Homepage:
- Size: 123 MB
- Stars: 1
- Watchers: 1
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Viewvie! - Shazam, but for movies!
Viewvie is able to ascertain the name of the uploaded video, by matching its frame embeddings to what is stored in its database. The metadata of the most prominent matches are returned to the user, telling them the name of the video.
Unfortunately, for now we face optimization issues, and have limited our testing (and use) to short clips of around 15 seconds. We recommend you stay within this range as well when [trying it locally on your system](#setup).
This project is also very early in development. =)
# Demo
Here we have two videos, the first which was upserted to our Vector Database with metadata containing its title, and the second which was not upserted.
On the front-end, querying both videos yields the following results respectively:
For the first video, the application correctly predicted the name of the video, as it was present in the dataset. For our second video, it correctly ascertained that there was no match for the video, as it was not in the database. Feel free to try your own inputs for the querying, but only the ones in the test-input directory, or any frames (you can upload images) or snippets from the same videos will yield any results.
Here is also an interesting graphical representation of the current vectors in the Pinecone Database! It was made using Atlas Nomic.
# Setup
Just a disclaimer, unfortunately, in the face of our time (and budget) constraints, we do not have support for Mac or Linux users. This is because I say so. (Unfortunately like a shortsightedly architectured house, I only have Windows.)
Let's get started.
First, fork this repo, so you can have your own personal playground.
Then, use ```git clone ``` in the terminal of your preferred IDE.
In order to get up and running, you must have NodeJs installed. You can [download the latest version here](https://nodejs.org/en/download). This installation will include the Node Package Manager (npm).
You can check if NodeJs and npm are installed correctly by running ```node -v``` and ```npm -v``` respectively. If it does not recognize either as a command, check if Node is set as a PATH environmental variable. If it is not, [set Node as a PATH environmental variable.](https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/)
Once Node is set up, let's get our project dependencies set up with ```npm run install-all```.
That covers our NodeJs dependencies. Time for our Python dependencies.
Now, open your terminal. You can do this by hitting ```Win + R```, typing 'cmd' and hitting enter.
Run ```python -m venv viewvie ``` in your CLI to initialize the virtual environment.
This will create a virtual environment file (named viewvie) in your user directory.
Use ```viewvie\Scripts\activate``` in your CLI to activate the virtual environment. If your commands (even before your C: directory) start with ```(viewvie)```, the environment has been set up correctly!
This is where we will install dependencies for our embedder module.
Install the python dependencies by running each one in your CLI:
```pip install mediapipe```
```pip install opencv-python```
After these packages have installed, you can type ```deactivate``` in the CLI to deactivate the virtual environment.
Now, we can safely run ```npm run start-all``` from the 'viewvie' directory to start the application! You will see a ```http://localhost:5173/``` in your console - click it to open the application in your browser.
Of course, project is missing .env file with Pinecone API key. You can set this up by adding a ```.env``` file to the 'backend' directory, and adding ```PINECONE_API_KEY=''``` to it. Contact me for details on the Pinecone API key. =)
For now, we have a very limited pool of movies in our database. There is functionality in the application to add more, but we want to straighten out load balancing first before we let users test this out. Temporarily, we will provide sample video clips to test the applications. They will be labeled with the actual movie name (which you wouldn't actually know). Some will match movies in our database, and some will not. They can be downloaded in the 'test-input' directory.