https://github.com/drewthomasson/offlineyoutube
Offline Youtube with advanced vector searching
https://github.com/drewthomasson/offlineyoutube
pip youtube youtube-downloader yt-dlp-gui
Last synced: 22 days ago
JSON representation
Offline Youtube with advanced vector searching
- Host: GitHub
- URL: https://github.com/drewthomasson/offlineyoutube
- Owner: DrewThomasson
- License: mit
- Created: 2024-10-28T16:09:08.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-11-07T19:58:33.000Z (6 months ago)
- Last Synced: 2025-04-17T00:18:00.929Z (28 days ago)
- Topics: pip, youtube, youtube-downloader, yt-dlp-gui
- Language: Python
- Homepage:
- Size: 19 MB
- Stars: 31
- Watchers: 3
- Forks: 6
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# **Offline YouTube Video Search Application**
This application allows users to **extract transcripts from YouTube videos**, **upload their own video/audio files**, **create searchable vector databases**, and **perform semantic searches** using a **Gradio web interface** or **command-line interface (CLI)**. It's powered by `faster-whisper` for transcription, `FAISS` for vector search, and `sentence-transformers` for text embeddings.
---
## **Features**
- Extract transcripts from individual videos, playlists, and entire channels.
- **Upload your own video or audio files for processing.**
- Automatically detect playlists, channels, and individual video links.
- Automatically download video thumbnails.
- Store transcripts and create a searchable vector database.
- Perform semantic searches on video content.
- Supports **Gradio web interface** and **CLI** for flexible usage.
- Easily add more videos or your own files to the dataset.
---
## **Web Interface**
### **Add Videos Tab**
- **Enter playlist, channel, and/or video URLs (comma-separated).**
- **Upload your own video/audio files.**
- **Option to process entire channels when a channel URL is provided.**
- **Option to keep videos stored locally or not.**
### **Search Tab**
- **Enter your search query to find relevant snippets.**
- **View top relevant videos with thumbnails and play local videos if available.**
- **View detailed results with timestamps and direct links.**
---
## **Installation**
Ensure you have Python installed (>= 3.8). Then, pip install:
(Requires Python 3.10 for Apple Silicon Macs)
```bash
pip install offlineyoutube
```
---
## **Usage**
The app provides **two ways to interact**:
1. **Gradio Web Interface**
2. **Command-Line Interface (CLI)**
### **1. Running the Gradio Web Interface**
Launch the web interface:
```bash
offlineyoutube ui
```
or simply:
```bash
offlineyoutube
```
Then, open the URL (usually `http://127.0.0.1:7860`) in your browser.
#### **Gradio Interface Tabs:**
- **Add Videos:**
- Enter playlist URLs, channel URLs, and/or individual video URLs (comma-separated).
- **Upload your own video or audio files for processing.**
- **Option to process entire YouTube channels when a channel URL is provided.**
- **Option to keep videos stored locally or not.**
- The app will automatically detect whether each link is a playlist, channel, or a video.
- Videos and uploaded files will be transcribed, and the database will be updated with the content.
- **Search:**
- Enter search queries to find relevant snippets from the video transcripts.
- Results are ranked based on semantic similarity and include video thumbnails.
- **If local videos are available, you can play them directly in the interface.**
---
### **2. Command-Line Interface (CLI)**
The CLI provides more flexibility for programmatic use.
#### **Commands Overview**
Use the `--help` command to view available commands and examples:
```bash
offlineyoutube --help
```
**Output:**
```
usage: offlineyoutube [-h] {add,search,ui} ...
YouTube Video Search Application
positional arguments:
{add,search,ui} Available commands
add Add videos to the database
search Search the video database
ui Run the Gradio web interface
optional arguments:
-h, --help Show this help message and exit
Examples:
# Add videos from a playlist and keep videos locally
offlineyoutube add --input "https://www.youtube.com/playlist?list=YOUR_PLAYLIST_ID" --keep_videos
# Add specific videos without keeping videos locally
offlineyoutube add --input "https://www.youtube.com/watch?v=VIDEO_ID1,https://www.youtube.com/watch?v=VIDEO_ID2"
# Add videos from a channel (process entire channel)
offlineyoutube add --input "https://www.youtube.com/channel/CHANNEL_ID" --process_channel
# Search the database with a query
offlineyoutube search --query "Your search query" --top_k 5
# Run the Gradio web interface
offlineyoutube ui
```
---
### **Examples of CLI Usage**
#### **1. Adding Videos**
- **Add Playlists and Videos:**
```bash
offlineyoutube add --input "https://www.youtube.com/playlist?list=YOUR_PLAYLIST_ID,https://www.youtube.com/watch?v=VIDEO_ID"
```
- **Add Specific Videos Without Keeping Them Locally:**
```bash
offlineyoutube add --input "https://www.youtube.com/watch?v=dQw4w9WgXcQ,https://www.youtube.com/watch?v=9bZkp7q19f0"
```
- **Add Videos from a Channel (Process Entire Channel):**
```bash
offlineyoutube add --input "https://www.youtube.com/channel/CHANNEL_ID" --process_channel
```
- **Add Videos and Keep Videos Stored Locally:**
```bash
offlineyoutube add --input "https://www.youtube.com/watch?v=VIDEO_ID" --keep_videos
```
#### **2. Searching the Database**
- **Perform a Search:**
```bash
offlineyoutube search --query "machine learning tutorials" --top_k 5
```
---
### **How It Works**
1. **Adding Videos and Uploaded Files:**
- The app accepts a list of links and automatically detects whether each link is a playlist, channel, or an individual video.
- **You can upload your own video or audio files for processing.**
- It downloads video audio (or uses uploaded files) and transcribes it using `faster-whisper`.
- Thumbnails are downloaded and saved locally.
- The transcript data is saved in `datasets/transcript_dataset.csv`.
- A vector database is updated using FAISS with embeddings generated by `sentence-transformers`.
2. **Incremental Updating:**
- Videos and uploaded files are processed one by one, and the dataset and vector database are updated incrementally.
- This ensures efficient processing, especially when dealing with large datasets.
3. **Searching the Database:**
- When a query is entered, the app computes its embedding and searches the FAISS index for relevant video snippets.
- The top results are displayed with thumbnails, titles, and links to the videos.
- **If local videos are available, you can play them directly in the interface.**
---
### **FAQ**
#### **1. How do I add multiple playlists, channels, and videos at once?**
Simply provide a comma-separated list of URLs, and the app will automatically detect and process each link:
```bash
offlineyoutube add --input "https://www.youtube.com/playlist?list=PLAYLIST_ID1,https://www.youtube.com/watch?v=VIDEO_ID,https://www.youtube.com/channel/CHANNEL_ID"
```
If you want to process entire channels, make sure to include the `--process_channel` flag:
```bash
offlineyoutube add --input "https://www.youtube.com/channel/CHANNEL_ID" --process_channel
```
#### **2. How can I upload my own video or audio files for processing?**
In the Gradio web interface, navigate to the **Add Videos** tab. Use the **"Upload your own video/audio files"** option to upload one or multiple files. The app will process these files and add them to the database.
#### **3. Why aren’t new videos or uploaded files showing up in search results?**
Ensure that the videos or files have been fully processed and that the vector database has been updated. The app handles this automatically, but processing may take time for large videos, playlists, or channels.
#### **4. How do I prevent videos from being stored locally?**
By default, the app keeps videos stored locally. To change this behavior, use the `--keep_videos` flag and set it to `False`:
```bash
offlineyoutube add --input "VIDEO_OR_PLAYLIST_URL" --keep_videos False
```
In the Gradio interface, uncheck the **"Keep videos stored locally"** option in the **Add Videos** tab.
#### **5. Can I process entire YouTube channels?**
Yes! Use the `--process_channel` flag when adding videos via the CLI:
```bash
offlineyoutube add --input "https://www.youtube.com/channel/CHANNEL_ID" --process_channel
```
In the Gradio interface, check the **"Process entire channel when a channel URL is provided"** option in the **Add Videos** tab.
#### **6. Can I search the database without launching the Gradio interface?**
Yes! Use the `search` command via the CLI:
```bash
offlineyoutube search --query "Your query" --top_k 5
```
---
### **Project Structure**
```
.
├── app.py # Main application script (Gradio + CLI)
├── functions.py # Helper functions for transcription, FAISS, etc.
├── datasets/
│ ├── transcript_dataset.csv # CSV file storing transcripts
│ └── vector_index.faiss # FAISS vector index
├── thumbnails/ # Folder for storing video thumbnails
├── videos/ # Folder for storing downloaded videos (if keep_videos is True)
├── tmp/ # Temporary folder for videos (if keep_videos is False)
├── uploaded_files/ # Folder for storing uploaded files
```
---
### **Known Limitations**
- **Processing Time:** Transcribing videos and generating embeddings can be time-consuming, especially for long videos, large playlists, or channels.
- **Storage Requirements:** Keeping videos stored locally will require additional disk space. Use the `--keep_videos False` option if storage is a concern.
- **Large Datasets:** As the dataset grows, querying may take longer. Consider optimizing the FAISS index for very large datasets.
---
### **Contributing**
Feel free to fork the repository, open issues, or submit pull requests if you'd like to contribute to this project.
---
### **License**
This project is licensed under the MIT License. See the LICENSE file for details.
---
### **Acknowledgments**
- **faster-whisper** for fast transcription.
- **FAISS** for efficient vector search.
- **Gradio** for the interactive web interface.
- **yt-dlp** for downloading video content.
---