https://github.com/ayeshaaaaaaaaa/video-description-and-summarization-using-blip-and-bart-models
This project processes videos by extracting frames, generating detailed visual descriptions for each frame using the BLIP model, and then summarizing these descriptions with the BART model.
https://github.com/ayeshaaaaaaaaa/video-description-and-summarization-using-blip-and-bart-models
bart blip video-description
Last synced: 4 months ago
JSON representation
This project processes videos by extracting frames, generating detailed visual descriptions for each frame using the BLIP model, and then summarizing these descriptions with the BART model.
- Host: GitHub
- URL: https://github.com/ayeshaaaaaaaaa/video-description-and-summarization-using-blip-and-bart-models
- Owner: Ayeshaaaaaaaaa
- Created: 2024-09-15T10:01:51.000Z (9 months ago)
- Default Branch: main
- Last Pushed: 2024-09-15T10:24:20.000Z (9 months ago)
- Last Synced: 2024-12-30T06:23:21.121Z (6 months ago)
- Topics: bart, blip, video-description
- Language: Jupyter Notebook
- Homepage:
- Size: 36.1 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Video-Description-and-Summarization-Using-BLIP-and-BART-Models
This project processes videos by extracting frames, generating detailed visual descriptions for each frame using the BLIP model, and then summarizing these descriptions with the BART model.
## Key features include:
**Frame Extraction:** Extracts 1 frame per second from the input video for efficient processing.
**Visual Captioning:** Uses Salesforce's BLIP image-captioning model to generate natural language descriptions for each extracted frame.
**Summarization:** Combines the generated frame descriptions and summarizes them into a cohesive video summary using Facebook's BART large CNN model.
## Workflow Explanation:
**Video Loading:** The video is loaded using OpenCV's cv2.VideoCapture function, which reads the video file specified by its path. The program checks whether the video has been successfully opened. If not, it exits with an error message.
**FPS and Frame Count Extraction:** The video's frames per second (FPS) and the total number of frames are extracted using OpenCV’s CAP_PROP_FPS and CAP_PROP_FRAME_COUNT. These values are then used to calculate the video’s total duration.
**Model Initialization:** Two models are loaded:
**BLIP Model (Salesforce/blip-image-captioning-base):** Used for generating natural language captions for individual frames.
**BART Model (facebook/bart-large-cnn):** Used for summarizing the textual descriptions generated by BLIP into a more concise and meaningful summary.
Both models are loaded into memory and moved to the appropriate device (GPU if available, otherwise CPU).
**Frame Extraction:** Frames are extracted at the rate of 1 frame per second from the video. This is done by reading each frame and selecting frames at intervals matching the FPS value (e.g., for a 30 FPS video, every 30th frame is selected). Extracted frames are stored in the frame_list.
**Frame Descriptions:** For each extracted frame, the BLIP model generates a visual description. This step calls the generate_description function (not provided in the code snippet, but assumed to exist), which processes the frame and outputs a caption for the content depicted in that frame.
**Combining Frame Descriptions:** Once all frames have been described, the individual descriptions are concatenated into a single large text block. This forms a comprehensive description of the video based on visual content across multiple frames.
**Summarization:** The combined text of frame descriptions is fed into the BART model. The model processes the text and generates a concise summary. The summary is printed out as the final description of the video.
## Output:
