https://github.com/sizhky/ssbd
https://github.com/sizhky/ssbd
Last synced: about 2 months ago
JSON representation
- Host: GitHub
- URL: https://github.com/sizhky/ssbd
- Owner: sizhky
- License: apache-2.0
- Created: 2023-10-17T16:42:05.000Z (over 1 year ago)
- Default Branch: dev
- Last Pushed: 2023-10-23T18:45:47.000Z (over 1 year ago)
- Last Synced: 2025-02-08T11:44:47.593Z (3 months ago)
- Language: HTML
- Homepage: https://sizhky.github.io/ssbd/
- Size: 44.2 MB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: changelog.md
- License: LICENSE
Awesome Lists containing this project
README
# clip-video-classifier
## Install
``` sh
git clone https://github.com/sizhky/ssbd
cd ssbd
pip install -e .
```
# How to use
Use
``` bash
clip-video-classifier --help
clip-video-classifier COMMAND --help
```
to see help for all commands
------------------------------------------------------------------------
To train a transformer model on SSBD videos using CLIP as feature
extractor
1. First setup your annotations
``` bash
DATA_DIR="/mnt/347832F37832B388/ml-datasets/ssbd"
RAW_VIDEO_DIR="$DATA_DIR/ssbd-raw-videos"
clip-video-classifier setup-annotations $DATA_DIR
clip-video-classifier download-raw-videos $DATA_DIR/annotations.csv $RAW_VIDEO_DIR
clip-video-classifier setup-annotations $DATA_DIR --fill-gaps --videos-folder $RAW_VIDEO_DIR
```
2. Next extract frames for each video
``` bash
# change the num-frames-per-sec in the below script if needed
$ chmod +x scripts/extract_frames.sh
$ ./extract_frames.sh
```
3. Now extract embeddings for each `frames.tensor` file saved
``` bash
clip-video-classifier frames-to-embeddings "/mnt/347832F37832B388/ml-datasets/ssbd/ssbd-frames/5fps" "/mnt/347832F37832B388/ml-datasets/ssbd/ssbd-embeddings/5fps" "ViT-B/32" "cuda"
```
4. Finally you can run the notebook
`nbs/models/02_transformer_clip.ipynb` by pointing to the
approprirate ssbd-embeddings folder and setting the right
hyperparameters
# API
You can directly run the notebook `nbs/model/03_infer.ipynb` to load the
deeplearning model and make predictions on every 5 second intervals
or
You can launch a fastapi server
`uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload` and send a
post request like so
