{"id":26175575,"url":"https://github.com/jayleicn/moment_detr","last_synced_at":"2025-04-07T08:24:34.420Z","repository":{"id":39413662,"uuid":"387856300","full_name":"jayleicn/moment_detr","owner":"jayleicn","description":"[NeurIPS 2021] Moment-DETR code and QVHighlights dataset","archived":false,"fork":false,"pushed_at":"2024-04-18T09:14:11.000Z","size":36026,"stargazers_count":294,"open_issues_count":28,"forks_count":49,"subscribers_count":8,"default_branch":"main","last_synced_at":"2025-03-28T09:42:54.979Z","etag":null,"topics":["pytorch","video-retrieval"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2107.09609","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jayleicn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-07-20T16:47:39.000Z","updated_at":"2025-03-28T08:55:59.000Z","dependencies_parsed_at":"2023-01-21T23:00:57.968Z","dependency_job_id":null,"html_url":"https://github.com/jayleicn/moment_detr","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2Fmoment_detr","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2Fmoment_detr/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2Fmoment_detr/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2Fmoment_detr/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jayleicn","download_url":"https://codeload.github.com/jayleicn/moment_detr/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247616377,"owners_count":20967379,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["pytorch","video-retrieval"],"created_at":"2025-03-11T20:57:00.344Z","updated_at":"2025-04-07T08:24:34.398Z","avatar_url":"https://github.com/jayleicn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Moment-DETR\n\n[QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries](https://arxiv.org/abs/2107.09609), NeurIPS 2021\n\n[Jie Lei](http://www.cs.unc.edu/~jielei/), \n[Tamara L. Berg](http://tamaraberg.com/), [Mohit Bansal](http://www.cs.unc.edu/~mbansal/)\n\nThis repo contains a copy of QVHighlights dataset for moment retrieval and highlight detections. For details, please check [data/README.md](data/README.md)\nThis repo also hosts the Moment-DETR model (see overview below), a new model that predicts moment coordinates and saliency scores end-to-end based on a given text query. This released code supports pre-training, fine-tuning, and evaluation of Moment-DETR on the QVHighlights datasets. It also supports running prediction on your own raw videos and text queries. \n\n\n![model](./res/model_overview.png)\n\n\n## Table of Contents\n\n* [Getting Started](#getting-started)\n    * [Prerequisites](#prerequisites)\n    * [Training](#training)\n    * [Inference](#inference)\n    * [Pretraining and Finetuning](#pretraining-and-finetuning)\n    * [Evaluation and Codalab Submission](#evaluation-and-codalab-submission)\n    * [Train Moment-DETR on your own dataset](#train-moment-detr-on-your-own-dataset)\n* [Demo: Run predictions on your own videos and queries](#run-predictions-on-your-own-videos-and-queries)\n* [Acknowledgement](#acknowledgement)\n* [LICENSE](#license)\n\n\n\n## Getting Started \n\n### Prerequisites\n0. Clone this repo\n\n```\ngit clone https://github.com/jayleicn/moment_detr.git\ncd moment_detr\n```\n\n1. Prepare feature files\n\nDownload [moment_detr_features.tar.gz](https://drive.google.com/file/d/1Hiln02F1NEpoW8-iPZurRyi-47-W2_B9/view?usp=sharing) (8GB), \nextract it under project root directory:\n```\ntar -xf path/to/moment_detr_features.tar.gz\n```\nThe features are extracted using Linjie's [HERO_Video_Feature_Extractor](https://github.com/linjieli222/HERO_Video_Feature_Extractor). \nIf you want to use your own choices of video features, please download the raw videos from this [link](https://nlp.cs.unc.edu/data/jielei/qvh/qvhilights_videos.tar.gz).\n\n2. Install dependencies.\n\nThis code requires Python 3.7, PyTorch, and a few other Python libraries. \nWe recommend creating conda environment and installing all the dependencies as follows:\n```\n# create conda env\nconda create --name moment_detr python=3.7\n# activate env\nconda actiavte moment_detr\n# install pytorch with CUDA 11.0\nconda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch\n# install other python packages\npip install tqdm ipython easydict tensorboard tabulate scikit-learn pandas\n```\nThe PyTorch version we tested is `1.9.0`.\n\n### Training\n\nTraining can be launched by running the following command:\n```\nbash moment_detr/scripts/train.sh \n```\nThis will train Moment-DETR for 200 epochs on the QVHighlights train split, with SlowFast and Open AI CLIP features. The training is very fast, it can be done within 4 hours using a single RTX 2080Ti GPU. The checkpoints and other experiment log files will be written into `results`. For training under different settings, you can append additional command line flags to the command above. For example, if you want to train the model without the saliency loss (by setting the corresponding loss weight to 0):\n```\nbash moment_detr/scripts/train.sh --lw_saliency 0\n```\nFor more configurable options, please checkout our config file [moment_detr/config.py](moment_detr/config.py).\n\n### Inference\nOnce the model is trained, you can use the following command for inference:\n```\nbash moment_detr/scripts/inference.sh CHECKPOINT_PATH SPLIT_NAME  \n``` \nwhere `CHECKPOINT_PATH` is the path to the saved checkpoint, `SPLIT_NAME` is the split name for inference, can be one of `val` and `test`.\n\n### Pretraining and Finetuning\nMoment-DETR utilizes ASR captions for weakly supervised pretraining. To launch pretraining, run:\n```\nbash moment_detr/scripts/pretrain.sh \n```  \nThis will pretrain the Moment-DETR model on the ASR captions for 100 epochs, the pretrained checkpoints and other experiment log files will be written into `results`. With the pretrained checkpoint, we can launch finetuning from a pretrained checkpoint `PRETRAIN_CHECKPOINT_PATH` as:\n```\nbash moment_detr/scripts/train.sh  --resume ${PRETRAIN_CHECKPOINT_PATH}\n```\nNote that this finetuning process is the same as standard training except that it initializes weights from a pretrained checkpoint. \n\n\n### Evaluation and Codalab Submission\nPlease check [standalone_eval/README.md](standalone_eval/README.md) for details.\n\n\n### Train Moment-DETR on your own dataset\nTo train Moment-DETR on your own dataset, please prepare your dataset annotations following the format \nof QVHighlights annotations in [data](./data), and extract features using [HERO_Video_Feature_Extractor](https://github.com/linjieli222/HERO_Video_Feature_Extractor).\nNext copy the script [moment_detr/scripts/train.sh](./moment_detr/scripts/train.sh) and modify the dataset specific parameters \nsuch as annotation and feature paths. Now you are ready to use this script for training as described in [Training](#training).\n\n\n## Run predictions on your own videos and queries\nYou may also want to run Moment-DETR model on your own videos and queries. \nFirst you need to add a few libraries for feature extraction to your environment. Before this, you should have already installed PyTorch and other libraries for running Moment-DETR following instuctions in previous sections.\n```bash\npip install ffmpeg-python ftfy regex\n```\nNext, run the example provided in this repo:\n```bash\nPYTHONPATH=$PYTHONPATH:. python run_on_video/run.py\n```\nThis will load the Moment-DETR model [checkpoint](run_on_video/moment_detr_ckpt/model_best.ckpt) trained with CLIP image and text features, and make predictions for the video [RoripwjYFp8_60.0_210.0.mp4](run_on_video/example/RoripwjYFp8_60.0_210.0.mp4) with its associated query in [run_on_video/example/queries.jsonl](run_on_video/example/queries.jsonl).\nThe output will look like the following:\n```\nBuild models...\nLoading feature extractors...\nLoading CLIP models\nLoading trained Moment-DETR model...\nRun prediction...\n------------------------------idx0\n\u003e\u003e query: Chef makes pizza and cuts it up.\n\u003e\u003e video_path: run_on_video/example/RoripwjYFp8_60.0_210.0.mp4\n\u003e\u003e GT moments: [[106, 122]]\n\u003e\u003e Predicted moments ([start_in_seconds, end_in_seconds, score]): [\n    [49.967, 64.9129, 0.9421], \n    [66.4396, 81.0731, 0.9271], \n    [105.9434, 122.0372, 0.9234], \n    [93.2057, 103.3713, 0.2222], \n    ..., \n    [45.3834, 52.2183, 0.0005]\n   ]\n\u003e\u003e GT saliency scores (only localized 2-sec clips): \n    [[2, 3, 3], [2, 3, 3], ...]\n\u003e\u003e Predicted saliency scores (for all 2-sec clip): \n    [-0.9258, -0.8115, -0.7598, ..., 0.0739, 0.1068]   \n```\nYou can see the 3rd ranked moment `[105.9434, 122.0372]` matches quite well with the ground truth of `[106, 122]`, with a confidence score of `0.9234`.\nYou may want to refer to [data/README.md](data/README.md) for more info about how the ground-truth is organized.\nYour predictions might slightly differ from the predictions here, depends on your environment.\n\nTo run predictions on your own videos and queries, please take a look at the `run_example` function inside the [run_on_video/run.py](run_on_video/run.py) file.\n\n\n## Acknowledgement\nWe thank [Linjie Li](https://scholar.google.com/citations?user=WR875gYAAAAJ\u0026hl=en) for the helpful discussions.\nThis code is based on [detr](https://github.com/facebookresearch/detr) and [TVRetrieval XML](https://github.com/jayleicn/TVRetrieval). We used resources from [mdetr](https://github.com/ashkamath/mdetr), [MMAction2](https://github.com/open-mmlab/mmaction2), [CLIP](https://github.com/openai/CLIP), [SlowFast](https://github.com/facebookresearch/SlowFast) and [HERO_Video_Feature_Extractor](https://github.com/linjieli222/HERO_Video_Feature_Extractor). We thank the authors for their awesome open-source contributions. \n\n## LICENSE\nThe annotation files are under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) license, see [./data/LICENSE](data/LICENSE). All the code are under [MIT](https://opensource.org/licenses/MIT) license, see [LICENSE](./LICENSE).\n \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayleicn%2Fmoment_detr","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjayleicn%2Fmoment_detr","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayleicn%2Fmoment_detr/lists"}