{"id":28677468,"url":"https://github.com/thunlp-mt/museg","last_synced_at":"2025-08-23T14:13:05.153Z","repository":{"id":298045199,"uuid":"990108084","full_name":"THUNLP-MT/MUSEG","owner":"THUNLP-MT","description":"Repo for paper \"MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding\".","archived":false,"fork":false,"pushed_at":"2025-06-09T04:32:19.000Z","size":2276,"stargazers_count":7,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-09T05:23:31.555Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/THUNLP-MT.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-05-25T14:19:47.000Z","updated_at":"2025-06-09T04:32:23.000Z","dependencies_parsed_at":"2025-06-09T05:34:01.856Z","dependency_job_id":null,"html_url":"https://github.com/THUNLP-MT/MUSEG","commit_stats":null,"previous_names":["thunlp-mt/museg"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/THUNLP-MT/MUSEG","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMUSEG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMUSEG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMUSEG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMUSEG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/THUNLP-MT","download_url":"https://codeload.github.com/THUNLP-MT/MUSEG/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/THUNLP-MT%2FMUSEG/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271751925,"owners_count":24814707,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-06-14T00:06:03.545Z","updated_at":"2025-08-23T14:13:05.130Z","avatar_url":"https://github.com/THUNLP-MT.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding\n\n[[📖 Paper](https://arxiv.org/abs/2505.20715)] [[🤗 MUSEG-7B](https://huggingface.co/Darwin-Project/MUSEG-7B)] [[🤗 MUSEG-3B](https://huggingface.co/Darwin-Project/MUSEG-3B)]\n[[🤖 MUSEG-7B](https://www.modelscope.cn/models/Darwin_Project/MUSEG-7B)] [[🤖 MUSEG-3B](https://www.modelscope.cn/models/Darwin_Project/MUSEG-3B)]\n\n## Introduction\n\nWe propose MUSEG 🌟, a novel RL-based method that enhances temporal understanding by introducing timestamp-aware multi-segment grounding. MUSEG enables MLLMs to align queries with multiple relevant video segments, promoting more comprehensive temporal reasoning ⏳. To facilitate effective learning, we design a customized RL training recipe with phased rewards that progressively guides the model toward temporally grounded reasoning. Extensive experiments on temporal grounding and time-sensitive video QA tasks demonstrate that MUSEG significantly outperforms existing methods and generalizes well across diverse temporal understanding scenarios 🚀.\n\n## Performance\n\n\u003cimg src=\"./images/performance.png\" style=\"zoom:85%;\"/\u003e\n\n## Environment Setup\n\n```bash\nconda create -n museg python=3.11\nconda activate museg\npip install -r requirements.txt\npip install flash_attn==2.7.4.post1\napt update\napt install openjdk-11-jdk\n```\n\n## Training\n\n### Data Preparation\n\n- Download videos of [Charades](https://prior.allenai.org/projects/charades) (480p version), and [E.T. Instruct 164k](https://huggingface.co/datasets/PolyU-ChenLab/ET-Instruct-164K) (only videos of activitynet, didemo, hacs and tacos are needed). You may also download the videos directly with the following commands.\n\n```bash\nwget https://ai2-public-datasets.s3-us-west-2.amazonaws.com/charades/Charades_v1_480.zip\nwget https://huggingface.co/datasets/PolyU-ChenLab/ET-Instruct-164K/resolve/main/videos/activitynet.tar.gz\nwget https://huggingface.co/datasets/PolyU-ChenLab/ET-Instruct-164K/resolve/main/videos/didemo.tar.gz\nwget https://huggingface.co/datasets/PolyU-ChenLab/ET-Instruct-164K/resolve/main/videos/hacs.tar.gz\nwget https://huggingface.co/datasets/PolyU-ChenLab/ET-Instruct-164K/resolve/main/videos/tacos.tar.gz\n```\n\n- Unzip and organize videos as follows.\n\n```plain\nMUSEG\n ├─ scripts\n ├─ src\n ├─ data\n └─ videos\n     ├─ charades\n     └─ et_instruct_164k\n         ├─ activitynet\n         ├─ didemo\n         ├─ hacs\n         └─ tacos\n```\n\n### Model Preparation\n\nWe use [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct) and [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) as our base models.\n\n### Start Training\n\n```bash\n./scripts/train_stage_1.sh\n./scripts/train_stage_2.sh\n```\n\nRemember to set path of checkpoints before training. It takes about 22 hours for MUSEG-7B stage 1 training and 27 hours for MUSEG-7B stage 2 training with 8 × A100(80G) GPUs.\n\n## Evaluation\n\n### Data Preparation\n\n- Download videos of [Charades](https://prior.allenai.org/projects/charades) (480p version), [Perception Test](https://github.com/google-deepmind/perception_test) (validation set), [THUMOS14](https://www.crcv.ucf.edu/THUMOS14/home.html) (validation set), [THUMOS15](http://www.thumos.info/home.html) (validation set), and [E.T. Bench](https://huggingface.co/datasets/PolyU-ChenLab/ETBench). You may also download the videos directly with the following commands.\n\n```bash\nwget https://ai2-public-datasets.s3-us-west-2.amazonaws.com/charades/Charades_v1_480.zip\nwget https://storage.googleapis.com/dm-perception-test/zip_data/valid_videos.zip\nwget https://storage.googleapis.com/thumos14_files/TH14_validation_set_mp4.zip\nwget http://storage.googleapis.com/www.thumos.info/thumos15_zips/thumos15_validation_complete.zip\nfiles=(\n    \"charades_sta\" \"cross_task\" \"ego4d_nlq\" \"hirest\" \"ht_step\" \"perception_test\" \"qa_ego4d\"\n    \"qvhighlights\" \"star\" \"summe\" \"thumos14\" \"thumos15\" \"tvsum\" \"youcook2\" \"youtube_highlights\"\n)\nfor file in \"${files[@]}\"; do\n    wget https://huggingface.co/datasets/PolyU-ChenLab/ETBench/resolve/main/videos/${file}.tar.gz\ndone\n```\n\n- Unzip and organize videos as follows.\n\n```plain\nMUSEG\n ├─ scripts\n ├─ src\n ├─ data\n └─ videos\n     ├─ charades\n     ├─ perception_test\n     ├─ thumos14\n     ├─ thumos15\n     └─ et_bench\n         ├─ charades_sta\n         ├─ cross_task\n         ├─ ego4d_nlq\n         ├─ hirest\n         └─ ...\n```\n\n- (Optional) Some of videos in E.T. Bench may not be loaded correctly by our evaluation script. You may run the following script to fix this.\n\n```bash\npython data/eval/et_bench/process_videos.py\n```\n\n### Model Preparation\n\nDownload MUSEG-7B [here](https://huggingface.co/Darwin-Project/MUSEG-7B) and MUSEG-3B [here](https://huggingface.co/Darwin-Project/MUSEG-3B).\n\n### Start Evaluation\n\n```bash\n# Charades-STA\n./scripts/eval_grounding_charades_sta.sh\n# Perception Test\n./scripts/eval_grounding_perception_test.sh\n# THUMOS14\n./scripts/eval_grounding_thumos14.sh\n# THUMOS15\n./scripts/eval_grounding_thumos15.sh\n# E.T. Bench\n./scripts/eval_et_bench.sh\n# E.T. Bench (Subset)\n./scripts/eval_et_bench_subset.sh\n```\n\n## Acknowledgements\n\nWe sincerely appreciate the contributions of the open-source community. We thank the following projects:\n\n[Open-R1-Video](https://github.com/Wang-Xiaodong1899/Open-R1-Video), [Video-R1](https://github.com/tulerfeng/Video-R1), [R1-V](https://github.com/Deep-Agent/R1-V), [E.T. Bench](https://github.com/PolyU-ChenLab/ETBench).\n\n## Citation\n\nIf you find our work helpful for your research, please consider citing our work.\n\n```plain\n@article{luo2025museg,\n    title={MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding}, \n    author={Fuwen Luo and Shengfeng Lou and Chi Chen and Ziyue Wang and Chenliang Li and Weizhou Shen and Jiyue Guo and Peng Li and Ming Yan and Ji Zhang and Fei Huang and Yang Liu},\n    journal={arXiv preprint arXiv:2505.20715},\n    year={2025}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fmuseg","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthunlp-mt%2Fmuseg","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthunlp-mt%2Fmuseg/lists"}