{"id":26175566,"url":"https://github.com/jayleicn/tvqaplus","last_synced_at":"2025-04-14T21:33:06.484Z","repository":{"id":38253643,"uuid":"182856581","full_name":"jayleicn/TVQAplus","owner":"jayleicn","description":"[ACL 2020] PyTorch code for TVQA+: Spatio-Temporal Grounding for Video Question Answering","archived":false,"fork":false,"pushed_at":"2022-10-25T04:32:28.000Z","size":14663,"stargazers_count":128,"open_issues_count":2,"forks_count":24,"subscribers_count":9,"default_branch":"master","last_synced_at":"2025-03-28T09:42:55.114Z","etag":null,"topics":["dataset","pytorch","tvqa","video-question-answering"],"latest_commit_sha":null,"homepage":"http://tvqa.cs.unc.edu","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jayleicn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-04-22T19:45:26.000Z","updated_at":"2025-03-18T09:47:14.000Z","dependencies_parsed_at":"2023-01-19T15:31:03.461Z","dependency_job_id":null,"html_url":"https://github.com/jayleicn/TVQAplus","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FTVQAplus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FTVQAplus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FTVQAplus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jayleicn%2FTVQAplus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jayleicn","download_url":"https://codeload.github.com/jayleicn/TVQAplus/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248964266,"owners_count":21190498,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","pytorch","tvqa","video-question-answering"],"created_at":"2025-03-11T20:56:58.053Z","updated_at":"2025-04-14T21:33:06.456Z","avatar_url":"https://github.com/jayleicn.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"## TVQA+: Spatio-Temporal Grounding for Video Question Answering\n\n![qa_example](imgs/qa_example_pair.png)\n\nWe present the task of Spatio-Temporal Video Question Answering, which requires intelligent \nsystems to simultaneously retrieve relevant moments and detect referenced visual concepts \n(people and objects) to answer natural language questions about videos. \nWe first augment the [TVQA](http://tvqa.cs.unc.edu/) dataset with 310.8k bounding boxes, linking depicted objects to \nvisual concepts in questions and answers. \nWe name this augmented version as TVQA+.\nWe then propose Spatio-Temporal Answerer with Grounded Evidence (STAGE), \na unified framework that grounds evidence in both the spatial and temporal domains to \nanswer questions about videos. \nComprehensive experiments and analyses demonstrate the effectiveness of our framework and \nhow the rich annotations in our TVQA+ dataset can contribute to the question answering task. \nAs a side product, by performing this joint task, our model is able to produce more insightful \nintermediate results. \n\n\nIn this repository, we provide PyTorch Implementation of the STAGE model, along with basic \npreprocessing and evaluation code for TVQA+ dataset.\n\n\n[TVQA+: Spatio-Temporal Grounding for Video Question Answering](https://arxiv.org/abs/1904.11574)\u003cbr\u003e\n[Jie Lei](http://www.cs.unc.edu/~jielei/),  [Licheng Yu](http://www.cs.unc.edu/~licheng/), \n[Tamara L. Berg](http://Tamaraberg.com), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/). \n   [[PDF]](https://arxiv.org/abs/1904.11574) \n\n\n### Resources\n- Data: [TVQA+ dataset](http://tvqa.cs.unc.edu/download_tvqa_plus.html), please use this [new link](https://nlp.cs.unc.edu/data/jielei/tvqa/tvqa_public_html/download_tvqa_plus.html)\n- Website: [http://tvqa.cs.unc.edu](http://tvqa.cs.unc.edu)\n- Submission: [codalab evaluation server](https://competitions.codalab.org/competitions/22705)\n- Related works: [TVR (Moment Retrieval)](https://github.com/jayleicn/TVRetrieval), [TVC (Video Captioning)](https://github.com/jayleicn/TVCaption), [TVQA (Localized VideoQA)](https://github.com/jayleicn/TVQA)\n\n\n\n### Model\n- **STAGE Overview**. Spatio-Temporal Answerer with Grounded Evidence (STAGE), a unified framework that grounds evidence in both the spatial and temporal domains to answer questions about videos.  \n![model_overview](imgs/model_overview.png)\n\n\n- **Prediction Examples**\n![example_predictions](imgs/model_prediction.png) \n\n\n### Requirements\n- Python 2.7\n- PyTorch 1.1.0 (should work for 0.4.0 - 1.2.0)\n- tensorboardX\n- tqdm\n- h5py\n- numpy\n\n\n### Training and Evaluation\n1, Download and uncompress preprocessed features from \n[Google Drive](https://drive.google.com/open?id=1GnknXfs9qKE-WVaUgUeKfCTLHjyzqCHG).\n\n```\n\u0026 uncompress the file into project root directory, you should get a dir `tvqa_plus_stage_features` \ncontaining all the required feature files.\ncd $PROJECT_ROOT; tar -xf tvqa_plus_stage_features_new.tar.gz\n```\n[gdrive](https://github.com/prasmussen/gdrive) is a good tool to use for downloading the file. \n**The features are changed, you have to re-download the features if you have our previous version**\n\n2, Run in `debug` mode to test your environment, path settings:\n```\nbash run_main.sh debug\n```\n\n3, Train the full STAGE model:\n```\nbash run_main.sh --add_local\n```\nnote you will need around 30 GB of memory to load the data. Otherwise, you can additionally add `--no_core_driver` flag to stop loading \nall the features into memory. After training, you should be able to get ~72.00% QA Acc, which is comparable to the reported number.\nThe trained model and config file are stored at `${$PROJECT_ROOT}/results/${MODEL_DIR}`\n\n4, Inference\n```\nbash run_inference.sh --model_dir ${MODEL_DIR} --mode ${MODE}\n```\n`${MODE}` could be `valid` or `test`. After inference, you will get a `${MODE}_inference_predictions.json` \nfile in `${MODEL_DIR}`, which is similar to the sample prediction file here `eval/data/val_sample_prediction.json`.\n\n5, Evaluation\n```\ncd eval; python eval_tvqa_plus.py --pred_path ../results/${MODEL_DIR}/valid_inference_predictions.json --gt_path data/tvqa_plus_val.json\n```\nNote you can only evaluate val prediction here. \nTo evaluate test set, please follow instructions [here](http://tvqa.cs.unc.edu/leaderboard.html).  \n\n\n### Citation\n```\n@inproceedings{lei2019tvqa,\n  title={TVQA+: Spatio-Temporal Grounding for Video Question Answering},\n  author={Lei, Jie and Yu, Licheng and Berg, Tamara L and Bansal, Mohit},\n  booktitle={Tech Report, arXiv},\n  year={2019}\n}\n```\n\n### TODO\n1. [x] Add data preprocessing scripts (provided preprocessed features)\n2. [x] Add model and training scripts\n3. [x] Add inference and evaluation scripts\n\n\n### Contact\n- Dataset: faq-tvqa-unc [at] googlegroups.com\n- Model: Jie Lei, jielei [at] cs.unc.edu\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayleicn%2Ftvqaplus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjayleicn%2Ftvqaplus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjayleicn%2Ftvqaplus/lists"}