{"id":13787816,"url":"https://github.com/li-xirong/w2vvpp","last_synced_at":"2025-05-12T02:30:28.615Z","repository":{"id":36320528,"uuid":"200593330","full_name":"li-xirong/w2vvpp","owner":"li-xirong","description":"W2VV++: A fully deep learning solution for ad-hoc video search","archived":false,"fork":false,"pushed_at":"2024-07-25T10:56:37.000Z","size":110,"stargazers_count":28,"open_issues_count":2,"forks_count":15,"subscribers_count":8,"default_branch":"master","last_synced_at":"2024-11-18T01:39:14.669Z","etag":null,"topics":["avs","deep-learning","query-representation","video-retrieval"],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/li-xirong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-05T06:14:03.000Z","updated_at":"2023-12-10T14:26:28.000Z","dependencies_parsed_at":"2024-11-18T01:46:05.863Z","dependency_job_id":null,"html_url":"https://github.com/li-xirong/w2vvpp","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/li-xirong%2Fw2vvpp","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/li-xirong%2Fw2vvpp/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/li-xirong%2Fw2vvpp/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/li-xirong%2Fw2vvpp/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/li-xirong","download_url":"https://codeload.github.com/li-xirong/w2vvpp/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253662519,"owners_count":21944090,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["avs","deep-learning","query-representation","video-retrieval"],"created_at":"2024-08-03T21:00:31.830Z","updated_at":"2025-05-12T02:30:28.314Z","avatar_url":"https://github.com/li-xirong.png","language":"Python","funding_links":[],"categories":["Implementations"],"sub_categories":[],"readme":"# w2vvpp\nW2VV++: A fully deep learning solution for ad-hoc video search. The code assumes [video-level CNN features](https://github.com/xuchaoxi/video-cnn-feat) have been extracted. \n\n## Requirements\n* Ubuntu 16.04\n* cuda 10\n* python 2.7.12\n* PyTorch 1.2.0\n* tensorboard 1.14.0\n* numpy 1.16.4\n\nWe used virtualenv to setup a deep learning workspace that supports PyTorch. Run the following script to install the required packages.\n\n```\nvirtualenv --system-site-packages ~/w2vvpp\nsource ~/w2vvpp/bin/activate\npip install -r requirements.txt\ndeactivate\n```\n\n## Get started\n\n### Data\n\nThe sentence encoding network for W2VV++, namely ```MultiScaleTxtEncoder```, needs a pretrained word2vec (w2v) model. In this work, we use a w2v trained on English tags associated with 30 million Flickr images.  Run the following script to download the Flickr w2v model and extract the folder at $HOME/VisualSearch/. The zipped model is around 3.1 gigabytes, so the download may take a while.\n\n```bash\nROOTPATH=$HOME/VisualSearch\nmkdir -p $ROOTPATH; cd $ROOTPATH\n\n# download and extract pre-trained word2vec\nwget http://lixirong.net/data/w2vv-tmm2018/word2vec.tar.gz\ntar zxf word2vec.tar.gz\n```\n\nThe following three datasets are used for training, validation and testing: tgif-msrvtt10k, tv2016train and iacc.3. For more information about these datasets, please see https://github.com/li-xirong/avs.\n\n\n**Video feature data**\n+ 4096-dim resnext101-resnet152: [tgif-msrvtt10k](http://lixirong.net/data/mm2019/tgif-msrvtt10k-mean_resnext101-resnet152.tar.gz)(1.6G), [tv2016train](http://lixirong.net/data/mm2019/tv2016train-mean_resnext101-resnet152.tar.gz)(2.9M), [iacc.3](http://lixirong.net/data/mm2019/iacc.3-mean_resnext101-resnet152.tar.gz)(4.7G)\n\n```bash\n# get visual features per dataset\nwget http://lixirong.net/data/mm2019/tgif-msrvtt10k-mean_resnext101-resnet152.tar.gz\nwget http://lixirong.net/data/mm2019/tv2016train-mean_resnext101-resnet152.tar.gz\nwget http://lixirong.net/data/mm2019/iacc.3-mean_resnext101-resnet152.tar.gz\n```\n\n**Sentence data**\n+ Sentences: [tgif-msrvtt10k](http://lixirong.net/data/mm2019/tgif-msrvtt10k-sent.tar.gz), [tv2016train](http://lixirong.net/data/mm2019/tv2016train-sent.tar.gz)\n+ TRECVID 2016 / 2017 / 2018 AVS topics and ground truth: [iacc.3](http://lixirong.net/data/mm2019/iacc.3-avs-topics.tar.gz)\n+ TRECVID 2019 AVS topics and ground truth: [v3c1](http://lixirong.net/data/tv19/v3c1-avs-topics.tar.gz)\n\n```bash\n# get sentences\nwget http://lixirong.net/data/mm2019/tgif-msrvtt10k-sent.tar.gz\nwget http://lixirong.net/data/mm2019/tv2016train-sent.tar.gz\nwget http://lixirong.net/data/mm2019/iacc.3-avs-topics.tar.gz\n```\n\n\n**Pre-trained models**\n+ [w2vvpp_resnext101_resnet152_subspace_v190916.pth.tar](http://lixirong.net/data/mm2019/w2vvpp_resnext101_resnet152_subspace_v190916.pth.tar)(240 MB)\n\nModel | TV16 | TV17 | TV18 | OVERALL\n|--- | ---| ---| ---| ---|\n|w2vvpp_resnext101_resnet152_subspace_v190916 | 0.162 | 0.223 | 0.101 | 0.162 |\n\nModel | TV19|\n|--- | ---|\n|w2vvpp_resnext101_resnet152_subspace_v190916 | 0.139|\n\nNote that due to SGD based training, the performance of a single model learned from scratch might differ slightly from those reported in the ACMMM'19 paper. For better and stable performance, ensemble is suggested.\n\n### Scripts for training, testing and evaluation\n\nBefore executing the following scripts, please check if the environment (data, software, etc) is ready by running [test_env.py](test_env.py):\n```bash\npython test_env.py\n\ntest_rootpath (__main__.TestSuite) ... ok\ntest_test_data (__main__.TestSuite) ... ok\ntest_train_data (__main__.TestSuite) ... ok\ntest_val_data (__main__.TestSuite) ... ok\ntest_w2v_dir (__main__.TestSuite) ... ok\n\n----------------------------------------------------------------------\nRan 5 tests in 0.001s\n\nOK\n```\n\n#### Do everything from sratch\n\n```bash\nsource ~/w2vvpp/bin/activate\n# build vocabulary on the training set\n./do_build_vocab.sh\n\n# train w2vvpp on tgif-msrvtt10k based on config \"w2vvpp_resnext101-resnet152_subspace\"\ntrainCollection=tgif-msrvtt10k\nvalCollection=tv2016train\nval_set=setA\nmodel_config=w2vvpp_resnext101-resnet152_subspace\n\n./do_train.sh $trainCollection $valCollection $val_set $model_config\n\n# test w2vvpp on iacc.3\nmodel_path=$rootpath/$trainCollection/w2vvpp_train/$valCollection/$val_set/$model_config/runs_0/model_best.pth.tar\nsim_name=$trainCollection/$valCollection/$val_set/$model_config/runs_0\n\n./do_test.sh iacc.3 $model_path $sim_name tv16.avs.txt,tv17.avs.txt,tv18.avs.txt\n\ncd tv-avs-eval\n./do_eval.sh iacc.3 tv16 $sim_name\n./do_eval.sh iacc.3 tv17 $sim_name\n./do_eval.sh iacc.3 tv18 $sim_name\n```\n\n#### Test and evaluate a pre-trained model\n\nAssume the model has been placed at the following path:\n\n```bash\n~/VisualSearch/w2vvpp/w2vvpp_resnext101_resnet152_subspace_v190916.pth.tar\n```\n\n```bash\n\n# apply a pre-trained w2vvpp model on iacc.3 for answering tv16 / tv17 / tv18 queries\n\n./do_test.sh iacc.3 ~/VisualSearch/w2vvpp/w2vvpp_resnext101_resnet152_subspace_v190916.pth.tar w2vvpp_resnext101_resnet152_subspace_v190916 tv16.avs.txt,tv17.avs.txt,tv18.avs.txt\n\n# evaluate the performance\ncd tv-avs-eval\n./do_eval.sh iacc.3 tv16 w2vvpp_resnext101_resnet152_subspace_v190916 # tv16 infAP: 0.162\n./do_eval.sh iacc.3 tv17 w2vvpp_resnext101_resnet152_subspace_v190916 # tv17 infAP: 0.223\n./do_eval.sh iacc.3 tv18 w2vvpp_resnext101_resnet152_subspace_v190916 # tv18 infAP: 0.101\n```\n\n```bash\n# apply a pre-trained w2vvpp model on v3c1 for answering tv19 queries\n./do_test.sh v3c1 ~/VisualSearch/w2vvpp/w2vvpp_resnext101_resnet152_subspace_v190916.pth.tar w2vvpp_resnext101_resnet152_subspace_v190916 tv19.avs.txt\n# evaluate the performance\ncd tv-avs-eval\n./do_eval.sh v3c1 tv19 w2vvpp_resnext101_resnet152_subspace_v190916 # tv19 infAP: 0.139\n```\n\n## Tutorials\n\n1. [Use a pre-trained w2vv++ model to encode a given sentence](tutorial.ipynb)\n\n\n## Citation\n\n```\n@inproceedings{mm19-w2vvpp,\ntitle = {{W2VV}++: Fully Deep Learning for Ad-hoc Video Search},\nauthor = {Xirong Li and Chaoxi Xu and Gang Yang and Zhineng Chen and Jianfeng Dong},\nyear = {2019},\nbooktitle = {ACMMM},\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fli-xirong%2Fw2vvpp","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fli-xirong%2Fw2vvpp","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fli-xirong%2Fw2vvpp/lists"}