{"id":44369141,"url":"https://github.com/toloka/wsdmcup2023","last_synced_at":"2026-02-11T19:11:18.846Z","repository":{"id":103175593,"uuid":"533740389","full_name":"Toloka/WSDMCup2023","owner":"Toloka","description":"Toloka Visual Question Answering Challenge at WSDM Cup 2023","archived":false,"fork":false,"pushed_at":"2024-05-01T14:57:20.000Z","size":5493,"stargazers_count":31,"open_issues_count":1,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-09-05T03:26:37.758Z","etag":null,"topics":["challenge","competition","shared-task","toloka","visual-question-answering","wsdmcup","wsdmcup2023"],"latest_commit_sha":null,"homepage":"https://toloka.ai/challenges/wsdm2023/","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Toloka.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE-APACHE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-09-07T11:44:10.000Z","updated_at":"2025-04-14T17:54:23.000Z","dependencies_parsed_at":"2024-05-01T16:17:27.467Z","dependency_job_id":"c412906f-7475-4cdd-8015-15d0fac55838","html_url":"https://github.com/Toloka/WSDMCup2023","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Toloka/WSDMCup2023","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Toloka%2FWSDMCup2023","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Toloka%2FWSDMCup2023/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Toloka%2FWSDMCup2023/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Toloka%2FWSDMCup2023/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Toloka","download_url":"https://codeload.github.com/Toloka/WSDMCup2023/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Toloka%2FWSDMCup2023/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29341800,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-11T18:58:20.535Z","status":"ssl_error","status_checked_at":"2026-02-11T18:56:44.814Z","response_time":97,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["challenge","competition","shared-task","toloka","visual-question-answering","wsdmcup","wsdmcup2023"],"created_at":"2026-02-11T19:11:18.190Z","updated_at":"2026-02-11T19:11:18.840Z","avatar_url":"https://github.com/Toloka.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Toloka Visual Question Answering Challenge at WSDM Cup 2023\n\nWe challenge you with a visual question answering task! **Given an image and a textual question, draw the bounding box around the object correctly responding to that question.**\n\n| Question | Image and Answer |\n| --- | --- |\n| What do you use to hit the ball? | \u003cimg src=\"https://tlkfrontprod.azureedge.net/portal-production/static/uploaded/images/KUsGAc_eqdMcNxkBXzzl/KUsGAc_eqdMcNxkBXzzl_webp_1280_x2.webp\" width=\"228\" alt=\"What do you use to hit the ball?\"\u003e |\n| What do people use for cutting? | \u003cimg src=\"https://tlkfrontprod.azureedge.net/portal-production/static/uploaded/images/brXEVYckNLfQKcfNu4DF/brXEVYckNLfQKcfNu4DF_webp_1280_x2.webp\" width=\"228\" alt=\"What do people use for cutting?\"\u003e |\n| What do we use to support the immune system and get vitamin C? | \u003cimg src=\"https://tlkfrontprod.azureedge.net/portal-production/static/uploaded/images/HQ0A-ZvZCGCmYfTs83K7/HQ0A-ZvZCGCmYfTs83K7_webp_1280_x2.webp\" width=\"228\" alt=\"What do we use to support the immune system and get vitamin C?\"\u003e |\n\n## Links\n\n- **Competition:** \u003chttps://toloka.ai/challenges/wsdm2023\u003e\n- **CodaLab:** \u003chttps://codalab.lisn.upsaclay.fr/competitions/7434\u003e\n- **Dataset:** \u003chttps://doi.org/10.5281/zenodo.7057740\u003e\n\n## Citation\n\nPlease cite the challenge results or dataset description as follows.\n\n- Ustalov D., Pavlichenko N., Koshelev S., Likhobaba D., and Smirnova A. [Toloka Visual Question Answering Benchmark](https://arxiv.org/abs/2309.16511). 2023. arXiv: [2309.16511 [cs.CV]](https://arxiv.org/abs/2309.16511).\n\n```bibtex\n@inproceedings{TolokaWSDMCup2023,\n  author    = {Ustalov, Dmitry and Pavlichenko, Nikita and Koshelev, Sergey and Likhobaba, Daniil and Smirnova, Alisa},\n  title     = {{Toloka Visual Question Answering Benchmark}},\n  year      = {2023},\n  eprint    = {2309.16511},\n  eprinttype = {arxiv},\n  eprintclass = {cs.CV},\n  language  = {english},\n}\n```\n\n## Dataset\n\nOur dataset consists of the images associated with textual questions. One entry (instance) in our dataset is a question-image pair labeled with the ground truth coordinates of a bounding box containing the visual answer to the given question. The images were obtained from a CC BY-licensed subset of the Microsoft Common Objects in Context dataset, [MS COCO](https://cocodataset.org/). All data labeling was performed on the Toloka crowdsourcing platform, \u003chttps://toloka.ai/\u003e. We release the entire dataset under the CC BY license:\n\n- Zenodo: \u003chttps://doi.org/10.5281/zenodo.7057740\u003e\n- Hugging Face Hub: \u003chttps://huggingface.co/datasets/toloka/WSDMCup2023\u003e\n- Kaggle: \u003chttps://www.kaggle.com/datasets/dustalov/toloka-wsdm-cup-2023-vqa\u003e\n- GitHub Packages: \u003chttps://github.com/Toloka/WSDMCup2023/pkgs/container/wsdmcup2023\u003e\n\nLicensed under the Creative Commons Attribution 4.0 License. See LICENSE-CC-BY.txt file for more details.\n\n## Zero-Shot Baselines\n\nWe provide zero-shot baselines in `zeroshot_baselines` folder. All notebooks are made to run in Colab\n\n#### YOLOR + CLIP\n\nThis baseline was provided to participants of WSDM Cup 2023 Challenge. First, it uses a detection model, YOLOR, to generate candidate rectangles. Then, it applies CLIP to measure the similarity between the question and a part of the image bounded by each candidate rectangle. To make a prediction, it uses the candidate with the highest similarity. This baseline method achieves **IoU = 0.21** on private test subset.\n\nLicensed under the Apache License, Version 2.0. See LICENSE-APACHE.txt file for more details.\n\n#### OVSeg + SAM\n\nAnother zero-shot baseline, called OVSeg, utilizes SAM as a proposal generator instead of MaskFormer in the original setup. This approach achieves **IoU = 0.35** on the private test subset.\n\nLicensed under the Creative Commons Attribution 4.0 License.\n\n#### OFA + SAM\n\nLast one is primarily based on OFA, combined with bounding box correction using SAM. To solve the task, we followed a two-step zero-shot setup.\n\nFirst, we address the Visual Question Answering, where the model is given a prompt `{question} Name an object in the picture` along with an image. The model provides the name of a clue object to the question.\n\nIn the second step, an object corresponding to the answer from the previous step is annotated using the prompt `which region does the text \"{answer}\" describe?`, resulting in IoU = 0.42.\n\nSubsequently, with the obtained bounding boxes, SAM generates the corresponding masks for the annotated object, which are then transformed into bounding boxes. This enabled us to achieve **IoU = 0.45** with this baseline.\n\nLicensed under the Apache License, Version 2.0.\n\n## Crowdsourcing Baseline\n\nWe evaluated how well non-expert human annotators can solve our task by running a dedicated round of crowdsourcing annotations on the [Toloka](https://toloka.ai/) crowdsourcing platform. We found them to tackle this task successfully without knowing the ground truth. On all three subsets of our data, the average IoU value was 0.87 \u0026pm; 0.01, which we consider as a *strong human baseline* for our task. Krippendorff's \u0026alpha; coefficients for the public test was 0.68 and for the private test was 0.66, showing the decent agreement between the responses; we used 1 \u0026minus; IoU as the distance metric when calculating the \u0026alpha; coefficient. We selected the bounding boxes which were the most similar to the ground truth data to indicate the upper bound of non-expert annotation quality; `*_crowd_baseline.csv` files contain these responses.\n\nLicensed under the Creative Commons Attribution 4.0 License. See LICENSE-CC-BY.txt file for more details.\n\n## Reproduction\n\nThe final score will be evaluated on the private test dataset during Reproduction phase. We kindly ask you to create a docker image and share it with us by December 19th 23:59 AoE in [this form](https://docs.google.com/forms/d/e/1FAIpQLSfWt-c2OvfXPcOQ-J7EmIh1AOAjiojH7RT33bRgchI4evtvLw/viewform?usp=sf_link). We put an instruction how to create a docker image in `reproduction` directory. \n\nWe will run your solution on a machine with one Nvidia A100 80 GB GPU, 16 CPU cores, and 200 GB of RAM. Your Docker image must perform the inference in at most 3 hours on this machine. In other words, the docker run command must finish in 3 hours.\n\nDon't hesitate to contact us at research@toloka.ai if you have any questions or suggestions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoloka%2Fwsdmcup2023","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftoloka%2Fwsdmcup2023","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftoloka%2Fwsdmcup2023/lists"}