{"id":28511247,"url":"https://github.com/esborisova/scivqa","last_synced_at":"2025-07-06T17:05:19.597Z","repository":{"id":296413367,"uuid":"814027016","full_name":"esborisova/SciVQA","owner":"esborisova","description":"SciVQA: Scientific Visual Question Answering shared task","archived":false,"fork":false,"pushed_at":"2025-06-23T11:45:11.000Z","size":192769,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-06-23T12:38:17.056Z","etag":null,"topics":["chart-understanding","shared-task","vqa"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/esborisova.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-12T07:47:16.000Z","updated_at":"2025-06-23T11:45:15.000Z","dependencies_parsed_at":"2025-06-23T12:39:22.182Z","dependency_job_id":null,"html_url":"https://github.com/esborisova/SciVQA","commit_stats":null,"previous_names":["esborisova/scivqa"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/esborisova/SciVQA","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esborisova%2FSciVQA","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esborisova%2FSciVQA/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esborisova%2FSciVQA/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esborisova%2FSciVQA/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/esborisova","download_url":"https://codeload.github.com/esborisova/SciVQA/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/esborisova%2FSciVQA/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":263315908,"owners_count":23447587,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chart-understanding","shared-task","vqa"],"created_at":"2025-06-08T23:38:23.082Z","updated_at":"2025-07-03T11:31:28.325Z","avatar_url":"https://github.com/esborisova.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# \u003cimg src=\"SciVQA_logo.gif\" alt=\"drawing\" width=\"300\"/\u003e\n**Scientific Visual Question Answering (SciVQA)** shared task was orginised as part of the [Scholarly Document Processing workshop (SDP)](https://sdproc.org/2025/) at ALC 2025. In this challenge, participants developed multimodal QA systems using images of scientific figures, their captions, associated natural language QA pairs, and optionally additional metadata. The competition was hosted on the [Codabench platform](https://www.codabench.org/competitions/5904/). \n\nThis repository stores the code used for constructing the SciVQA dataset and developing the competition baseline. \n\n# Data\n\nThe SciVQA dataset comprises **3000** images of real-world figures extracted from English scientific publications in Computational Linguistics available in arXiv and ACL Anthology. The images are collected from the two pre-existing datasets: \n- __ACL-Fig \u003cimg src='https://img.shields.io/badge/arXiv-2023-darkred'\u003e \u003ca href='https://arxiv.org/abs/2301.12293'\u003e\u003cimg src='https://img.shields.io/badge/PDF-blue'\u003e\u003c/a\u003e \u003ca href='https://huggingface.co/datasets/citeseerx/ACL-fig'\u003e\u003cimg src='https://img.shields.io/badge/Dataset-gold'\u003e\u003c/a\u003e__\n\n- __SciGraphQA \u003cimg src='https://img.shields.io/badge/arXiv-2023-darkred'\u003e \u003ca href='https://arxiv.org/abs/2308.03349'\u003e\u003cimg src='https://img.shields.io/badge/PDF-blue'\u003e\u003c/a\u003e \u003ca href='https://huggingface.co/datasets/alexshengzhili/SciGraphQA-295K-train?row=0'\u003e\u003cimg src='https://img.shields.io/badge/Dataset-gold'\u003e\u003c/a\u003e__\n\nEach figure is availabe as PNG and associated with 7 QA pairs according to the custom schema (see below). All figures are automatically annotated using the Gemini 1.5-flash model and then manually validated by graduate students with Computational Linguistics background. SciVQA contains **21000** QA pairs in total. The language of all QA pairs is English. \n\n**The dataset is publicly available on [Hugging Face](https://huggingface.co/datasets/katebor/SciVQA).** \n\n# QA pair types schema\n\n\u003cimg src=\"qa_pairs_schema.png\" alt=\"drawing\" width=\"550\"/\u003e\n\n- **Closed-ended** - it is possible to answer a question based only on a given data source, i.e., an image or an image and a caption. No additional resources such as the main text of a publication, other documents/figures/tables, etc. are required.\n- **Unanswerable** - it is not possible to infer an answer based solely on a given data source.\n- **Infinite answer set** - there are no predefined answer options., e.g., \"What is the sum of Y and Z?\".\n- **Finite answer set** - associated with a limited range of answer options. Such QA pairs fall into two subcategories:\n  - **Binary** - require a yes/no or true/false answer, e.g., \"Is the percentage of positive tweets equal to 15%?\".\n  - **Non-binary** - require to choose from a set of **four** predefined answer options where one or more are correct, e.g., \"What is the maximum value of the green bar at the threshold equal to 10?\" Answer options: \"A: 5, B: 10, C: 300, D: None of the above\".\n- **Visual** - address or incorporate information on one or more of the **six visual attributes** of a figure, i.e., *shape*, *size*, *position*, *height*, *direction* or *colour*. E.g., \"In the bottom left figure, what is the value of the blue line at an AL of 6?\". Here the visual aspects are: position (bottom left), colour (blue), and shape (line).\n- **Non-visual** - do not involve any of the six visual aspects of a figure defined in our schema, e.g., \"What is the minimum value of X?\".\n\n# Repository structure\n```\n    ├── data               # qa prompts, annotation guidelines, csv files with human answers and predictions from the baseline\n    ├── src               \n    │   ├── data_pred      # data preparation scripts\n    │   ├── baseline       # code for baseline\n    │   ├── eval           # scoring script      \n    └──         \n```\n\n\n# Cite\n```bibtex\n@inproceedings{borisova-scivqa-2025,\n  title = \"{SciVQA} 2025: Overview of the First Scientific Visual Question Answering Shared Task\",\n  author = \"Borisova, Ekaterina and Rauscher, Nikolas and Rehm, Georg\",\n  booktitle = \"Proceedings of the 5th Workshop on Scholarly  Document Processing (SDP)\",\n  year = \"2025\",\n  address = \"Vienna, Austria\",\n  comment = \"accepted\" }\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesborisova%2Fscivqa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fesborisova%2Fscivqa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fesborisova%2Fscivqa/lists"}