{"id":31977508,"url":"https://github.com/OpenGVLab/ChartAst","last_synced_at":"2025-10-14T21:48:40.436Z","repository":{"id":215805006,"uuid":"738542732","full_name":"OpenGVLab/ChartAst","owner":"OpenGVLab","description":"[ACL 2024] ChartAssistant is a chart-based vision-language model for universal chart comprehension and reasoning.","archived":false,"fork":false,"pushed_at":"2024-09-07T09:31:16.000Z","size":16220,"stargazers_count":118,"open_issues_count":9,"forks_count":9,"subscribers_count":6,"default_branch":"main","last_synced_at":"2025-06-06T12:07:21.468Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OpenGVLab.png","metadata":{"files":{"readme":"Readme.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE_llama2","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-01-03T13:21:03.000Z","updated_at":"2025-06-03T06:26:51.000Z","dependencies_parsed_at":"2024-01-09T04:14:02.095Z","dependency_job_id":"e958b487-af34-4a55-97e8-89b2a835e0c2","html_url":"https://github.com/OpenGVLab/ChartAst","commit_stats":null,"previous_names":["opengvlab/chartast"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/OpenGVLab/ChartAst","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FChartAst","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FChartAst/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FChartAst/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FChartAst/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OpenGVLab","download_url":"https://codeload.github.com/OpenGVLab/ChartAst/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OpenGVLab%2FChartAst/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279021374,"owners_count":26087023,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-14T02:00:06.444Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-10-14T21:48:31.920Z","updated_at":"2025-10-14T21:48:40.420Z","avatar_url":"https://github.com/OpenGVLab.png","language":"Python","funding_links":[],"categories":["📊 Chart Understanding \u0026 Generation"],"sub_categories":["Chart-to-Code \u0026 Reproducibility"],"readme":"# ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning (ACL 2024)\n\nThis is the PyTorch implementation of the paper **ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning**, the paper is available at https://arxiv.org/abs/2401.02384\n\nWe have developed ChartAssistant, which uses large-scale chart data to align and instruction tuning. The model possesses powerful mathematical computation capabilities and achieves state-of-the-art performance on multiple datasets without the need for downstream dataset finetuning.\n\n**This repo is built upon [LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory/tree/main)**\n\n## ChartSFT\nWe have released the ChartSFT in https://huggingface.co/datasets/FanqingM/ChartAssistant 🔥🔥🔥\n\n## Note\n\n- **Some people may have mistakenly tested our chartqa dataset. We have made the chartqa test results public as chartqa_all.json.**\n\n\n- 5.20 We have released the ChartSFT in https://huggingface.co/datasets/FanqingM/ChartAssistant 🔥🔥🔥\n  Here is some example:\n  - MathQA:\n    ```\n    {\n    \"image\": \"/mnt/petrelfs/share_data/mengfanqing1/mengfanqing/plotQA/train/png/png/35651.png\",\n    \"conversations\": [\n        {\n            \"from\": \"human\",\n            \"value\": \"Please use commandline to solve the math question: What is the difference between the highest and the second highest % of children under 5 who received treatment of Malaria ?\"\n        },\n        {\n            \"from\": \"gpt\",\n            \"value\": \"{'step1': {'func1': 'select', 'arg1': 'malaria', 'output1': ['10.000', '0.900']}, 'step2': {'func2': 'numpy.sort', 'arg2': 'output1', 'output2': 'sorted_arr'}, 'step3': {'func3': 'getitem', 'arg3':['sorted_arr', '-1'], 'output3': 'maximum'}, 'step4': {'func4': 'numpy.delete', 'arg4': ['sorted_arr', '-1'], 'output4': 'new_arr'}, 'step5': {'func5': 'getitem', 'arg5': ['new_arr', '-1'], 'output5': 'second_maximum'}, 'step6': {'func6': 'numpy.subtract', 'arg6': ['maximum', 'second_maximum'], 'output6': 'diff'}}\"\n        }\n    ]},\n    ```\n  - ReferQA:\n  \n    ```\n    {\n            \"image\": \"/mnt/petrelfs/share_data/luquanfeng/referring_box/train/img/52342_1_arrow.png\",\n            \"conversations\": [\n                {\n                    \"from\": \"human\",\n                    \"value\": \"Please use commandline to solve the math question about the referring box: What is the median years of the legend represented by the blue arrow per amount(current us$)?\"\n                },\n                {\n                    \"from\": \"gpt\",\n                    \"value\": \"{'step1': {'func1': 'select', 'arg1': 'ecuador', 'output1': ['139370000.000', '160050000.000', '165410000.000', '174120000.000', '214220000.000']}, 'step2': {'func2': 'numpy.median', 'arg2': 'outpu\n    t1', 'output2': 'median'}}\"\n                }\n            ]\n        },\n    ```\n  \n  - Arxiv QA:\n  \n    ```\n    {\n          \"image\": \"/mnt/petrelfs/share_data/mengfanqing1/mengfanqing/arxiv/Img_QA/vbar/1806.09244_table_0.png\",\n          \"conversations\": [\n              {\n                  \"from\": \"human\",\n                  \"value\": \"Please answer my question based on the chart: How many counties are included in each dataset?\"\n              },\n              {\n                  \"from\": \"gpt\",\n                  \"value\": \"The number of counties included in each dataset are as follows: \\n- Brazil-Soybean: 1529 counties \\n- US-Soybean: 1814 counties \\n- US-Maize: 2204 counties.\"\n              }\n          ]\n      },\n    ```\n  \n    \n  \n    \n\n- 2.15 We update the [paper](https://arxiv.org/abs/2401.02384),with better performance and more experiments and corrected experimental results.\n  \n- 1.11: The ChartAssistant, which has undergone two-stage training on ChartSFT, has been open-sourced. You can download it through the following link.\n  - https://pan.baidu.com/s/1t0QPLDfULNovnYKtsQxjOQ  password: 10el\n  - [HuggingFace](https://huggingface.co/FanqingM/ChartAssistant) : put consolidated.00-of-02.model.pth and consolidated.01-of-02.model.pth in one directory, and replace pretrained_path in the scipt as it. \n\n\n- 1.10: We update the paper(ChartAssistant.pdf), primarily making updates to the model, correcting some errors in the article, and providing more detailed explanations. \n\n## ChartAssisstant\n\nCharts play a vital role in data visualization, understanding data patterns, and informed decision-making. However, their unique combination of graphical elements (e.g., bars, lines) and textual components (e.g., labels, legends) poses challenges for general-purpose multimodal models. While vision-language models trained on chart data excel in comprehension, they struggle with generalization. To address these challenges, we propose ChartAssistant, a chart-based vision-language model for universal chart comprehension and reasoning. ChartAssistant leverages ChartSFT, a comprehensive dataset covering diverse chart-related tasks with basic (e.g. bars and pies) and specialized (e.g. radars, and bubbles) chart types. It undergoes a two-stage training process, starting with pre-training on chart-to-table parsing to align chart and text, followed by multitask instruction-following fine-tuning. This approach enables ChartAssistant to achieve competitive performance across various chart tasks. **Experimental results demonstrate significant performance gains over the state-of-the-art UniChart and Chartllama method, especially outperforming them on real-world chart data with zero-shot setting.** \n\n\n\n![image-20240104143625786](./demo.png)\n\n## Environment\nIt is same as [LLaMA2-Accessory](https://github.com/Alpha-VLLM/LLaMA2-Accessory/tree/main)\n\n## Inference\n\nreplace pretrained_path as the pretrained model path\n```\nsh accessory/exps/finetune/mm/test.sh\n# Please use the params in the test.sh\n# run accessory/single_turn_eval.py\n```\n\n## Training\n```\nsh accessory/exps/finetune/mm/chart.sh\n# run accessory/main_finetune.py\n```\n## Gradio demo\n```\nsh accessory/demo/start.sh\n```\n\n\n\n\n## Concat\nif you have any questions about this work, you can email Fanqing Meng using mengfanqing33@gmail.com or just by wechat: mfq2052063742\n\n## To Do List\n\n- [x] Create the git repository.\n\n- [x] Open source the model and model weight.\n\n- [x] Open source the inference script.\n\n- [x] Open source the dataset (ChartSFT).\n\n  \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenGVLab%2FChartAst","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FOpenGVLab%2FChartAst","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FOpenGVLab%2FChartAst/lists"}