{"id":19520155,"url":"https://github.com/osu-nlp-group/tablellama","last_synced_at":"2025-04-26T07:31:35.649Z","repository":{"id":207489584,"uuid":"719373912","full_name":"OSU-NLP-Group/TableLlama","owner":"OSU-NLP-Group","description":"[NAACL'24] Dataset, code and models for \"TableLlama: Towards Open Large Generalist Models for Tables\".","archived":false,"fork":false,"pushed_at":"2024-05-14T03:04:55.000Z","size":22431,"stargazers_count":127,"open_issues_count":9,"forks_count":13,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-04-04T10:03:49.302Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://osu-nlp-group.github.io/TableLlama/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSU-NLP-Group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-11-16T02:54:08.000Z","updated_at":"2025-03-12T06:32:09.000Z","dependencies_parsed_at":"2023-11-27T06:25:05.108Z","dependency_job_id":"1bcdf2f0-3296-484e-8a27-2bdb3807aa4c","html_url":"https://github.com/OSU-NLP-Group/TableLlama","commit_stats":null,"previous_names":["osu-nlp-group/tablellama"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FTableLlama","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FTableLlama/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FTableLlama/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSU-NLP-Group%2FTableLlama/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSU-NLP-Group","download_url":"https://codeload.github.com/OSU-NLP-Group/TableLlama/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250953590,"owners_count":21513396,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-11T00:24:06.165Z","updated_at":"2025-04-26T07:31:30.641Z","avatar_url":"https://github.com/OSU-NLP-Group.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003eTableLlama \u003cbr\u003e Towards Open Large Generalist Models for Tables\u003c/h1\u003e\n\n\u003cdiv align=\"center\"\u003e\n 🔥 🔥 🔥 This repo contains the code, data, and models for \u003ca href=\"https://arxiv.org/pdf/2311.09206.pdf\"\u003eTableLlama\u003c/a\u003e.\nCheck out our \u003ca href = \"https://osu-nlp-group.github.io/TableLlama/\"\u003e[Project Page]\u003c/a\u003e for more results and analysis!\n\u003c/div\u003e\n\n\u003cbr\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/OSU-NLP-Group/TableLlama/blob/main/imgs/tablellama_figure1.png\" width=\"100%\" title=\"Introduction Figure\"\u003e\n\u003c/div\u003e\nFigure 1: An overview of TableInstruct and TableLlama. TableInstruct includes a wide variety of realistic tables and tasks with instructions. We make the first step towards developing open-source generalist models for tables with TableInstruct and TableLlama.\n\n\u003cbr\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/OSU-NLP-Group/TableLlama/blob/main/imgs/tablellama_figure2.png\" width=\"100%\" title=\"Examplars Figure\"\u003e\n\u003c/div\u003e\nFigure 2: Illustration of three exemplary tasks: (a) Column type annotation. This task is to annotate the selected column with the correct semantic types. (b) Row population. This task is to populate rows given table metadata and partial row entities. (c) Hierarchical table QA. For subfigures (a) and (b), we mark candidates with red color in the \"task instruction\" part. The candidate set size can be hundreds to thousands in TableInstruct.\n\n\n\u003ch3\u003eRelease progress\u003c/h3\u003e\n\n- :ballot_box_with_check: Training Dataset for TableLlama (check `/data_v3` of 🤗 [TableInstruct Dataset](https://huggingface.co/datasets/osunlp/TableInstruct/)) \n- :ballot_box_with_check: TableLlama-7B model \n- :ballot_box_with_check: Code for Fine-tuning and Inference \n- :ballot_box_with_check: Evaluate Dataset of TableInstruct (check `/eval_data` of 🤗 [TableInstruct Dataset](https://huggingface.co/datasets/osunlp/TableInstruct/)) \n\u003c!-- - :white_large_square: Code for Fine-tuning and Centralized training (TODO) --\u003e\n\n\u003ch3\u003eUpdates\u003c/h3\u003e\n\n- 2024/3/13: Our paper has been accepted by NAACL 2024!\n- 2024/3/21: We refine the prompts of 4 out-of-domain evaluation datasets: FEVEROUS, HybridQA, WikiSQL and WikiTQ of [TableInstruct](https://huggingface.co/datasets/osunlp/TableInstruct/) and update the results. Check the new results!\n- 2024/3/21: We add the results of closed-source LLMs: GPT-3.5 and GPT-4.\n\n### Datasets and Models\nOur dataset and models are all available at Huggingface.\n\n🤗 [TableInstruct Dataset](https://huggingface.co/datasets/osunlp/TableInstruct/)\n                                       \t\n🤗 [TableLlama-7B](https://osu-nlp-group.github.io/TableLlama/)   \t\n\nThe model is fine-tuned with the TableInstruct dataset using LongLoRA (7B), fully fine-tuning version as the base model, which replaces the vanilla attention mechanism of the original Llama-2 (7B) with shift short attention. The training takes 9 days on a 48 80*A100 cluster. Check out our paper for more details.\n\nTableInstruct includes a comprehensive table-based instruction tuning dataset that covers a variety of real-world tables and realistic tasks. We include 14 datasets of 11 tasks in total. \n\nThe model is evaluated on 8 in-domain datasets of 8 tasks and 6 out-of-domain datasets of 4 tasks.\n\n\n## **Introduction**\nWe introduce TableLlama and TableInstruct: the FIRST open-source generalist LLM and instruction tuning dataset for tables. The TableLlama model is trained on TableInstruct Dataset, a meticulously curated instruction tuning dataset for tables. TableLlama is tuned on **2.6 million** table-based task data, and can handle up to **8K** context!\n\n\n## **Installation**\n\nClone this repository and install the required packages:\n\n```bash\ngit clone https://github.com/OSU-NLP-Group/TableLlama.git\ncd TableLlama\npip install -r requirements.txt\npip install flash-attn --no-build-isolation\n```\n\n## **Training and Inference**\n\n### **Fine-tuning**\n\nTo train the 7B model, run:\n\n```bash\ntorchrun --nproc_per_node=8 supervised_fine_tune.py  \\\n        --model_name_or_path $MODEL_DIR \\\n        --bf16 True \\\n        --output_dir $OUTPUT_DIR  \\\n        --model_max_length 8192 \\\n        --use_flash_attn True \\\n        --data_path $DATA_DIR \\\n        --cache_dir /ML-A800/hf_cache  \\\n        --low_rank_training False \\\n        --num_train_epochs 2  \\\n        --per_device_train_batch_size 3     \\\n        --per_device_eval_batch_size 2     \\\n        --gradient_accumulation_steps 1     \\\n        --evaluation_strategy \"no\"     \\\n        --save_strategy \"steps\"     \\\n        --save_steps 2000     \\\n        --save_total_limit 4     \\\n        --learning_rate 2e-5     \\\n        --weight_decay 0.0     \\\n        --warmup_ratio 0.03     \\\n        --lr_scheduler_type \"cosine\"     \\\n        --logging_steps 1     \\\n        --deepspeed \"/ds_configs/stage2.json\" \\\n        --tf32 True \\\n        --run_name $RUN_NAME\n```\n\n**Addressing OOM**\nTo train the 7B model with super large data size, if you encounter OOM issue, we provide code for streaming. You can run:\n```bash\ntorchrun --nproc_per_node=8 supervised_fine_tune_stream.py  \\\n        --model_name_or_path $MODEL_DIR \\\n        --bf16 True \\\n        --output_dir $OUTPUT_DIR  \\\n        --model_max_length 8192 \\\n        --use_flash_attn True \\\n        --data_path $DATA_DIR \\\n        --gpu_size $GPU_SIZE \\\n        --data_size $DATA_SIZE \\\n        --cache_dir /ML-A800/hf_cache  \\\n        --low_rank_training False \\\n        --num_train_epochs 2  \\\n        --per_device_train_batch_size 3     \\\n        --per_device_eval_batch_size 2     \\\n        --gradient_accumulation_steps 1     \\\n        --evaluation_strategy \"no\"     \\\n        --save_strategy \"steps\"     \\\n        --save_steps 2000     \\\n        --save_total_limit 4     \\\n        --learning_rate 2e-5     \\\n        --weight_decay 0.0     \\\n        --warmup_ratio 0.03     \\\n        --lr_scheduler_type \"cosine\"     \\\n        --logging_steps 1     \\\n        --deepspeed \"/ds_configs/stage2.json\" \\\n        --tf32 True \\\n        --run_name $RUN_NAME\n       \n```\n\n### **Inference**\n```bash\npython3 inference_rel_extraction_col_type.py  \\\n        --base_model $MODEL_DIR \\\n        --context_size 8192 \\\n        --max_gen_len 128 \\\n        --flash_attn True \\\n        --input_data_file  /test_data/test_col_type.json \\\n        --output_data_file $OUTPUT_DIR/col_type_pred.json\n```\n\n## **Evaluation**\n\nThe folder `eval_scripts` includes evaluation scripts for all the in-domain test sets. To run the script, take HiTab (hierarchical table QA task) as an example:\n\n```bash\ncd eval_scripts\npython evaluate_hitab.py --file_pred $OUTPUT_DIR/hitab_pred.json\n```\n\n## Prompt Format\n\n```\nBelow is an instruction that describes a task, paired with an input that provides further context. Write a response that\nappropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Question:\n{question}\n\n### Response:\n```\n\n\n- The instruction is designed to point out the task and give a detailed task description.\n- The input is designed to provide the information about the table. We concatenate table metadata (if any) such as the Wikipedia page title, section\ntitle and table caption with the serialized table as table input. We use '[TLE]' to represent the beginning of the table metadata, and use '[TAB]' to represent the beginning of the serialized table.\n- The question is to accommodate all the information the model needs to complete the task and prompt the model to generate an answer.\n- Task prompts examples (For more example prompts for other tasks, please refer to Appendix E in our paper.)\n\n\u003cbr\u003e\n\u003cdiv align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/OSU-NLP-Group/TableLlama/blob/main/imgs/hitab.png\" width=\"100%\" title=\"hierarchical table qa\"\u003e\n  \u003cimg src=\"https://github.com/OSU-NLP-Group/TableLlama/blob/main/imgs/fetaqa.png\" width=\"100%\" title=\"hierarchical table qa\"\u003e\n  \u003cimg src=\"https://github.com/OSU-NLP-Group/TableLlama/blob/main/imgs/hybridqa.png\" width=\"100%\" title=\"hierarchical table qa\"\u003e\n  \u003cimg src=\"https://github.com/OSU-NLP-Group/TableLlama/blob/main/imgs/tabfact.png\" width=\"100%\" title=\"hierarchical table qa\"\u003e\n\u003c/div\u003e\n\n**Note:** \n\n- If you directly use our model for inference on your data, please make sure you organize the data in the same way as the examples shown above and in our paper Appendix. The performance will vary significantly along with the prompts.\n\n\n## **Citation**\n\nPlease cite our paper if you use our data, model or code. Please also kindly cite the original dataset papers. \n\n```\n@misc{zhang2023tablellama,\n  title={TableLlama: Towards Open Large Generalist Models for Tables}, \n  author={Tianshu Zhang and Xiang Yue and Yifei Li and Huan Sun},\n  year={2023},\n  eprint={2311.09206},\n  archivePrefix={arXiv},\n  primaryClass={cs.CL}\n}\n```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Ftablellama","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosu-nlp-group%2Ftablellama","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosu-nlp-group%2Ftablellama/lists"}