{"id":28100750,"url":"https://github.com/poloclub/tsr-convstem","last_synced_at":"2025-05-13T18:38:51.672Z","repository":{"id":207429263,"uuid":"700883836","full_name":"poloclub/tsr-convstem","owner":"poloclub","description":"High-Performance Transformers for Table Structure Recognition Need Early Convolutions","archived":false,"fork":false,"pushed_at":"2024-04-03T01:33:57.000Z","size":246,"stargazers_count":27,"open_issues_count":2,"forks_count":2,"subscribers_count":6,"default_branch":"main","last_synced_at":"2024-05-12T00:47:37.652Z","etag":null,"topics":["receptive-field","table-structure-recognition","transformers"],"latest_commit_sha":null,"homepage":"https://arxiv.org/abs/2311.05565","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/poloclub.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-10-05T13:37:32.000Z","updated_at":"2024-04-15T09:40:58.000Z","dependencies_parsed_at":"2024-04-03T02:48:54.298Z","dependency_job_id":null,"html_url":"https://github.com/poloclub/tsr-convstem","commit_stats":null,"previous_names":["poloclub/tsr-convstem"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poloclub%2Ftsr-convstem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poloclub%2Ftsr-convstem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poloclub%2Ftsr-convstem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/poloclub%2Ftsr-convstem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/poloclub","download_url":"https://codeload.github.com/poloclub/tsr-convstem/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254005008,"owners_count":21998167,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["receptive-field","table-structure-recognition","transformers"],"created_at":"2025-05-13T18:38:51.081Z","updated_at":"2025-05-13T18:38:51.660Z","avatar_url":"https://github.com/poloclub.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# High-Performance Transformers for Table Structure Recognition Need Early Convolutions\n\n[![arxiv badge](https://img.shields.io/badge/arXiv-2311.05565-red)](https://arxiv.org/abs/2311.05565)\n[![license](https://img.shields.io/badge/License-MIT-success)](https://github.com/poloclub/wizmap/blob/main/LICENSE)\n\n[High-Performance Transformers for Table Structure Recognition Need Early Convolutions](https://arxiv.org/abs/2311.05565). ShengYun Peng, Seongmin Lee, Xiaojing Wang, Rajarajeswari Balasubramaniyan, Duen Horng Chau. In *NeurIPS 2023 Second Table Representation Learning Workshop*, 2023. (Oral)\n\n📖 \u003ca href=\"https://arxiv.org/abs/2311.05565\"\u003e Research Paper\u003c/a\u003e \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\n🚀 \u003ca href=\"https://shengyun-peng.github.io/tsr-convstem\"\u003e Project Page\u003c/a\u003e \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp; \n\n\n\u003cp align=\"center\"\u003e\n    \u003cimg src=\"imgs/pipeline.png\" alt=\"drawing\" width=\"100%\"/\u003e\n\u003c/p\u003e\n\nTable structure recognition (TSR) aims to convert tabular images into a machine-readable format, where a visual encoder extracts image features and a textual decoder generates table-representing tokens. Existing approaches use classic convolutional neural network (CNN) backbones for the visual encoder and transformers for the textual decoder. However, this hybrid CNN-Transformer architecture introduces a complex visual encoder that accounts for nearly half of the total model parameters, markedly reduces both training and inference speed, and hinders the potential for self-supervised learning in TSR. In this work, we design a lightweight visual encoder for TSR without sacrificing expressive power. We discover that a convolutional stem can match classic CNN backbone performance, with a much simpler model. The convolutional stem strikes an optimal balance between two crucial factors for high-performance TSR: a higher receptive field (RF) ratio and a longer sequence length. This allows it to \"see\" an appropriate portion of the table and \"store\" the complex table structure within sufficient context length for the subsequent transformer. \n\n\u003e Our latest work [UniTable](https://github.com/poloclub/unitable) has been fully released, achieving SOTA performance on four of the largest table recognition datasets! We have also released the first-of-its-kind Jupyter Notebook of the entire inference pipeline, which can fully digitalize your tabular image to HTML!\n\n## News\n`Oct. 2023` - Paper accepted by [NeurIPS'23 Table Representation Learning Workshop](https://table-representation-learning.github.io/)\n\n`Oct. 2023` - Paper selected as [oral](https://openreview.net/group?id=NeurIPS.cc/2023/Workshop/TRL)\n\n## Get Started\n1. Prepare PubTabNet dataset available [here](https://github.com/ibm-aur-nlp/PubTabNet/tree/master#getting-data)\n2. Change the \"pubtabnet_dir\" in [Makefile](./Makefile) to \"your path to PubTabNet\"\n3. Set up venv\n```bash\nmake .venv_done\n```\n\n## Training, Testing \u0026 Evaluation\n1. Train an instance of visual encoder with ResNet-18\n```bash\nmake experiments/r18_e2_d4_adamw/.done_train_structure\n```\n2. Test + Compute teds score\n```bash\nmake experiments/r18_e2_d4_adamw/.done_teds_structure\n```\n3. All models in ablations are defined in \"Experiment Configurations\" section of [Makefile](./Makefile). Replace \"r18_e2_d4_adamw\" with any other configuration for training and testing.\n\n## Citation\n```bibtex\n@inproceedings{peng2023high,\n  title={High-Performance Transformers for Table Structure Recognition Need Early Convolutions},\n  author={Peng, Anthony and Lee, Seongmin and Wang, Xiaojing and Balasubramaniyan, Rajarajeswari Raji and Chau, Duen Horng},\n  booktitle={NeurIPS 2023 Second Table Representation Learning Workshop},\n  year={2023}\n}\n```\n## Contact\nIf you have any questions, feel free to [open an issue](https://github.com/poloclub/tsr-convstem/issues/new) or contact [Anthony Peng](https://shengyun-peng.github.io/) (CS PhD @Georgia Tech).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoloclub%2Ftsr-convstem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpoloclub%2Ftsr-convstem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpoloclub%2Ftsr-convstem/lists"}