{"id":20305518,"url":"https://github.com/vitae-transformer/samtext","last_synced_at":"2026-03-05T22:30:54.812Z","repository":{"id":160474621,"uuid":"634776031","full_name":"ViTAE-Transformer/SAMText","owner":"ViTAE-Transformer","description":"The official repo for the technical report \"Scalable Mask Annotation for Video Text Spotting\"","archived":false,"fork":false,"pushed_at":"2023-05-03T02:19:52.000Z","size":780,"stargazers_count":17,"open_issues_count":2,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-16T03:32:07.875Z","etag":null,"topics":["dataset","deep-learning","sam","scene-text-spotting","segment-anything-model","video-text-spotting"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ViTAE-Transformer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-05-01T06:45:31.000Z","updated_at":"2024-04-11T18:32:45.000Z","dependencies_parsed_at":"2023-06-04T10:00:31.642Z","dependency_job_id":null,"html_url":"https://github.com/ViTAE-Transformer/SAMText","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ViTAE-Transformer/SAMText","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FSAMText","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FSAMText/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FSAMText/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FSAMText/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ViTAE-Transformer","download_url":"https://codeload.github.com/ViTAE-Transformer/SAMText/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ViTAE-Transformer%2FSAMText/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30152836,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T21:15:50.531Z","status":"ssl_error","status_checked_at":"2026-03-05T21:15:11.173Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dataset","deep-learning","sam","scene-text-spotting","segment-anything-model","video-text-spotting"],"created_at":"2024-11-14T17:08:48.191Z","updated_at":"2026-03-05T22:30:54.793Z","avatar_url":"https://github.com/ViTAE-Transformer.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ch1 align=\"center\"\u003e Scalable Mask Annotation for Video Text Spotting\u003ca href=\"https://arxiv.org/abs/2305.01443\"\u003e\u003cimg src=\"https://img.shields.io/badge/arXiv-Paper-\u003ccolor\u003e\"\u003e\u003c/a\u003e \u003c/h1\u003e\n\u003cp align=\"center\"\u003e\n\u003ch4 align=\"center\"\u003eThis is the official repository of the paper \u003ca href=\"https://arxiv.org/abs/2305.01443\"\u003eScalable Mask Annotation for Video Text Spotting\u003c/a\u003e.\u003c/h4\u003e\n\u003ch5 align=\"center\"\u003e\u003cem\u003eHaibin He, Jing Zhang, Mengyang Xu, Juhua Liu, Bo Du, Dacheng Tao\u003c/em\u003e\u003c/h5\u003e\n\u003cp align=\"center\"\u003e\n  \u003ca href=\"#news\"\u003eNews\u003c/a\u003e |\n  \u003ca href=\"#abstract\"\u003eAbstract\u003c/a\u003e |\n  \u003ca href=\"#method\"\u003eMethod\u003c/a\u003e |\n  \u003ca href=\"#usage\"\u003eUsage\u003c/a\u003e |\n  \u003ca href=\"#results\"\u003eResults\u003c/a\u003e |\n  \u003ca href=\"#statement\"\u003eStatement\u003c/a\u003e\n\u003c/p\u003e\n\n\n\n\n\n\n\n# News\n\n***02/05/2023***\n\n- The paper is post on arxiv! The code will be made public available once cleaned up.\n\n- Relevant Project: \n\n  \u003e [**DPText-DETR: Towards Better Scene Text Detection with Dynamic Points in Transformer** ](https://arxiv.org/abs/2207.04491) | [Code](https://github.com/ymy-k/DPText-DETR)\n  \u003e\n  \u003e Maoyuan Ye, Jing Zhang, Shanshan Zhao, Juhua Liu, Bo Du, Dacheng Tao\n  \u003e\n  \u003e [**DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting** ](https://arxiv.org/pdf/2211.10772v3) | [Code](https://github.com/ViTAE-Transformer/DeepSolo)\n  \u003e\n  \u003e Maoyuan Ye,  Jing Zhang, Shanshan Zhao, Juhua Liu, Tongliang Liu, Bo Du, Dacheng Tao\n  \u003e\n  \u003e [**I3CL: Intra- and Inter-Instance Collaborative Learning for Arbitrary-shaped Scene Text Detection** ](https://arxiv.org/abs/2108.01343) | [Code](https://github.com/ViTAE-Transformer/I3CL)\n  \u003e\n  \u003e Bo Du, Jian Ye, Jing Zhang, Juhua Liu, Dacheng Tao\n\n  Other applications of [ViTAE](https://github.com/ViTAE-Transformer/ViTAE-Transformer) inlcude: [ViTPose](https://github.com/ViTAE-Transformer/ViTPose) | [Remote Sensing](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing) | [Matting](https://github.com/ViTAE-Transformer/ViTAE-Transformer-Matting) | [VSA](https://github.com/ViTAE-Transformer/ViTAE-VSA) | [Video Object Segmentation](https://github.com/ViTAE-Transformer/VOS-LLB)\n\n# Abstract\n\n\u003cp align=\"left\"\u003eVideo text spotting refers to localizing, recognizing, and tracking textual elements\nsuch as captions, logos, license plates, signs, and other forms of text within consecutive\nvideo frames. However, current datasets available for this task rely on\nquadrilateral ground truth annotations, which may result in including excessive\nbackground content and inaccurate text boundaries. Furthermore, methods trained\non these datasets often produce prediction results in the form of quadrilateral boxes,\nwhich limits their ability to handle complex scenarios such as dense or curved text.\nTo address these issues, we propose a scalable mask annotation pipeline called\nSAMText for video text spotting.SAMText leverages the \u003ca href=\"https://arxiv.org/abs/2304.02643\"\u003eSAM\u003c/a\u003e model to\ngenerate mask annotations for scene text images or video frames at scale. Using\nSAMText, we have created a large-scale dataset, SAMText-9M, that contains over\n2,400 video clips sourced from existing datasets and over 9 million mask annotations.\nWe have also conducted a thorough statistical analysis of the generated\nmasks and their quality, identifying several research topics that could be further\nexplored based on this dataset. \n\n\n\n\n# Method\n\u003cfigure\u003e\n\u003cimg src=\"figs/opening.png\"\u003e\n\u003cfigcaption align = \"center\"\u003e\u003cb\u003eFigure 1: Overview of the SAMText pipeline that builds upon the \u003ca href=\"https://arxiv.org/abs/2304.02643\"\u003eSAM\u003c/a\u003e   approach to generate\nmask annotations for scene text images or video frames at scale. The input bounding box may be\nsourced from existing annotations or derived from a scene text detection model.\u003c/b\u003e\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\n\n\n\n# Usage\nThe code and dataset will be released soon.\n\n\n\n# Results\n# The Quality of Generated Masks\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/figure3.png\"\u003e\n\u003cfigcaption align = \"center\"\u003e\u003cb\u003eFigure 3: The distribution of IoU between the generated\nmasks and ground truth masks in the COCOText\ntraining dataset:  \u003ca href=\"https://arxiv.org/abs/1601.07140\"\u003eCOCO_Text V2\u003c/a\u003e  \n \u003c/b\u003e\u003c/figcaption\u003e\n\u003c/figure\u003e\nTo evaluate the performance of SAMText, we\nselect the COCO-Text training dataset [25] as it\nprovides ground truth mask annotations for text\ninstances. Specifically, we randomly sample\n10% of the training data and calculate the IoU\nbetween the masks generated by SAMText and\ntheir corresponding ground truth masks. Our\nfindings show that SAMText has high accuracy,\nwith an average IoU of 0.70.  Figure 3 presents\nthe histogram of IoU scores. Notably, the majority\nof IoU scores are centered around 0.75,\nsuggesting that SAMText performs well.\n\n\n\n\n\n# Visualization of Generated Masks\n\n\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/figure2.jpg\"\u003e\n\u003cfigcaption align = \"center\"\u003e\u003cb\u003eFigure 2: Some visualization results of the generated masks in five datasets using the SAMText\npipeline. The top row shows the scene text frames while the bottom row shows the generated masks.\u003c/a\u003e  \n \u003c/b\u003e\u003c/figcaption\u003e\n\u003c/figure\u003e\n\nIn Figure 2, we show some visualization results of the generated masks in five datasets using the\nSAMText pipeline. The top row shows the scene text frames while the bottom row shows the\ngenerated masks. As can be seen, the generated masks possess fewer background components and\nalign more precisely with the text boundaries than the bounding boxes. As a result, the generated\nmask annotations facilitate conducting more comprehensive research on this dataset, e.g., video text\nsegmentation and video text spotting using mask annotations.\n\n\n\n\n\n\n## Dataset Statistics and Analysis\n### The size distribution.\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/figure4.png\"\u003e\n\u003cfigcaption align = \"center\"\u003e\u003cb\u003eFigure 4: (a) The mask size distributions of the ICDAR15, RoadText-1k, LSVDT, and DSText datasets.\nMasks exceeding 10,000 pixels are excluded from the statistics. (b) The mask size distributions of\nthe BOVText datasets. Masks exceeding 80,000 pixels are excluded from the statistics.\u003c/a\u003e  \n \u003c/b\u003e\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\n\n### The IoU and COV distribution.\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/figure5.png\"\u003e\n\u003cfigcaption align = \"center\"\u003e\u003cb\u003eFigure 5: (a) The distribution of IoU between the generated masks and ground truth bounding boxes\nin each dataset. (b) The CoV distribution of mask size changes for the same individual in consecutive\nframes in all five datasets, excluding the CoV scores exceeding 1.0 from the statistics.\u003c/a\u003e  \n \u003c/b\u003e\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\n\n### The spatial distribution.\n\n\u003cfigure\u003e\n\u003cimg src=\"figs/figure6.png\"\u003e\n\u003cfigcaption align = \"center\"\u003e\u003cb\u003eFigure 6: Visualization of the heatmaps that depict the spatial distribution of the generated masks in\nthe five video text spotting datasets employed to establish SAMText-9M.\u003c/a\u003e  \n \u003c/b\u003e\u003c/figcaption\u003e\n\u003c/figure\u003e\n\n\n\n# Statement\n\nThis project is for research purpose only. For any other questions please contact [haibinhe@whu.edu.cn](mailto:haibinhe@whu.edu.cn).\n\n\n\n## Citation\n\nIf you find SAMText helpful, please consider giving this repo a star:star: and citing:\n\n```\n@inproceedings{SAMText,\n  title={Scalable Mask Annotation for Video Text Spotting},\n  author={Haibin He, Jing Zhang, Mengyang Xu, Juhua Liu, Bo Du, Dacheng Tao},\n  booktitle={arxiv},\n  year={arXiv preprint arXiv:2305.01443}\n}\n```\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitae-transformer%2Fsamtext","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvitae-transformer%2Fsamtext","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvitae-transformer%2Fsamtext/lists"}