{"id":18764273,"url":"https://github.com/chineseocr/trocr-chinese","last_synced_at":"2025-10-10T04:13:04.337Z","repository":{"id":42458037,"uuid":"477624494","full_name":"chineseocr/trocr-chinese","owner":"chineseocr","description":"transformers ocr for chinese","archived":false,"fork":false,"pushed_at":"2023-01-13T16:21:30.000Z","size":183,"stargazers_count":387,"open_issues_count":42,"forks_count":60,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-05-26T02:07:28.582Z","etag":null,"topics":["chinese-ocr","pytorch","seq2seq"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chineseocr.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-04-04T09:05:39.000Z","updated_at":"2025-05-16T02:17:37.000Z","dependencies_parsed_at":"2023-02-09T16:16:28.704Z","dependency_job_id":null,"html_url":"https://github.com/chineseocr/trocr-chinese","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/chineseocr/trocr-chinese","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chineseocr%2Ftrocr-chinese","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chineseocr%2Ftrocr-chinese/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chineseocr%2Ftrocr-chinese/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chineseocr%2Ftrocr-chinese/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chineseocr","download_url":"https://codeload.github.com/chineseocr/trocr-chinese/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chineseocr%2Ftrocr-chinese/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279002690,"owners_count":26083442,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-10T02:00:06.843Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["chinese-ocr","pytorch","seq2seq"],"created_at":"2024-11-07T18:29:18.966Z","updated_at":"2025-10-10T04:13:04.314Z","avatar_url":"https://github.com/chineseocr.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# 基于trocr(beit+roberta)实现对中文场景文字识别\ntrocr原地址(https://github.com/microsoft/unilm/tree/master/trocr)\n## 实现功能\n- [x]  单行/多行文字/横竖排文字识别\n- [x]  不规则文字（印章，公式等）\n- [x]  转 onnx模型\n- [ ]  表格识别\n- [ ]  模型蒸馏/DML(协作学习)\n- [ ]  Prompt Learning\n\n## 环境编译\n```\ndocker build --network=host -t trocr-chinese:latest .\ndocker run --gpus all -it -v /tmp/trocr-chinese:/trocr-chinese trocr-chinese:latest bash\n\n```\n## 训练\n### 初始化模型到自定义训练数据集\n#### 字符集准备参考cust-data/vocab.txt\n```\nvocab.txt\n1\n2\n...\na\nb\nc\n```\n```[python]\npython gen_vocab.py \\\n       --dataset_path \"dataset/cust-data/0/*.txt\" \\\n       --cust_vocab ./cust-data/vocab.txt\n\n```\n### 初始化自定义数据集模型\n#### 下载预训练模型trocr模型权重\n链接: https://pan.baidu.com/s/1rARdfadQlQGKGHa3de82BA  密码: 0o65.  \ngoogle driver: https://drive.google.com/drive/folders/1ibOVCHu33asiMUaFT9FzvhFNM4z25cJY?usp=share_link  \n```\npython init_custdata_model.py \\   \n    --cust_vocab ./cust-data/vocab.txt \\  \n    --pretrain_model ./weights \\\n    --cust_data_init_weights_path ./cust-data/weights\n    \n## cust_vocab 词库文件   \n## pretrain_model 预训练模型权重   \n## cust_data_init_weights_path 自定义模型初始化模型权重保存位置   \n\n```\n\n### 训练模型\n#### 数据准备,数据结构如下图所示\n```\ndataset/cust-data/0/0.jpg\ndataset/cust-data/0/0.txt\n...\ndataset/cust-data/100/10000.jpg\ndataset/cust-data/100/10000.txt\n```\n\n#### 训练模型\n```\npython train.py \\\n       --cut_data_init_weights_path ./cust-data/weights \\\n       --checkpoint_path ./checkpoint/trocr-custdata \\\n       --dataset_path \"./dataset/cust-data/*/*.jpg\" \\\n       --per_device_train_batch_size 8 \\\n       --CUDA_VISIBLE_DEVICES 1\n```\n\n#### 评估模型\n##### 拷贝checkpoint/trocr-custdata训练完成的pytorch_model.bin 到 ./cust-data/weights 目录下\n\n```[python]\npython eval.py \\\n    --dataset_path \"./data/cust-data/test/*/*.jpg\" \\\n    --cust_data_init_weights_path ./cust-data/weights    \n```\n\n## 测试模型\n```\n## 拷贝训练完成的pytorch_model.bin 到 ./cust-data/weights 目录下\nindex = 2300 ##选择最好的或者最后一个step模型\ncp ./checkpoint/trocr-custdata/checkpoint-$index/pytorch_model.bin ./cust-data/weights\npython app.py --cust_data_init_weights_path ./cust-data/weights --test_img test/test.jpg\n```\n## 转onnx \n```shell\npython -m \\\n    transformers.onnx \\\n    hand-write \\\n    --feature=vision2seq-lm \\\n    hand-write-onnx --atol 1e-4\n\ncp hand-write/vocab.json hand-write-onnx/\n\npython onnx_test.py --model hand-write-onnx --test_img ./img/hand.png\n\n```\n\n## 预训练模型\n| 模型        | cer(字符错误率)           | acc(文本行)  | 下载地址  |训练数据来源 |训练耗时(GPU:3090) | \n| ------------- |:-------------:| -----:|-----:|-----:|-----:|\n| hand-write(中文手写)      |0.011 | 0.940 |[hand-write](https://pan.baidu.com/s/19f7iu9tLHkcT_zpi3UfqLQ)  密码: punl |[数据集地址](https://aistudio.baidu.com/aistudio/datasetdetail/102884/0) |8.5h(10epoch)|\n| seal-ocr(印章识别)      |0.006 | 0.956 | 整理后开放下载|- |\n| im2latex(数学公式识别)      |- | - |- |[im2latex](https://zenodo.org/record/56198#.YkniL25Bx_S) ||\n| TAL_OCR_TABLE(表格识别)     |- | - |- |[TAL_OCR_TABLE](https://ai.100tal.com/dataset) |\n| TAL_OCR_MATH(小学低年级算式数据集)|- | - |- | [TAL_OCR_MATH](https://ai.100tal.com/dataset) |\n| TAL_OCR_CHN(手写中文数据集)|0.0455 | 0.674（标注质量不太高,例如：test_64/552.jpg 标注值:蝶恋花, 实际值:欧阳修  ） |[TAL_OCR_CHN](https://pan.baidu.com/s/1GbyVpbiKyVN1nJipvH2fpA)  密码: 9kd8 | [TAL_OCR_CHN](https://ai.100tal.com/dataset) |0.6h(20epoch)|\n| HME100K(手写公式)|- | - |- | [HME100K](https://ai.100tal.com/dataset) |\n\n备注:后续所有模型会开源在这个目录下链接,可以自由下载. https://pan.baidu.com/s/1uSdWQhJPEy2CYoEULoOhRA  密码: vwi2\n### 模型调用 \n#### 手写识别\n![image](img/hand.png)\n```\nunzip hand-write.zip \npython app.py --cust_data_init_weights_path hand-write --test_img test/hand.png\n\n## output: '醒我的昏迷,偿还我的天真。'\n```\n\n## 训练技巧\n###### 数据集较少时，可以采用数据增强的方法构造更多的数据，理论上几十万的数据（可不做数据增强，模型预训练已经见到过足够多的数据(票据类、证件类，打印、手写、拍照等场景)），可以收敛到90%以上的准确率（CER\u003c0.05）   \n###### 训练样本不要自己resize到384x384（后续会优化这个结构，目前预训练是384x384），保留原图即可，模型前处理processor会自动处理    \n###### 如果要训练识别多行文字，文字行之间可以加一个特殊字符标记，例如：\"1234\\n4567\\n89990\"   \n###### fine-tune中英文以外的语言效果可能不太好（足够多的数据及足够steps也能收敛），因为没有在其他语言上预训练    \n###### 遇到问题先分析一下自己的数据，然后增加一些训练的技巧去优化，不要指望模型解决100%的问题\n###### 本项目采用的encoder-decoder结构, 模型还是比较大，如果上生产对硬件开销大，也可以优化encoder（比如cnn结构的mobilenet，resnet）或者decoder（roberta-tiny），然后对其进行蒸馏\n###### 如果此项目不能解决您的问题，请选择其他项目，不要因为此项目影响自己的心情！！！\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchineseocr%2Ftrocr-chinese","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchineseocr%2Ftrocr-chinese","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchineseocr%2Ftrocr-chinese/lists"}