{"id":14980548,"url":"https://github.com/linxueyuanstdio/latex_ocr_pro","last_synced_at":"2025-10-20T11:18:38.771Z","repository":{"id":37385061,"uuid":"197900256","full_name":"LinXueyuanStdio/LaTeX_OCR_PRO","owner":"LinXueyuanStdio","description":":art: 数学公式识别增强版：中英文手写印刷公式、支持初级符号推导（数据结构基于 LaTeX 抽象语法树）Math Formula OCR Pro, supports handwrite, Chinese-mixed formulas and simple symbol reasoning (based on LaTeX AST). ","archived":false,"fork":false,"pushed_at":"2024-06-11T16:35:31.000Z","size":78122,"stargazers_count":1220,"open_issues_count":17,"forks_count":236,"subscribers_count":15,"default_branch":"master","last_synced_at":"2025-05-16T00:07:46.582Z","etag":null,"topics":["cnn","deep-learning","latex","lstm","ocr","rnn","seq2seq"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LinXueyuanStdio.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-07-20T08:11:05.000Z","updated_at":"2025-05-14T03:22:40.000Z","dependencies_parsed_at":"2024-09-24T05:01:49.556Z","dependency_job_id":"59770c7c-c25f-4e93-9734-f100d0550cd6","html_url":"https://github.com/LinXueyuanStdio/LaTeX_OCR_PRO","commit_stats":{"total_commits":97,"total_committers":6,"mean_commits":"16.166666666666668","dds":"0.11340206185567014","last_synced_commit":"57c8724b967cb458cddae08a6a6b4465bfff59d2"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinXueyuanStdio%2FLaTeX_OCR_PRO","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinXueyuanStdio%2FLaTeX_OCR_PRO/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinXueyuanStdio%2FLaTeX_OCR_PRO/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LinXueyuanStdio%2FLaTeX_OCR_PRO/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LinXueyuanStdio","download_url":"https://codeload.github.com/LinXueyuanStdio/LaTeX_OCR_PRO/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254442855,"owners_count":22071878,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cnn","deep-learning","latex","lstm","ocr","rnn","seq2seq"],"created_at":"2024-09-24T14:01:59.171Z","updated_at":"2025-10-20T11:18:38.691Z","avatar_url":"https://github.com/LinXueyuanStdio.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LaTeX_OCR_PRO\n\n数学公式识别，增强：中文公式、手写公式\n\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/visualization_6_short.gif)\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/6.png)\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/visualization_12_short.gif)\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/12.png)\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/visualization_14_short.gif)\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/14.png)\n\nSeq2Seq + Attention + Beam Search。结构如下：\n\n![](https://raw.githubusercontent.com/LinXueyuanStdio/LaTeX_OCR/master/art/architecture.jpg)\n\n* [1. 搭建环境](#1-搭建环境)\n* [2. 开始训练](#2-开始训练)\n* [3. 可视化](#3-可视化)\n* [4. 部署](#4-部署)\n* [5. 评价](#5-评价)\n* [6. 更多细节](#6-更多细节)\n    * [模型实现细节](./doc/How-it-work.md)\n    * [解决方案](./doc/Solution.md)\n* [7. 致谢](#7-致谢)\n* [8. 相关项目](#8-相关项目)\n* [9. 引用](#9-引用)\n\n## 1. 搭建环境\n\n1. python3.5 + tensorflow1.12.2\n2. `[可选]` latex (latex 转 pdf)\n3. `[可选]` ghostscript (图片处理)\n4. `[可选]` magick (pdf 转 png)\n\n### 如果你想直接训练，不想自己构建数据集：\n\n1. `[可选]` 新开一个虚拟环境\n   ```shell\n   virtualenv env35 --python=python3.5\n   source env35/bin/activate\n   ```\n2. 安装依赖\n   ```shell\n   pip install -r requirements.txt     // cpu 版\n   pip install -r requirements-gpu.txt // gpu 版\n   ```\n3. 下载数据集\n   ```shell\n   git submodule init\n   git submodule update\n   ```\n   \u003e 如果 git 速度太慢，您也可以手动下载数据集，放到 data 目录下。数据集仓库在 https://github.com/LinXueyuanStdio/Data-for-LaTeX_OCR\n   \u003e 数据仓库同时托管到 [huggingface (linxy/LaTeX_OCR)](https://huggingface.co/datasets/linxy/LaTeX_OCR)，欢迎使用！\n\n### 如果你想自己构建数据集，然后再训练：\n\n\u003cdetails\u003e\n  \u003csummary\u003eLinux\u003c/summary\u003e\n\n一键安装\n```shell\nmake install-linux\n```\n或\n1. 安装本项目依赖\n```shell\nvirtualenv env35 --python=python3.5\nsource env35/bin/activate\npip install -r requirements.txt\n```\n2. 安装 latex (latex 转 pdf)\n```shell\nsudo apt-get install texlive-latex-base\nsudo apt-get install texlive-latex-extra\n```\n3. 安装 ghostscript\n```shell\nsudo apt-get update\nsudo apt-get install ghostscript\nsudo apt-get install libgs-dev\n```\n4. 安装[magick](https://www.imagemagick.org/script/install-source.php) (pdf 转 png)\n```shell\nwget http://www.imagemagick.org/download/ImageMagick.tar.gz\ntar -xvf ImageMagick.tar.gz\ncd ImageMagick-7.*; \\\n./configure --with-gslib=yes; \\\nmake; \\\nsudo make install; \\\nsudo ldconfig /usr/local/lib\nrm ImageMagick.tar.gz\nrm -r ImageMagick-7.*\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003eMac\u003c/summary\u003e\n\n一键安装\n\n```shell\nmake install-mac\n```\n\n或\n1. 安装本项目依赖\n```shell\nsudo pip install -r requirements.txt\n```\n2. LaTeX\n\n我们需要 pdflatex，可以傻瓜式一键安装：[http://www.tug.org/mactex/mactex-download.html](http://www.tug.org/mactex/mactex-download.html)\n\n3. 安装[magick](https://www.imagemagick.org/script/install-source.php) (pdf 转 png)\n\n```shell\nwget http://www.imagemagick.org/download/ImageMagick.tar.gz\ntar -xvf ImageMagick.tar.gz\ncd ImageMagick-7.*; \\\n./configure --with-gslib=yes; \\\nmake;\\\nsudo make install; \\\nrm ImageMagick.tar.gz\nrm -r ImageMagick-7.*\n```\n\n\u003c/details\u003e\n\n## 2. 开始训练\n\n\n\u003cdetails\u003e\n  \u003csummary\u003e生成小数据集、训练、评价\u003c/summary\u003e\n\n提供了样本量为 100 的小数据集，方便测试。只需 2 分钟就可以根据 `./data/small.formulas/` 下的公式生成用于训练的图片。\n\n\u003e 注意：样本量很小，是无法有效训练模型的。这个小数据集仅用于确认代码有没有 bug。如果用于预测，那结果极差，因为数据不够。\n\n一步训练\n\n```\nmake small\n```\n或\n\n1. 生成数据集\n\n   用 LaTeX 公式生成图片，同时保存公式-图片映射文件，生成字典 __只用运行一次__\n\n    ```shell\n    # 默认\n    python build.py\n    # 或者\n    python build.py --data=configs/data_small.json --vocab=configs/vocab_small.json\n    ```\n\n2. 训练\n    ```\n    # 默认\n    python train.py\n    # 或者\n    python train.py --data=configs/data_small.json --vocab=configs/vocab_small.json --training=configs/training_small.json --model=configs/model.json --output=results/small/\n    ```\n\n3. 评价预测的公式\n    ```\n    # 默认\n    python evaluate_txt.py\n    # 或者\n    python evaluate_txt.py --results=results/small/\n    ```\n\n4. 评价数学公式图片\n\n    ```\n    # 默认\n    python evaluate_img.py\n    # 或者\n    python evaluate_img.py --results=results/small/\n    ```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e生成完整数据集、训练、评价\u003c/summary\u003e\n\n根据公式生成 70,000+ 数学公式图片需要 `2`-`3` 个小时\n\n一步训练\n\n```\nmake full\n```\n或\n\n1. 生成数据集\n\n   用 LaTeX 公式生成图片，同时保存公式-图片映射文件，生成字典 __只用运行一次__\n    ```\n    python build.py --data=configs/data.json --vocab=configs/vocab.json\n    ```\n\n2. 训练\n    ```\n    python train.py --data=configs/data.json --vocab=configs/vocab.json --training=configs/training.json --model=configs/model.json --output=results/full/\n    ```\n\n3. 评价预测的公式\n    ```\n    python evaluate_txt.py --results=results/full/\n    ```\n\n4. 评价数学公式图片\n    ```\n    python evaluate_img.py --results=results/full/\n    ```\n\n\u003c/details\u003e\n\n## 3. 可视化\n\n\u003cdetails\u003e\n  \u003csummary\u003e可视化训练过程\u003c/summary\u003e\n\n用 tensorboard 可视化训练过程\n\n小数据集\n\n```\ncd results/small\ntensorboard --logdir ./\n```\n\n完整数据集\n\n```\ncd results/full\ntensorboard --logdir ./\n```\n\u003c/details\u003e\n\n\u003cdetails\u003e\n  \u003csummary\u003e可视化预测过程\u003c/summary\u003e\n\n打开 `visualize_attention.ipynb`，一步步观察模型是如何预测 LaTeX 公式的。\n\n或者运行\n\n```shell\n# 默认\npython visualize_attention.py\n# 或者\npython visualize_attention.py --image=data/images_test/6.png --vocab=configs/vocab.json --model=configs/model.json --output=results/full/\n```\n\n可在 `--output` 下生成预测过程的注意力图。\n\n\u003c/details\u003e\n\n## 4. 部署\n\n\u003cdetails\u003e\n  \u003csummary\u003e部署为 Django 应用\u003c/summary\u003e\n\n1. 安装部署需要的环境\n   ```bash\n   pip install django\n   ```\n2. 开启服务\n   ```bash\n   python manage.py runserver 0.0.0.0:8010\n   ```\n3. 开启图片服务\n   ```bash\n   cd data/images_train\n   python -m SimpleHTTPServer 8020\n   ```\n4. 使用方法\n   在输入框里依次输入 `0.png`, `1.png` 等等，即可看到结果\n\n\u003c/details\u003e\n\n## 5. 评价\n\n|      指标       | 训练分数 | 测试分数 |\n| :-------------: | :------: | :------: |\n|   perplexity    |   1.12   |   1.13   |\n|  EditDistance   |  94.16   |  93.36   |\n|     BLEU-4      |  91.03   |  90.47   |\n| ExactMatchScore |  49.30   |  46.22   |\n\nperplexity 是越接近1越好，其余3个指标是越大越好。\n\n其中 EditDistance 和 BLEU-4 已达到业内先进水平\n\n将 perplexity 训练到 1.03 左右，ExactMatchScore 还可以再升，应该可以到 70 以上。\n\n机器不太好，训练太费时间了。\n\n## 6. 更多细节\n\n1. [模型实现细节](./doc/How-it-work.md)\n\n   包括数据获取、数据处理、模型架构、训练细节\n\n2. [解决方案](./doc/Solution.md)\n\n   包括 “如何可视化 Attention 层”、“在 win10 用 GPU 加速训练” 等等\n\n## 7. 致谢\n\n十分感谢 Harvard 和 Guillaume Genthial 、Kelvin Xu 等人提供巨人的肩膀。\n\n论文：\n1. [Show, Attend and Tell(Kelvin Xu...)](https://arxiv.org/abs/1502.03044)\n2. [Harvard's paper and dataset](http://lstm.seas.harvard.edu/latex/)\n3. [Seq2Seq for LaTeX generation](https://guillaumegenthial.github.io/image-to-latex.html).\n\n## 8. 相关项目\n\n[LaTeX_OCR 的 PyTorch 版: https://github.com/qs956/Latex_OCR_Pytorch](https://github.com/qs956/Latex_OCR_Pytorch) by [@qs956](https://github.com/qs956)\n\n## 9. 引用\n\nBibTeX\n\n```\n@misc{lin2024latex_ocr_pro,\n  title={LaTeX_OCR_PRO},\n  author={Xueyuan Lin},\n  year={2024},\n  publisher={GitHub},\n  howpublished={\\url{https://github.com/LinXueyuanStdio/LaTeX_OCR_PRO}},\n}\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinxueyuanstdio%2Flatex_ocr_pro","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flinxueyuanstdio%2Flatex_ocr_pro","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flinxueyuanstdio%2Flatex_ocr_pro/lists"}