{"id":15348673,"url":"https://github.com/domoritz/label_generator","last_synced_at":"2025-06-20T05:37:36.628Z","repository":{"id":31278386,"uuid":"34840284","full_name":"domoritz/label_generator","owner":"domoritz","description":"Training data generator for text detection","archived":false,"fork":false,"pushed_at":"2020-07-16T18:06:36.000Z","size":7343,"stargazers_count":39,"open_issues_count":1,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-06-19T23:53:55.426Z","etag":null,"topics":["deep-learning","neural-network","python","text-detection"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/domoritz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-04-30T07:21:26.000Z","updated_at":"2024-05-13T02:16:32.000Z","dependencies_parsed_at":"2022-09-09T07:40:27.476Z","dependency_job_id":null,"html_url":"https://github.com/domoritz/label_generator","commit_stats":null,"previous_names":[],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/domoritz/label_generator","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domoritz%2Flabel_generator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domoritz%2Flabel_generator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domoritz%2Flabel_generator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domoritz%2Flabel_generator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/domoritz","download_url":"https://codeload.github.com/domoritz/label_generator/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/domoritz%2Flabel_generator/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260890520,"owners_count":23077861,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deep-learning","neural-network","python","text-detection"],"created_at":"2024-10-01T11:51:06.992Z","updated_at":"2025-06-20T05:37:31.611Z","avatar_url":"https://github.com/domoritz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Text detection in screen images with a Convolutional Neural Network [![theoj](http://joss.theoj.org/papers/d2821f933fc95337202393e84189f4d9/status.svg)](http://joss.theoj.org/papers/d2821f933fc95337202393e84189f4d9) [![Build Status](https://travis-ci.org/domoritz/label_generator.svg?branch=master)](https://travis-ci.org/domoritz/label_generator)\n\n**Note: This was a class project where I wanted to learn about neural networks. If you want to do text detection in images, I suggest that you use something like [this approach](http://www.math.tau.ac.il/~turkel/imagepapers/text_detection.pdf).**\n\nThe repository contains a set of scripts to implement text detection from screen images. The idea is that we use a Convolutional Neural Network (CNN) to predict a heatmap of the probability of text in an image. But before we can predict anything, we need to train the network with a a set of pairs of images and training labels. We obtain the training data by extracting figures with embedded text from research papers.\n\n**This is a very involved process and you may want to use the labels that I already generated (you are welcome). We have around 500K good labels extracted from around 1M papers from arXiv and the ACL anthology.**\n\nPDF files, extracted figures and labels are in an S3 bucket at `s3://escience.washington.edu.viziometrics`. The PDF files for arXiv (extracted from [arXiv bulk access](http://arxiv.org/help/bulk_data_s3)) are in a separate bucket at `s3://arxiv-tars-pdfs`. The buckets have [requester pays](https://docs.aws.amazon.com/en_us/console/s3/requesterpaysbucket) enabled.\n\nPlease cite [the paper for this repo](https://www.theoj.org/joss-papers/joss.00235/10.21105.joss.00235.pdf) as\n\n```bib\n@article{Moritz2017,\n  doi = {10.21105/joss.00235},\n  url = {https://doi.org/10.21105/joss.00235},\n  year = {2017},\n  month = jul,\n  publisher = {The Open Journal},\n  volume = {2},\n  number = {15},\n  pages = {235},\n  author = {Dominik Moritz},\n  title = {Text detection in screen images with a Convolutional Neural Network},\n  journal = {The Journal of Open Source Software}\n}\n```\n\n## Requirements\n\nInstall OpenCV with python support. Also install scipy, matplotlib, and numpy for python (either through pip or apt). Also install freetype, ghostscript, imagemagic, and tesseract. Please check the compatible versions of [pdffigures](https://github.com/allenai/pdffigures) with your OS.\n\n## Generate training data\n\nYou can run this locally or on a server. I tested every script locally on a mac without any problems. Below are instructions for Linux.\n\nThe scripts use [pdffigures](http://pdffigures.allenai.org/) to generate a JSON file that describes each figure in a paper.\n\n### AWS instructions\n\nThese are the steps I had to run to generate the training data an EC2 machines on AWS. The execution is embarrassingly parallel and thus runs reasonably fast (a few hours to a day or two for a million papers). At the time of writing, I ran this on Ubuntu 14.04, but later version may work as well with some small modifications.\n\nThe commands below are what I used to extract the images and generate the labels. As described above, you don't need to rerun this unless you want to use different papers than the ones I already extracted figures from (see above). If you want to run the code, you need to change the output S3 bucket to a bucket that you have write access to.\n\n```sh\n# use tmux (maybe with attach)\ntmux\n\nsudo apt-get update\nsudo apt-get install git python-pip python-opencv python-numpy python-scipy python-matplotlib ghostscript libmagickwand-dev libfreetype6 parallel\n\ngit clone https://github.com/domoritz/label_generator.git\ncd label_generator\nsudo pip install -r requirements.txt\ngit submodule init\ngit submodule update\n\nsudo apt-get install libpoppler-dev libleptonica-dev pkg-config\n\n# we need gcc 4.9\nsudo add-apt-repository ppa:ubuntu-toolchain-r/test\nsudo apt-get update\nsudo apt-get install g++-4.9\n\n# compile pdffigures\nmake -C pdffigures DEBUG=0 CC='g++-4.9 -std=c++11'\n\n# at this point, you probably need to make a copy of the config file and update it\ncp config_sample.py config.py\nvim config.py\n\n# test with one file\npython label_gen.py read-s3 escience.washington.edu.viziometrics acl_anthology/pdf/C08-1099.pdf escience.washington.edu.viziometrics acl_anthology\n\n# get list of documents to process\naws s3 --region=us-west-2 ls s3://escience.washington.edu.viziometrics/acl_anthology/pdf/ | awk '{ print $4 }' \u003e acl_papers.txt\n\n# now run for real\nparallel --resume -j +6 --no-run-if-empty --eta --joblog /tmp/par.log python label_gen.py read-s3 escience.washington.edu.viziometrics acl_anthology/pdf/{} escience.washington.edu.viziometrics acl_anthology --dbg-image :::: acl_papers.txt\n\n# monitor progress\ntail -f /tmp/par.log\n\n# find bad labels\npython find_bad.py read-s3 escience.washington.edu.viziometrics acl_anthology/json \u003e anthology_bad.txt\n# you probably want to use this file to delete bad labels before you use it to train the CNN\n# Use: parallel rm -f data/{}-label.png :::: anthology_bad.txt\n\n# run find bad in parallel\nseq {0,19} | parallel -j 20 --eta python find_bad.py read-s3 escience.washington.edu.viziometrics arxiv/json --chunk={} --of=20 '\u003e' arxiv_bad_{}.txt\ncat arxiv_bad_*.txt \u003e arxiv_bad.txt\n\n# at this point you may want to upload the file with bad labels back to S3\n```\n\n### FAQ for common error messages\n\nThese are some common errors I have experienced.\n\n**I don't see my output** Try `--debug` and make sure that you have the correct folders set up if you use S3.\n\n**Failed to initialize libdc1394** `sudo ln /dev/null /dev/raw1394` https://stackoverflow.com/questions/12689304/ctypes-error-libdc1394-error-failed-to-initialize-libdc1394\n\n**ImportError: MagickWand shared library not found.** See https://github.com/dahlia/wand/issues/141\n\n### Try the figure extraction\n\n#### Local\n\n`python label_gen.py read testdata/paper.pdf /tmp/test --dbg-image --debug`\n\n#### With data from S3\n\n`python label_gen.py read-s3 escience.washington.edu.viziometrics test/pdf/C08-1092.pdf test/ --dbg-image --debug`\n\n\n## Train the neural network\n\nI used a different machine for training the network because AWS doesn't have good graphics cards.\n\nYou can use any CNN to get the prediction but I use [pjreddie/darknet](https://github.com/pjreddie/darknet). My fork is at [domoritz/darknet](https://github.com/domoritz/darknet) and a submodule of this repo.\n\nTo train the network, you need to put all figures and labels into one directory. Then generate a  file called `train.list` in `/data`. You can generate this file with `ls . | grep -v -- \"-label.png\" | awk '{print \"PATH_TO_FILES/\"$1}' \u003e ../all.list` in the directory with all the images. Then split the file into training and test data.\n\nThen train the network with `./darknet writing train cfg/writing.cfg`. This will generate a weight file every now and then. If for some reason some files are missing labels, use a python script like this to filter out files that don't have labels.\n\n```python\nimport sys\nimport os.path\n\nwith open(sys.argv[1]) as f:\n        for fname in f:\n                fname = fname.strip()\n                if not os.path.isfile(fname):\n                        print fname\n                lname = fname[:-4] + \"-label.png\"\n                if not os.path.isfile(lname):\n                        print fname\n```\n\n## Predict where text is and find text areas\n\nYou need a trained network. To test the network, run `echo \"PATH_TO_FILES/FIGURE.png\" | ./darknet writing test cfg/writing.cfg ../writing_backup/writing_ITER.weights`. If you append `out`, a prediction will be written to `out.png`.\n\nA prediction looks like this\n\n![Red boxes around extracted text](https://raw.githubusercontent.com/domoritz/label_generator/master/screenshots/hep-th0401120-Figure-23-prediction.png)\n\nIf you want to test the network on all your test data, use a script like\n\n```bash\nfor i in `cat $1` ; do\n    fname=`basename $i .png`\n    echo $i | ./darknet writing test cfg/writing.cfg ../writing_backup/writing_8500.weights PATH_FOR_PREDICTIONS/$fname-predicted\ndone\n```\n\nand run it with your list of training data as the input. This will write all the predictions into a directory. If you feel like moving all your other files (the ground truth, images and such), use a command like `cat test.list | xargs cp -t PATH_FOR_PREDICTIONS`.\n\nCool, now we have a bunch of images in one directory. Let's find out what the precision and recall are. First, create a list of all the files in the directory with `ls | grep -- \"-predicted.png\" \u003e _all.list`. Then just run `python rate.py ../predicted/predicted/_all.list`.\n\nAfter all this work, we can finally generate a prediction, find contours, fit boxes around contours and find text with tesseract. To do so, run `python predict.py PREDICTION FIGURE_IMAGE --debug`. You may see something like\n\n![Red boxes around extracted text](https://raw.githubusercontent.com/domoritz/label_generator/master/screenshots/text-debug.png)\n\n## Support\n\nPlease ask questions and files issues [on GitHub](https://github.com/domoritz/label_generator/issues/new).\n\n## Contribute\n\nContributions are welcome. Development happens on GitHub at [domoritz/label_generator](https://github.com/domoritz/label_generator). When sending a pull request, please compare the output of `python label_gen.py read testdata/paper.pdf /tmp/test` with the images in [`testoutput`](https://github.com/domoritz/label_generator/tree/master/testoutput).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdomoritz%2Flabel_generator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdomoritz%2Flabel_generator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdomoritz%2Flabel_generator/lists"}