https://github.com/sergiocorreia/quipucamayoc
dev repo for article
https://github.com/sergiocorreia/quipucamayoc
ocr ocr-post-processing ocr-python poppler table-extraction table-ocr textract
Last synced: 2 months ago
JSON representation
dev repo for article
- Host: GitHub
- URL: https://github.com/sergiocorreia/quipucamayoc
- Owner: sergiocorreia
- License: agpl-3.0
- Created: 2022-02-15T15:23:19.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-03-14T20:36:48.000Z (over 2 years ago)
- Last Synced: 2024-10-13T11:22:30.611Z (8 months ago)
- Topics: ocr, ocr-post-processing, ocr-python, poppler, table-extraction, table-ocr, textract
- Language: Python
- Homepage:
- Size: 30.3 MB
- Stars: 24
- Watchers: 5
- Forks: 5
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Citation: CITATION.cff
Awesome Lists containing this project
README
# Quipucamayoc: tools for digitizing historical data
[](https://pypi.python.org/pypi/quipucamayoc/)
[](https://github.com/sergiocorreia/quipucamayoc/actions?query=workflow%3A%22CI+Tests%22)
[](https://zenodo.org/badge/latestdoi/55024750)
[](https://github.com/sergiocorreia/quipucamayoc/releases)
[](https://pypi.python.org/pypi/quipucamayoc/)
[](https://pypi.org/project/quipucamayoc)
[](https://poppler.freedesktop.org/)
[quipucamayoc](http://scorreia.com/software/quipucamayoc/) is a Python package that simplifies the extraction of historical data from scanned images and PDFs.
It's designed to be modular and so it can be used together with other existing tools, and can be extended easily by users.
For an overview of how to use `quipucamayoc` to digitize historical data, see [this research article](http://scorreia.com/research/digitizing.pdf), which amongst other things details the different steps involved, the methods used, and provides practical examples.
For an user guide, documentation, and installation instructions, see (TODO).
If you want to contribute by improving the code or extending its functionality (much welcome!), head [here](/CONTRIBUTING.md).
## Installation
### Pip
To manage quipucamayoc using pip, open the command line and run:
- `pip install quipucamayoc` to install
- `pip install quipucamayoc[dev]` to include extra dependencies used when developing the code
- `pip install -U quipucamayoc` to upgrade
- `pip uninstall quipucamayoc` to remove
Note that `quipucamayoc` has been tested against Python 3.10 and newer versions, but should also work with Python 3.9.
### Git Install
After cloning the repo to your computer and navigating to the quipucamayoc folder, run:
- `pip install .` to install the package locally
- `pip install --no-cache-dir --editable .[dev]` to install locally with a symlink so changes are automatically updated (recommended for developers)
## After installation
### AWS
`quipucamayoc` can use AWS's `Textract` to OCR text and tables. Its configuration is quite cumbersome, so it has been automated for you. To do so, first install `quipucamayoc` and then follow these steps to specify your credentials:
1. Ensure you have an [Amazon AWS account](https://aws.amazon.com/).
- AWS Textract prices are listed [here](https://aws.amazon.com/textract/pricing/). If you just created an account you should be able to use the [free tier](https://aws.amazon.com/free/). Otherwise, you might need to set up a [payment method](https://console.aws.amazon.com/billing/home#/paymentpreferences/paymentmethods).
2. Now you need to create credentials so you can access AWS programmatically.
- The simplest method is to go to the [security credentials](https://console.aws.amazon.com/iam/home#security_credential) page (you can also go to it from the AWS console: click on your name on the top-right > click on security credentials). Then, ignore the security warning (see below), go the next page, scroll to Access Keys and click create an access key. Copy the `Access key` and `Secret access key` strings (akin to username and passwords).
- (*TODO*) Alternatively, AWS now recommends an alternative: instead of creating access keys for your user, create a new user with a more restricted access and then create an access key for this user. In this way, if you somehow lose your credentials (e.g. your computer is hacked) then hackers are limited in what they can do. To create an user, go to the [Identity and Access Management (IAM) console](https://console.aws.amazon.com/iam/home), and create a new user: Access Management > Users > Add Users. Select a name, press Next, then you need to attach certain policies (TODO: find out which policies are needed).
3. [Download](https://aws.amazon.com/cli/) and install the `aws` command line interface (CLI). *Update: `quipucamayoc` installs the `awscli` package so this step might not be necessary anymore*.
4. Go to the command line, and type [`aws configure`](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html) to enter your credentials. You need to enter your `Access key` and `Secret Access Key` strings. The `Default region name` can be the [AWS region](https://www.concurrencylabs.com/blog/choose-your-aws-region-wisely/) of your preference (most likely, `aws-east-1`) or left empty. `Default output format` can be left empty.
5. From the command line, you can now run the quipucamayoc command `quipu aws install`. This will setup your AWS account so you can use `Textract` programmatically (i.e., create an S3 bucket, an SNS topic, a SQS queue, a user with the required credentials, and then configure the user, bucket, topic, and queue so they can talk to each other.)
Notes:
- You can avoid steps 3-4 by directly [writing your credentials[(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)] to the `credentials` file.
- If you want to remove all quipucamayoc artifacts from your AWS account, you can run `quipu aws uninstall` from the command line.
## Usage
- From the command line, you can extract tables using AWS via `quipu extract-tables --filename `
## TODO
- [x] Automatically set up Textract pipeline
- [ ] Expose key functions as command line tools
- [ ] Allow parallel (async?) tasks. Useful for OpenCV (CPU-intensive) and Textract calls (IO-intensive). Consider also [uvloop](https://github.com/MagicStack/uvloop)
- [ ] Include Poppler by default on Windows
- [ ] Add mypy/(flake8|black)
### Interesting tools to explore:
- https://github.com/qurator-spk/eynollah
- https://github.com/qurator-spk/sbb_binarization
- https://github.com/leonlulu/DeepLayout
## Contributing
Feel free to submit push requests. For consistency, code should comply with [pep8](https://pypi.python.org/pypi/pep8) (as long as its reasonable), and with the style guides by [@kennethreitz](http://docs.python-guide.org/en/latest/writing/style/) and [google](http://google.github.io/styleguide/pyguide.html). Read more [here](/CONTRIBUTING.md).
## Citation
[(Download BibTex file here)](https://raw.githubusercontent.com/sergiocorreia/quipucamayoc/master/quipucamayoc.bib)
#### As text
-
Sergio Correia, Stephan Luck: “Digitizing Historical Balance Sheet Data: A Practitioner's Guide”, 2022; arXiv:2204.00052.
#### As BibTex
```bibtex
@misc{quipucamayoc,
Author = {Correia, Sergio and Luck, Stephan},
Title = {Digitizing Historical Balance Sheet Data: A Practitioner's Guide},
Year = {2022},
eprint = {arXiv:2204.00052},
journal={arXiv preprint arXiv:2204.00052}
}
```
## Acknowledgments
Quipucamayoc is built upon the work and improvements of many users and developers, from which it was heavily inspired, such as:
- [pdftabextract](https://github.com/WZBSocialScienceCenter/pdftabextract)
It is also relies for most of its work on the following open source projects:
- [Python](https://www.python.org/)
- [NumPy](https://numpy.org/)
- [OpenCV](https://opencv.org/) and [OpenCV-Python](https://github.com/opencv/opencv-python)
- [Poppler](https://poppler.freedesktop.org/)
## License
Quipucamayoc is developed under the [GNU Affero GPL v3 license](https://www.gnu.org/licenses/agpl-3.0.en.html).
## Why "quipucamayoc"?
The _quipucamayocs_ were the Inca empire officials in charge of desciphering (amonst other things) accounting information stored in quipus. Our goal for this package is to act as a sort of quipucamayoc, helping researchers in desciphering and extracting historical information, particularly balance sheets and numerical records.
