{"id":36495923,"url":"https://github.com/rossumai/docile","last_synced_at":"2026-01-12T02:03:41.048Z","repository":{"id":65777419,"uuid":"554243063","full_name":"rossumai/docile","owner":"rossumai","description":"DocILE: Document Information Localization and Extraction Benchmark","archived":false,"fork":false,"pushed_at":"2024-05-15T15:35:50.000Z","size":1092,"stargazers_count":138,"open_issues_count":4,"forks_count":10,"subscribers_count":13,"default_branch":"main","last_synced_at":"2025-11-14T05:19:24.264Z","etag":null,"topics":["benchmark","document","extraction","information","key","kie","kile","understanding"],"latest_commit_sha":null,"homepage":"https://docile.rossum.ai","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rossumai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-10-19T13:40:53.000Z","updated_at":"2025-11-10T10:50:04.000Z","dependencies_parsed_at":"2024-05-13T08:28:35.367Z","dependency_job_id":"ce0f5fb9-2937-4f73-8546-6c8b24f1f6a7","html_url":"https://github.com/rossumai/docile","commit_stats":{"total_commits":93,"total_committers":7,"mean_commits":"13.285714285714286","dds":"0.26881720430107525","last_synced_commit":"9da2379fc3f875837b9f850faf3912c21fa39899"},"previous_names":[],"tags_count":7,"template":false,"template_full_name":null,"purl":"pkg:github/rossumai/docile","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rossumai%2Fdocile","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rossumai%2Fdocile/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rossumai%2Fdocile/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rossumai%2Fdocile/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rossumai","download_url":"https://codeload.github.com/rossumai/docile/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rossumai%2Fdocile/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28331509,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T00:36:25.062Z","status":"online","status_checked_at":"2026-01-12T02:00:08.677Z","response_time":98,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["benchmark","document","extraction","information","key","kie","kile","understanding"],"created_at":"2026-01-12T02:01:05.002Z","updated_at":"2026-01-12T02:03:41.026Z","avatar_url":"https://github.com/rossumai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DocILE: Document Information Localization and Extraction Benchmark\n[![Tests](https://github.com/rossumai/docile/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/rossumai/docile/actions/workflows/tests.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\nRepository to work with the [DocILE dataset and benchmark](https://docile.rossum.ai/), used in the DocILE'23 CLEF Lab and ICDAR Competition.\nThe competition deadline is on ~~May 10~~ **May 24, 2023** and comes with a **$9000 prize pool**.\n\nThe repository consists of:\n* A python library, `docile`, making it easy to load the dataset, work with its annotations, pdfs and pre-computed OCR and run the evaluation.\n* An interactive [dataset browser notebook](docile/tools/dataset_browser.ipynb) to visualize the document annotations, predictions and evaluation results.\n* Training and inference scripts for [Baseline methods](baselines/) provided together with pretrained checkpoints and results on test and validation sets.\n* [Tutorials](tutorials/) to get quickly started with the repo.\n\nTable of Contents:\n* [Download the dataset](#download-the-dataset)\n* [Installation](#installation)\n* [Predictions format and running evaluation](#predictions-format-and-running-evaluation)\n* [Tasks and evaluation metrics](#tasks-and-evaluation-metrics)\n* [Pre-computed OCR](#pre-computed-ocr)\n* [Development instructions](#development-instructions)\n* [Dataset and benchmark paper and Supplementary Material](#dataset-and-benchmark-paper-and-supplementary-material)\n\n## Download the dataset\n\nFirst you need to obtain a secret token by following the instructions at https://docile.rossum.ai/. Then download and unzip the dataset by running:\n```bash\n./download_dataset.sh TOKEN annotated-trainval data/docile --unzip\n./download_dataset.sh TOKEN test data/docile --unzip\n./download_dataset.sh TOKEN synthetic data/docile --unzip\n./download_dataset.sh TOKEN unlabeled data/docile --unzip\n```\n\nRun `./download_dataset.sh --help` for more options, including how to only show urls (to download\nwith a different tool than curl), how to download smaller unlabeled/synthetic chunks or unlabeled\ndataset without pdfs (with pre-computed OCR only).\n\nYou can also work with the zipped datasets when you turn off image caching (check [Load and sample dataset](tutorials/load_and_sample_dataset.md) tutorial for details).\n\n## Installation\n\n### Option 1: Install as a library\nInstall the library with:\n```shell\npip install docile-benchmark\n```\n\nTo convert pdfs into images, the library uses https://github.com/Belval/pdf2image. On linux you might need to install:\n```shell\napt install poppler-utils\n```\nAnd on macOS:\n\n```shell\nbrew install poppler\n```\n\nNow you have all dependencies to work with the dataset annotations, pdfs, pre-comptued OCR and to run the evaluation. You can install extra dependencies by running the following (although using one of the provided dockerfiles, as explained below, might be easier in this case):\n```shell\npip install \"docile-benchmark[interactive]\"\npip install \"docile-benchmark[ocr]\"\n```\n\nThe first line installs additional dependencies allowing you to use the interactive dataset browser in [docile/tools/dataset_browser.py](docile/tools/dataset_browser.py) and the [tutorials](tutorials/). The second line let's you rerun the OCR predictions from scratch (e.g., if you'd like to run it with different parameters) but to make it work, you might need additional dependencies on your system. Check https://github.com/mindee/doctr for the installation instructions (for pytorch).\n\n### Option 2: Use docker\nThere are two Dockerfiles available with the dependencies preinstalled:\n\n* `Dockerfile` is a lighter, CPU-only, version with all necessary dependencies to use the dataset with the pre-computed OCR and interactive browser.\n* `Dockerfile.gpu` has CUDA GPU support and contains additional dependencies needed to recompute OCR predictions from scratch (not needed for standard usage).\n\nYou can use docker compose to manage the docker images. First update the settings in `docker-compose.yml` and the port for jupyter lab in `.env`. Then build the image with:\n```shell\ndocker compose build jupyter[-gpu]\n```\nwhere `jupyter` uses `Dockerfile` and `jupyter-gpu` uses `Dockerfile.gpu`. You can then start the jupyter server:\n```shell\ndocker compose up -d jupyter[-gpu]\n```\n\nJupyter lab can be then accessed at `https://127.0.0.1:${JUPYTER_PORT}` (retrieve the token from logs with `docker compose logs jupyter[-gpu]`). You can also login to the container with:\n```shell\ndocker compose exec jupyter bash\n```\nAfter that run `poetry shell` to activate the virtual environment with the `docile` library and its dependencies installed.\n\n## Predictions format and running evaluation\n\nTo evaluate predictions for tasks KILE or LIR on the test set, you have to make a submission to the benchmark on the [Robust Reading Competition](https://rrc.cvc.uab.es/?ch=26) website. To evaluate on the validation set, use the following command:\n```bash\ndocile_evaluate \\\n  --task LIR \\\n  --dataset-path path/to/dataset[.zip] \\\n  --split val \\\n  --predictions path/to/predictions.json \\\n  --evaluate-x-shot-subsets \"0,1-3,4+\" \\  # default, show evaluation for 0-shot, few-shot and many-shot layout clusters\n  --evaluate-synthetic-subsets \\  # optional, show evaluation on layout clusters with available synthetic data\n  --evaluate-fieldtypes \\  # optional, show breakdown per fieldtype\n  --evaluate-also-text \\  # optional, evaluate if the text prediction is correct\n  --store-evaluation-result LIR_val_eval.json  # optional, it can be loaded in the dataset browser\n```\n\nRun `docile_evaluate --help` for more information on the options. You can also run `docile_print_evaluation_report --evaluation-result-path LIR_val_eval.json` to print the results of a previously computed evaluation. It is also possible to run evaluation or load the evaluation result directly from code which provides even more options, such as computing metrics just for a single document, specific subset of documents (i.e., documents belonging to the same layout cluster) etc. Check [docile/evaluation/evaluate.py](docile/evaluation/evaluate.py) module for more details.\n\nPredictions need to be stored in a single json file (for each task separately) containing a mapping from `docid` to the predictions for that document, i.e.:\n```json\n{\n    \"docid1\": [\n        {\n            \"page\": 0,\n            \"bbox\": [0.2, 0.1, 0.4, 0.5],\n            \"fieldtype\": \"line_item_order_id\",\n            \"line_item_id\": 3,\n            \"score\": 0.8,\n            \"text\": \"Order 38\",\n            \"use_only_for_ap\": true\n        },\n        \"...\"\n    ],\n    \"docid2\": [{\"...\": \"...\"}, \"...\"],\n    \"...\"\n}\n```\nExplanation of the individual fields of the predictions:\n  * `page`: page index (from zero) the prediction belongs to\n  * `bbox`: relative coordinates (from 0 to 1) representing the `left`, `top`, `right`, `bottom` sides of the bbox, respectively\n  * `fieldtype`: the fieldtype (sometimes called category or key) of the prediction\n  * `line_item_id`: ID of the line item. This should be a different number for each line item, the order does not matter. Omit for KILE predictions.\n  * `score` [optional]: the confidence for this prediction, can be omitted (in that case predictions are taken in the order in which they are stored in the list)\n  * `text` [optional]: text of the prediction, evaluated in a secondary metric only (when `--evaluate-also-text` is used)\n  * `use_only_for_ap` [optional, default is False]: only use the prediction for AP metric computation, not for f1, precision and recall (useful for less confident predictions).\n\nYou can use [Field class](docile/dataset/field.py) to work with the predictions in code, in which case you can store them in a json file in the required format by using the `docile.dataset.store_predictions` function. You can also use the [BBox class](docile/dataset/bbox.py) to easily convert between relative and absolute coordinates (relative coordinates are used everywhere by default).\n\nUse [docile/tools/print_results.py](docile/tools/print_results.py) to display results of your predictions formatted in a table (choosing from table styles like github, latex or csv).\n\n## Tasks and evaluation metrics\n\nThe DocILE benchmark comes with two tracks, Key Information Localization and Extraction (KILE) and Line Item Recognition (LIR), as described in the [dataset and benchmark paper](#dataset-and-benchmark-paper). In both tracks, the goal is to localize key information of pre-defined categories, i.e., for each document, detect fields with their `fieldtype`, location (`page` and `bbox`) and optionally `text`. For LIR, fields have to be additionally grouped into line items. A Line Item (LI) is a tuple of fields (e.g., description, quantity, and price) describing a single object instance to be extracted, e.g., a row in a table.\n\nEvaluated metrics are Average Precision (AP), F1, precision and recall, computed over all fields across all documents and field types (micro-average). A predicted and a ground truth field can be matched if they have the same `fieldtype` and if they are in the same location, which is decided by the overlap of their bounding boxes with the [provided OCR words](#pre-computed-ocr), described more precisely in the [dataset and benchmark paper](#dataset-and-benchmark-paper):\n\n\u003cimg width=\"500\" alt=\"Definition of Pseudo-Character-Centers and correct localization.\" src=\"https://user-images.githubusercontent.com/1220288/222190727-e851a1d9-9c91-438e-bb8d-7ca2066620a4.png\"\u003e\n\nFor LIR, the fields have to additionally belong to corresponding Line Items. The mapping between `line_item_id` for predicted and ground truth fields is decided by taking the maximum matching which maximizes the total number of matched fields, when considering only fields with `use_only_for_ap=False` (because `use_only_for_ap=True` can be used to include also less confident predictions for AP computation which could negatively affect this matching).\n\nFor both KILE and LIR, the metrics are then computed like this:\n\n* For each document page, find matching between predicted and ground truth fields. If more predicted fields match the same ground truth field, the one with higher `score` is used (but primarily the one with `use_only_for_ap=False`).\n* Order all predictions by descending `score` (but again, predictions with `use_only_for_ap=True` come last). Break ties by the original order in which predictions were provided. See [evaluate.py](docile/evaluation/evaluate.py) for the precise ordering rules. Notice that the order does not influence F1, precision and recall but it influences AP.\n* Compute F1, precision and recall by counting number of true positives and false positives/negatives and AP by iteratively adding predictions and updating precision and recall. We use the COCO style of AP where \"gaps are filled\", i.e., the precision-recall curve becomes non-increasing.\n\n## Pre-computed OCR\n\nPre-computed OCR is provided with the dataset. The prediction was done using the [DocTR](https://github.com/mindee/doctr) library. On top of that, word boxes were snapped to text (check the code in [docile/dataset/document_ocr.py](docile/dataset/document_ocr.py)). These snapped word boxes are used in evaluation.\n\nTo get the OCR words for a single page, call `document.ocr.get_all_words(page)`, optionally passing `snapped=True` to get the snapped bounding boxes. In some rare cases, the list of words might be empty (when the page is blank). Also notice the predicted words are grouped into blocks. For some models it might be beneficial to re-order the words primarily by lines.\n\nWhile it should not be needed, it is possible to (re)generate OCR from scratch (including the snapping) with the provided `Dockerfile.gpu`. Just delete `DATASET_PATH/ocr` directory and then access the ocr for each document and page with `document.ocr.get_all_words(page, snapped=True)`.\n\n## Development instructions\n\nFor development, install [poetry](https://python-poetry.org/docs/) and run `poetry install`. Start a shell with the virtual environment activated with `poetry shell`. No other dependencies are needed to run pre-commit and the tests. It's recommended to use docker (as explained above) if you need the extra (interactive or ocr) dependencies.\n\nInstall pre-commit with `pre-commit install` (don't forget you need to prepend all commands with `poetry run ...` if you did not run `poetry shell` first).\n\nRun tests by calling `pytest tests`.\n\n## Dataset and benchmark paper and Supplementary Material\nThe dataset, the benchmark tasks and the evaluation criteria are described in detail in the [dataset paper](https://arxiv.org/abs/2302.05658) which was accepted to ICDAR 2023. The provided link is to arXiv version that includes Supplementary Material. To cite the dataset, please use the following BibTeX entry:\n```\n@misc{simsa2023docile,\n    title={{DocILE} Benchmark for Document Information Localization and Extraction},\n    author={{\\v{S}}imsa, {\\v{S}}t{\\v{e}}p{\\'a}n and {\\v{S}}ulc, Milan and U{\\v{r}}i{\\v{c}}{\\'a}{\\v{r}}, Michal and Patel, Yash and Hamdi, Ahmed and Koci{\\'a}n, Mat{\\v{e}}j and Skalick{\\`y}, Maty{\\'a}{\\v{s}} and Matas, Ji{\\v{r}}{\\'\\i} and Doucet, Antoine and Coustaty, Micka{\\\"e}l and Karatzas, Dimosthenis},\n    url = {https://arxiv.org/abs/2302.05658},\n    journal={arXiv preprint arXiv:2302.05658},\n    year={2023}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frossumai%2Fdocile","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frossumai%2Fdocile","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frossumai%2Fdocile/lists"}