https://github.com/scut-dlvclab/rfund
[MM'2024] Official release of RFUND introduced in the MM'2024 paper "PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction"
https://github.com/scut-dlvclab/rfund
document-ai document-understanding key-information-extraction ocr visual-information-extraction
Last synced: 2 months ago
JSON representation
[MM'2024] Official release of RFUND introduced in the MM'2024 paper "PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction"
- Host: GitHub
- URL: https://github.com/scut-dlvclab/rfund
- Owner: SCUT-DLVCLab
- Created: 2024-03-20T13:46:36.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2024-12-04T06:22:39.000Z (6 months ago)
- Last Synced: 2025-01-20T07:33:30.130Z (4 months ago)
- Topics: document-ai, document-understanding, key-information-extraction, ocr, visual-information-extraction
- Homepage:
- Size: 723 KB
- Stars: 18
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# RFUND: Relabeling FUNSD and XFUND dataset for Real-world Form Understanding
## Introduction
This repository is the official release of RFUND introduced in the MM'2024 paper "PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction".
RFUND is a relabeled version of FUNSD ([paper](https://ieeexplore.ieee.org/abstract/document/8892998),[dataset-release](https://guillaumejaume.github.io/FUNSD/)) and XFUND ([paper](https://aclanthology.org/2022.findings-acl.253/), [dataset-release](https://github.com/doc-analysis/XFUND)) dataset. We observed the following issues in their original annotations:
1. **Entity (block) level OCR results**. Real-world OCR engines usually produce line-level results, while the annotations in FUNSD and XFUND are at the entity (block) level. Text lines within the same entity are aggregated and serialized in human reading order, simplifying the task scope and failing to reflect real-world challenges.
2. **Inconsistent labeling granularity**. In FUNSD, while most contents are annotated at the entity level, multi-line entities with first-line indentation are annotated separately, in which the first line is split out and the rest are aggregated. XFUND exhibits variable granularity in annotations, with some contents labelled at the entity level and others at the line level. Such inconsistent labelling standards can hinder model training.
3. **Erroneous category annotations**. Entities in FUNSD/XFUND are categorized as "header", "question", "answer" and "other". We observed that certain entities in both FUNSD and XFUND have category labels that differ from human understanding. In the following example, the *answer* entity “Client confirmed agreement ...” was labelled as *other*, while the *other* entity “CONFIDENTIAL” was labelled as the *question*.
Our main modifications are as follows:
1. **Line-level annotation**: We relabeled the original dataset at the line level, which is more consistent with the output of real-world OCR engines.
2. **Linking annotation rectification**: We removed the redundant linking annotations caused by the inconsistent labelling granularity.
3. **Category annotation rectification**: We rectified the category annotations to make them consistent with human understanding.
## Usage
This repository only contains the [RFUND annotations](https://github.com/SCUT-DLVCLab/RFUND/releases). You can download the images from the original release of FUNSD and XFUND:
- [FUNSD Images](https://guillaumejaume.github.io/FUNSD/)
- [XFUND Images](https://github.com/doc-analysis/XFUND/releases/tag/v1.0)
The annotation JSON files are named as `{language}.{data_split}.json`. Language `en` refers to the FUNSD dataset. Contents in the annotation files are organized as follows:
```jsonc
{
"documents": [ // list that contains annotations for each sample
{
"uid": "86236474_6476",
"img": { // image information
"fname": "86236474_6476.png",
"height": 1000,
"width": 754
},
"entities": [
{
"id": 0, // entity id
"label": "question", // entity category (SER labels)
"lines": [ // list of lines in the entity, arranged in reading order
{
"id": "0", // line id
"text": "TO:", // line text content
"bbox": [ // line bounding box
47,
102,
74,
120
]
},
// ...
]
},
// ...
],
"relations": {
"kv_entity": [ // list of key-value pair relations (RE labels)
{
"from_id": 0, // question entity id
"to_id": 1, // answer entity id
},
// ...
],
"line_grouping": [ // list of line grouping relations
{
"from_id": "0", // id of the current line
"to_id": "1", // id of the next line
},
// ...
]
}
},
// ...
]
}
```
## License
RFUND is derived from FUNSD and XFUND, and its license adheres to the same terms and conditions as the original datasets. Please refer to the original licenses ([FUNSD](https://guillaumejaume.github.io/FUNSD/work/), [XFUND](https://github.com/doc-analysis/XFUND?tab=readme-ov-file#license)) for more details.
## Citation
If you find RFUND useful, please consider citing our paper:
```
@inproceedings{lin2024peneo,
title={PEneo: Unifying Line Extraction, Line Grouping, and Entity Linking for End-to-end Document Pair Extraction},
author={Lin, Zening and Wang, Jiapeng and Li, Teng and Liao, Wenhui and Huang, Dayi and Xiong, Longfei and Jin, Lianwen},
booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
pages={5171--5180},
year={2024}
}
```