Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/clearedge-ai/clearedge
Build a RAG preprocessing pipeline
https://github.com/clearedge-ai/clearedge
document-parser haystack langchain llamaindex llm ocr pdf pdf-ocr-extraction pdf-to-json pdf-to-text rag-pipeline retrieval-augmented-generation table-detection table-recognition
Last synced: 9 days ago
JSON representation
Build a RAG preprocessing pipeline
- Host: GitHub
- URL: https://github.com/clearedge-ai/clearedge
- Owner: Clearedge-AI
- License: apache-2.0
- Created: 2024-03-15T00:12:50.000Z (8 months ago)
- Default Branch: main
- Last Pushed: 2024-04-07T02:43:02.000Z (7 months ago)
- Last Synced: 2024-10-22T22:48:37.283Z (15 days ago)
- Topics: document-parser, haystack, langchain, llamaindex, llm, ocr, pdf, pdf-ocr-extraction, pdf-to-json, pdf-to-text, rag-pipeline, retrieval-augmented-generation, table-detection, table-recognition
- Language: Jupyter Notebook
- Homepage:
- Size: 24.7 MB
- Stars: 10
- Watchers: 3
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
[](https://colab.research.google.com/github/Clearedge-AI/clearedge/blob/main/notebooks/quicktour.ipynb)
## Overview
Clearedge is a Python package designed to simplify the process of extracting raw text and metadata from documents. You can use it to retrieve not only the text but also valuable metadata including titles, subheadings, page numbers, file names, bounding box (bbox) coordinates, and chunk types. Whether you're working on document analysis, data extraction projects, or building a RAG app with LLM, Clearedge provides a straightforward and efficient solution.
## Features
- Text Extraction: Extract raw text from documents (currently supports pdf only. other file types coming soon).
- Metadata Retrieval: Obtain metadata such as subheadings, page numbers, file names, bounding boxes and more.
- Bounding Box Coordinates: Access bbox coordinates for text chunks, enabling spatial analysis of text placement within documents.
- Chunk Type Identification: Identify types of text chunks (e.g., table, text and more) for advanced content analysis.
- Support for Multiple Formats (coming soon): Compatible with popular document formats, ensuring broad usability.
## Installation
### Prerequisites
To install clearedge, you will need Python 3.8 or later.
Since we use Tesseract, you will need extra dependencies.
For MacOS users, you need to run:
```shell
brew install tesseract
```
For ubuntu users, you need to run:
```shell
sudo apt install tesseract-ocr
```
### Latest release
You can then install the latest release of the package using [pypi](https://pypi.org/project/clearedge/) as follows:
```bash
pip install clearedge
```
## Quick Start
Here's a simple example to get you started with clearedge:
```python
from clearedge.reader.pdf import process_pdf
# Call the extractor with the path to your document
chunks = process_pdf('/path/to/your/document.pdf', use_ocr=True) # do not add use_ocr for faster processing. output is less accurate without ocr.
# Extract text and metadata
for chunk in chunks:
text, metadata = chunk.text, chunk.metadata
print(text) # Accessing extracted text
print(metadata.to_dict()) # Accessing metadata
```
## Documentation
For more detailed information on all the features and functionalities of Clearedge, please refer to the official documentation (coming soon).
## Contributing
Contributions to Clearedge are welcome! If you have suggestions for improvements or bug fixes, please feel free to:
Open an issue to discuss what you would like to change.
Submit pull requests for us to review.
## Citation
If you wish to cite this project, feel free to use this [BibTeX](http://www.bibtex.org/) reference:
```bibtex
@misc{clearedge2024,
title={clearedge: RAG preprocessor},
author={Clearedge AI},
year={2024},
publisher = {GitHub},
howpublished = {\url{https://github.com/Clearedge-AI/clearedge}}
}
```
## License
Clearedge is released under the Apache 2.0 License. See the [`LICENSE`](https://github.com/Clearedge-AI/clearedge?tab=Apache-2.0-1-ov-file#readme) file for more details.
## Acknowledgments
This project was inspired by the need for a simple, yet comprehensive tool for document analysis and metadata extraction. We thank all contributors and users for their support and feedback. Clearedge aims to be a valuable tool for developers, researchers, and anyone involved in processing and analyzing document content. We hope it simplifies your projects and helps you achieve your goals more efficiently.