https://github.com/oztrkoguz/visquerypdf

It automatically describes images in PDF files and generates questions from these descriptions. With its advanced RAG structure, it directs these questions directly to PDF text content, providing comprehensive information extraction and analysis.
https://github.com/oztrkoguz/visquerypdf

agent clip langchain langchain-agent langchain-python langchain-rag llm-agent multi-agent-systems ollama

Last synced: 3 months ago
JSON representation

It automatically describes images in PDF files and generates questions from these descriptions. With its advanced RAG structure, it directs these questions directly to PDF text content, providing comprehensive information extraction and analysis.

• Host: GitHub
• URL: https://github.com/oztrkoguz/visquerypdf
• Owner: oztrkoguz
• License: apache-2.0
• Created: 2024-06-29T05:17:51.000Z (12 months ago)
• Default Branch: main
• Last Pushed: 2024-06-29T06:57:15.000Z (12 months ago)
• Last Synced: 2025-01-25T15:15:08.460Z (5 months ago)
• Topics: agent, clip, langchain, langchain-agent, langchain-python, langchain-rag, llm-agent, multi-agent-systems, ollama
• Language: Python
• Homepage:
• Size: 1.56 MB
• Stars: 11
• Watchers: 2
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE.md

Awesome Lists containing this project

README

        
# VisQueryPDF

### The aim of the project is to verify the alignment between images and texts in documents.

### 2 different methods were discussed.

## 1.Method 

![yontem1](https://github.com/oztrkoguz/VisQueryPDF/assets/101019436/65b62ab9-c98a-44db-bee1-abd71e6d0714)

Images automatically extracted from the document were described using a VLM agent structure. Using the description results, questions were generated with a question generation agent. Subsequently, these questions were posed to the document using the RAG system, and answers were verified.

## 2.Method

![Adsz-2024-06-29-0711](https://github.com/oztrkoguz/VisQueryPDF/assets/101019436/2a0dd56e-8839-446c-b42b-3758c577cf86)

Images and texts are automatically extracted from the document. Text data undergoes processing using a summarization agent to obtain a concise summary. Subsequently, embeddings of images and texts are extracted using the CLIP model, and their similarities are compared.

### The first method achieved a similarity rate of 60%, whereas the other method showed similarities around 33%.

## Usage

```

git clone https://github.com/oztrkoguz/VisQueryPDF.git

cd VisQueryPDF

python main.py

```

## Requirements

```

Python > 3.10

langchain==0.2.6

langchain-chroma==0.1.1

langchain-community==0.0.38

langchain-core==0.1.52

langchain-openai==0.0.5

langchain-text-splitters==0.2.1

langsmith==0.1.82

ollama==0.2.1

```