https://github.com/patelvivekdev/contextual-chunks
Generate contextual chunks for Retrieval-Augmented Generation (RAG) using LLM
https://github.com/patelvivekdev/contextual-chunks
chunking contextual-chunks llm rag
Last synced: 10 months ago
JSON representation
Generate contextual chunks for Retrieval-Augmented Generation (RAG) using LLM
- Host: GitHub
- URL: https://github.com/patelvivekdev/contextual-chunks
- Owner: patelvivekdev
- License: mit
- Created: 2024-11-21T18:44:07.000Z (about 1 year ago)
- Default Branch: master
- Last Pushed: 2025-01-03T21:31:42.000Z (about 1 year ago)
- Last Synced: 2025-02-04T19:18:22.140Z (11 months ago)
- Topics: chunking, contextual-chunks, llm, rag
- Language: TypeScript
- Homepage: https://contextual-chunks.vercel.app
- Size: 226 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Contextual Chunks Generator
This project demonstrates how to generate contextual chunks for Retrieval-Augmented Generation (RAG) and other use cases using a PDF document. The application processes a PDF file, splits it into chunks, and generates contextual prompts to provide meaningful context for each chunk using a language model.
## Features
- Upload a PDF file to generate contextual chunks.
- Uses Google Generative AI API for generating contextual chunks.
- Displays generated chunks.
- Next.js for server-side rendering and API routes.
## Getting Started
### Prerequisites
- Node.js (version 18 or higher)
- npm or yarn
### Installation
1. Clone the repository:
```bash
git clone https://github.com/patelvivekdev/contextual-chunks.git
cd contextual-chunks
```
2. Install dependencies:
```bash
bun i
```
3. Create a `.env` file based on `.env.example` and add your Google Generative AI API key:
```bash
cp .env.example .env
```
4. Run the development server:
```bash
bun run dev
```
5. Open your browser and navigate to `http://localhost:3000`.
## Dependencies
- `@ai`: Vercel SDK for using AI services.
- `@ai-sdk/google`: SDK for Google Generative AI.
- `@langchain/textsplitters`: Library for splitting text into chunks.
- `pdf-parse`: Library for parsing PDF files.
- `react-dropzone`: Library for handling file uploads.
## License
This project is licensed under the MIT License.