https://github.com/activeloopai/deeplake-bedrock

https://github.com/activeloopai/deeplake-bedrock

Last synced: 10 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/activeloopai/deeplake-bedrock
• Owner: activeloopai
• Created: 2024-11-10T16:16:26.000Z (about 1 year ago)
• Default Branch: main
• Last Pushed: 2024-11-10T16:27:28.000Z (about 1 year ago)
• Last Synced: 2025-04-09T15:10:55.135Z (10 months ago)
• Language: Jupyter Notebook
• Size: 938 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Awesome Lists containing this project

README

          

# Multimodal Retrieval with Deep Lake, Bedrock, and AI Integrations 

This project showcases a multimodal retrieval system combining the **Deep Lake**  REST API, **Bedrock** , and **OpenAI**  for image and text processing. It allows users to query a dataset with text prompts, retrieve relevant images, and leverage AI models for context-aware responses.

## Overview 

This solution is designed to enable multimodal data retrieval and answer generation by:

 

- **Querying Datasets** : Supports querying datasets in Deep Lake and retrieving top matches based on text-based queries.

 

- **AI-Powered Insights** : Uses Bedrock and OpenAI models for answering questions about retrieved images or text segments.

 

- **Deep Memory** : Integrates deep memory retrieval to enhance context-aware responses, leveraging the Deep Lake database for improved AI-driven answers.

## Project Components 

1. Main Retrieval and Processing (`main.py`)

    The core script:

    - Queries Deep Lake datasets based on user-defined prompts.

    - Uses Bedrock and Deep Lake integrations to retrieve relevant images.

    

    - Leverages functions from `bedrock_code.py` and `deeplake_deepmemory.py` to generate context-aware answers from images and text.

    - Saves and displays retrieved images for each query.

2. Bedrock Integration (`bedrock_code.py`)

    This file provides:

    

    - Functions to send messages to Bedrock’s AI models (e.g., Claude 3 Sonnet) using the AWS `boto3` client.

    - Supports both image and text-based question-answering by converting inputs into compatible formats for Bedrock’s AI models.

    - A sample use case for processing an image with Bedrock’s API.

3. Deep Memory and Embedding (`deeplake_deepmemory.py`)

    This module includes:

    - Functions to retrieve context from a Deep Lake vector database based on user queries.

    - An embedding function using OpenAI’s embedding model for efficient text similarity and context-aware retrieval.

    - Options to enable or disable deep memory

4. Utility Functions (`utils.py`)

    Helper functions to:

    - Convert images to byte format for API compatibility.

    - Structure messages for both text and image prompts, formatting them to interact with Bedrock and OpenAI models.

    - Generate embeddings for text inputs using OpenAI’s API.

5. Notebook Examples (`notebook_bedrock.ipynb` and `notebook.ipynb`)

    Notebooks demonstrate how to use the multimodal retrieval system:

 

    - **Multimodal Retrieval with Bedrock and Deep Lake** : Examples of setting up queries, sending them to the Deep Lake API, and processing responses.

    

    - **Visualization** : Displays images retrieved based on query results and shows how Bedrock can answer questions about these images.

6. Requirements (`requirements.txt`)

The necessary Python libraries for this project:

 

    - `deeplake V3`: For handling datasets and vector stores.

    

    - `openai`: To access embedding functions for text processing.

## Installation 

Install the necessary dependencies:

```bash

pip install -r requirements.txt

```

## Environment Setup 

Define environment variables for secure access to APIs:

 

- `TOKEN`: Deep Lake API access token.

 

- `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`: AWS credentials for Bedrock.

 

- `ACTIVELOOP_TOKEN`: Token for ActiveLoop’s Deep Lake.

 

- `OPENAI_API_KEY`: OpenAI API key for embedding functions.