https://github.com/zakariaf/rag-cache

Last synced: 5 months ago
JSON representation
Host: GitHub
URL: https://github.com/zakariaf/rag-cache
Owner: zakariaf
Created: 2025-11-11T07:33:18.000Z (8 months ago)
Default Branch: main
Last Pushed: 2025-11-28T10:40:47.000Z (7 months ago)
Last Synced: 2025-11-30T18:10:18.460Z (7 months ago)
Language: Python
Size: 776 KB
Stars: 4
Watchers: 0
Forks: 2
Open Issues: 17
Metadata Files:
- Readme: README.md
Awesome Lists containing this project

README

          # RAG Cache

A high-performance caching layer for Large Language Model (LLM) queries with exact and semantic matching capabilities.

## 🎯 What is RAG Cache?

RAG Cache sits between your application and LLM providers (OpenAI, Anthropic) to:

- **💰 Save Money** - Avoid paying for duplicate queries

- **⚡ Reduce Latency** - Return cached responses in ~2ms instead of ~10 seconds

- **🛡️ Increase Reliability** - Reduce dependency on external APIs

### How It Works

```

User Query → Exact Match (Redis) → Semantic Match (Qdrant) → LLM (if needed)

                   ↓                       ↓                      ↓

              ~1ms hit              ~450ms hit              ~8,500ms

```

## 🚀 Quick Start

### Prerequisites

- Python 3.11+

- Docker & Docker Compose

- OpenAI API key

### Installation

```bash

# Clone the repository

git clone 

cd "Rag cache"

# Create virtual environment

python3.11 -m venv venv

source venv/bin/activate

# Install dependencies

pip install -r requirements.txt

# Configure environment

cp .env.example .env

# Edit .env and add your OPENAI_API_KEY

# Start Redis & Qdrant

docker-compose up -d redis qdrant

# Run the application

uvicorn app.main:app --host 0.0.0.0 --port 8000

```

### Verify Installation

```bash

# Health check

curl http://localhost:8000/health

```

Expected response:

```json

{

  "status": "healthy",

  "environment": "development",

  "version": "0.1.0"

}

```

## 📡 API Usage

### Query Endpoint

```bash

curl -X POST http://localhost:8000/api/v1/query \

  -H "Content-Type: application/json" \

  -d '{

    "query": "What is machine learning?",

    "use_cache": true

  }'

```

**First Request (Cache Miss):**

```json

{

  "response": "Machine learning is a subset of AI...",

  "provider": "openai",

  "model": "gpt-4o-mini-2024-07-18",

  "usage": {

    "prompt_tokens": 12,

    "completion_tokens": 418,

    "total_tokens": 430

  },

  "cache_info": {

    "cache_hit": false,

    "cache_type": null

  },

  "latency_ms": 10090.93

}

```

**Second Request (Exact Cache Hit - Redis):**

```json

{

  "response": "Machine learning is a subset of AI...",

  "cache_info": {

    "cache_hit": true,

    "cache_type": "exact"

  },

  "latency_ms": 1.11

}

```

**Similar Query (Semantic Cache Hit - Qdrant):**

```bash

curl -X POST http://localhost:8000/api/v1/query \

  -H "Content-Type: application/json" \

  -d '{"query": "Can you explain machine learning?", "use_cache": true}'

```

```json

{

  "response": "Machine learning is a subset of AI...",

  "cache_info": {

    "cache_hit": true,

    "cache_type": "semantic",

    "similarity_score": 0.8563

  },

  "latency_ms": 457.89

}

```

### Other Endpoints

| Endpoint | Method | Description |

|----------|--------|-------------|

| `/health` | GET | Health check |

| `/docs` | GET | Swagger UI documentation |

| `/api/v1/query` | POST | Process query with caching |

| `/api/v1/metrics` | GET | Application metrics |

| `/api/v1/metrics/prometheus` | GET | Prometheus format metrics |

## 📊 Real Performance Tests

Tests performed on November 27, 2025:

### Cache Hit Performance

| Test | Cache Type | Latency | Speedup |

|------|------------|---------|---------|

| First Query (Cache Miss) | LLM Call | 8,573ms | baseline |

| Same Query (Exact Match) | **Redis** | 1.11ms | **7,723x faster** |

| Similar Query (Semantic Match) | **Qdrant** | 457ms | **18.7x faster** |

| Different Query (Cache Miss) | LLM Call | 8,154ms | - |

### Semantic Cache Example

```bash

# Original cached query

"What is deep learning?"

# Similar query (semantic match)

"Can you explain deep learning to me?"

# → Similarity Score: 0.856

# → Latency: 457ms (instead of ~8,500ms)

# → Saved: 811 tokens

```

### Cache Types Explained

| Cache | Storage | Latency | Use Case |

|-------|---------|---------|----------|

| **Exact** | Redis | ~1ms | Identical queries |

| **Semantic** | Qdrant | ~450ms | Similar questions |

| **Miss** | LLM API | ~8,500ms | New unique queries |

## 📈 Metrics

```bash

curl http://localhost:8000/api/v1/metrics

```

```json

{

  "application": {

    "name": "RAGCache",

    "environment": "development",

    "version": "0.1.0"

  },

  "pipeline": {

    "total_requests": 0,

    "cache_hits": 0,

    "cache_hit_rate": 0.0,

    "avg_latency_ms": 0.0

  },

  "cache": {

    "total_keys": 286,

    "memory_used_bytes": 1585248,

    "hits": 2,

    "misses": 2032,

    "hit_rate": 0.1

  },

  "config": {

    "semantic_cache_enabled": true,

    "exact_cache_enabled": true,

    "similarity_threshold": 0.85,

    "cache_ttl_seconds": 3600

  }

}

```

### Prometheus Metrics

```bash

curl http://localhost:8000/api/v1/metrics/prometheus

```

```

# HELP ragcache_info Application information

# TYPE ragcache_info gauge

ragcache_info{version="0.1.0",environment="development"} 1

# HELP ragcache_requests_total Total requests processed

# TYPE ragcache_requests_total counter

ragcache_requests_total 0

# HELP ragcache_cache_hits_total Total cache hits

# TYPE ragcache_cache_hits_total counter

ragcache_cache_hits_total 0

# HELP ragcache_redis_keys Total Redis keys

# TYPE ragcache_redis_keys gauge

ragcache_redis_keys 283

```

## 🏗️ Architecture

```

┌─────────────────────────────────────────────────────────────────┐

│                    Your Application                              │

└────────────────────────────────┬────────────────────────────────┘

                                 │ HTTP

                                 ▼

┌─────────────────────────────────────────────────────────────────┐

│                      RAG Cache (FastAPI)                         │

│                                                                  │

│  ┌──────────────────────────────────────────────────────────┐   │

│  │                    Query Service                          │   │

│  │  1. Check Redis (exact match)                             │   │

│  │  2. Check Qdrant (semantic match)                         │   │

│  │  3. Call LLM if no cache hit                              │   │

│  │  4. Store response in cache                               │   │

│  └──────────────────────────────────────────────────────────┘   │

│                                                                  │

│  ┌────────────┐  ┌────────────┐  ┌──────────────────────────┐   │

│  │   Redis    │  │   Qdrant   │  │    LLM Providers         │   │

│  │  (Exact)   │  │ (Semantic) │  │  OpenAI / Anthropic      │   │

│  └────────────┘  └────────────┘  └──────────────────────────┘   │

└─────────────────────────────────────────────────────────────────┘

```

## ⚙️ Configuration

### Environment Variables

| Variable | Default | Description |

|----------|---------|-------------|

| `OPENAI_API_KEY` | - | OpenAI API key (required) |

| `ANTHROPIC_API_KEY` | - | Anthropic API key (optional) |

| `REDIS_HOST` | `localhost` | Redis host |

| `QDRANT_HOST` | `localhost` | Qdrant host |

| `SEMANTIC_SIMILARITY_THRESHOLD` | `0.85` | Similarity threshold (0.0-1.0) |

| `CACHE_TTL_SECONDS` | `3600` | Cache TTL in seconds |

| `DEFAULT_MODEL` | `gpt-3.5-turbo` | Default LLM model |

## 📁 Project Structure

```

app/

├── api/              # FastAPI routes & middleware

│   ├── routes/       # API endpoints

│   └── middleware/   # Auth, rate limiting, logging

├── cache/            # Redis & Qdrant clients

├── llm/              # LLM provider integrations

│   ├── openai_provider.py

│   └── anthropic_provider.py

├── embeddings/       # Text embedding generation

├── pipeline/         # Query processing pipeline

├── monitoring/       # Prometheus metrics & alerts

├── optimization/     # Performance optimization

├── models/           # Pydantic data models

├── services/         # Business logic

└── utils/            # Utilities (logger, hasher)

```

## 🧪 Running Tests

```bash

# Run all tests

pytest

# Run with coverage

pytest --cov=app

# Run specific test file

pytest tests/unit/test_config.py -v

```

## 🐳 Docker

### Start Services

```bash

# Start all services

docker-compose up -d

# Start only Redis and Qdrant

docker-compose up -d redis qdrant

# View logs

docker-compose logs -f

# Stop services

docker-compose down

```

### Production Deployment

```bash

# Use production compose file

docker-compose -f docker-compose.prod.yml up -d

# Or use deployment script

./scripts/deploy.sh production

```

## 💰 Cost Savings Example

| Scenario | Without Cache | With Cache (50% hit rate) |

|----------|---------------|---------------------------|

| 10,000 queries/day | ~$50/day | ~$25/day |

| Average latency | ~10,000ms | ~5,000ms |

| P95 latency | ~15,000ms | ~2ms (cache hit) |

## 📚 Documentation

- [API Documentation](docs/API.md)

- [Architecture](docs/ARCHITECTURE.md)

- [Deployment Guide](docs/DEPLOYMENT.md)

- [Configuration Guide](docs/CONFIGURATION.md)

- [Testing Guide](docs/TESTING.md)

- [Troubleshooting](docs/TROUBLESHOOTING.md)

- [Security Best Practices](docs/SECURITY.md)

- [Performance Tuning](docs/PERFORMANCE.md)

## 🛠️ Tech Stack

| Component | Technology |

|-----------|------------|

| Language | Python 3.11 |

| Framework | FastAPI |

| Exact Cache | Redis 7.2 |

| Semantic Cache | Qdrant 1.6 |

| LLM Providers | OpenAI, Anthropic |

| Embeddings | sentence-transformers |

| Testing | pytest |

| Containerization | Docker |

## 🤝 Contributing

See [CONTRIBUTING.md](docs/CONTRIBUTING.md) for guidelines.

## 📄 License

MIT License - See [LICENSE](LICENSE) file.

---

**Built with ❤️**
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/zakariaf/rag-cache

Awesome Lists containing this project

README
