Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/codefuse-ai/modelcache
A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.
https://github.com/codefuse-ai/modelcache
llm semantic-cache
Last synced: 5 days ago
JSON representation
A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.
- Host: GitHub
- URL: https://github.com/codefuse-ai/modelcache
- Owner: codefuse-ai
- License: other
- Created: 2023-11-01T01:56:10.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2025-01-03T08:11:25.000Z (about 2 months ago)
- Last Synced: 2025-02-07T18:02:57.895Z (12 days ago)
- Topics: llm, semantic-cache
- Language: Python
- Homepage:
- Size: 2.84 MB
- Stars: 916
- Watchers: 21
- Forks: 49
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
ModelCache
## Contents
- [Contents](#contents)
- [News](#news)
- [Introduction](#introduction)
- [Architecture](#architecture)
- [Quick start](#quick-start)
- [Dependencies](#dependencies)
- [Start service](#start-service)
- [Start demo](#start-demo)
- [Service Startup With Docker-compose](#service-startup-with-docker-compose)
- [Start normal service](#start-normal-service)
- [Visit the service](#visit-the-service)
- [Write cache](#write-cache)
- [Query cache](#query-cache)
- [Clear cache](#clear-cache)
- [Function comparison](#function-comparison)
- [Features](#features)
- [Todo List](#todo-list)
- [Adapter](#adapter)
- [Embedding model\&inference](#embedding-modelinference)
- [Scalar Storage](#scalar-storage)
- [Vector Storage](#vector-storage)
- [Ranking](#ranking)
- [Service](#service)
- [Acknowledgements](#acknowledgements)
- [Contributing](#contributing)
## News
- π₯π₯[2024.10.22] Added tasks for 1024 developer day.
- π₯π₯[2024.04.09] Added Redis Search to store and retrieve embeddings in multi-tenant. This can reduce the interaction time between Cache and vector databases to 10ms.
- π₯π₯[2023.12.10] Integrated LLM embedding frameworks such as 'llmEmb', 'ONNX', 'PaddleNLP', 'FastText', and the image embedding framework 'timm' to bolster embedding functionality.
- π₯π₯[2023.11.20] Integrated local storage, such as sqlite and faiss. This enables you to initiate quick and convenient tests.
- [2023.08.26] codefuse-ModelCache...
### Introduction
Codefuse-ModelCache is a semantic cache for large language models (LLMs). By caching pre-generated model results, it reduces response time for similar requests and improves user experience.
This project aims to optimize services by introducing a caching mechanism. It helps businesses and research institutions reduce the cost of inference deployment, improve model performance and efficiency, and provide scalable services for large models. Through open-source, we aim to share and exchange technologies related to large model semantic cache.
## Architecture
## Quick start
You can find the start script in `flask4modelcache.py` and `flask4modelcache_demo.py`.
- `flask4modelcache_demo.py`: A quick test service that embeds SQLite and FAISS. No database configuration required.
- `flask4modelcache.py`: The standard service that requires MySQL and Milvus configuration.
### Dependencies
- Python: V3.8 or above
- Package installation
```shell
pip install -r requirements.txt
```
### Start service
#### Start demo
1. Download the embedding model bin file from [Hugging Face](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main). Place it in the `model/text2vec-base-chinese` folder.
2. Start the backend service:
```shell
cd CodeFuse-ModelCache
```
```shell
python flask4modelcache_demo.py
```
#### Service Startup With Docker-compose
1. Download the embedding model bin file from [Hugging Face](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main). Place it in the `model/text2vec-base-chinese` folder.
2. Configure docker network, only need to execute once
```shell
cd CodeFuse-ModelCache
```
```shell
docker network create modelcache
```
3. Execute the docker-compose command
```shell
# When the modelcache image does not exist locally for the first time, or when the Dockerfile is changed
docker-compose up --build
# This is not the first run and the Dockerfile has not changed
docker-compose up
```
#### Start normal service
Before you start standard service, do these steps:
1. Install MySQL and import the SQL file from `reference_doc/create_table.sql`.
2. Install vector database Milvus.
3. Configure database access in:
- `modelcache/config/milvus_config.ini`
- `modelcache/config/mysql_config.ini`
4. Download the embedding model bin file from [Hugging Face](https://huggingface.co/shibing624/text2vec-base-chinese/tree/main). Put it in `model/text2vec-base-chinese`.
5. Start the backend service:
```bash
python flask4modelcache.py
```
## Visit the service
The service provides three core RESTful API functionalities: Cache-Writing, Cache-Querying, and Cache-Clearing.
### Write cache
```python
import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'insert'
scope = {"model": "CODEGPT-1008"}
chat_info = [{"query": [{"role": "system", "content": "You are an AI code assistant and you must provide neutral and harmless answers to help users solve code-related problems."}, {"role": "user", "content": "δ½ ζ―θ°?"}],
"answer": "Hello, I am an intelligent assistant. How can I assist you?"}]
data = {'type': type, 'scope': scope, 'chat_info': chat_info}
headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))
```
### Query cache
```python
import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'query'
scope = {"model": "CODEGPT-1008"}
query = [{"role": "system", "content": "You are an AI code assistant and you must provide neutral and harmless answers to help users solve code-related problems."}, {"role": "user", "content": "Who are you?"}]
data = {'type': type, 'scope': scope, 'query': query}
headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))
```
### Clear cache
```python
import json
import requests
url = 'http://127.0.0.1:5000/modelcache'
type = 'remove'
scope = {"model": "CODEGPT-1008"}
remove_type = 'truncate_by_model'
data = {'type': type, 'scope': scope, 'remove_type': remove_type}
headers = {"Content-Type": "application/json"}
res = requests.post(url, headers=headers, json=json.dumps(data))
```
## Function comparison
We've implemented several key updates to our repository. We've resolved network issues with Hugging Face and improved inference speed by introducing local embedding capabilities. Due to limitations in SqlAlchemy, we've redesigned our relational database interaction module for more flexible operations. We've added multi-tenancy support to ModelCache, recognizing the need for multiple users and models in LLM products. Lastly, we've made initial adjustments for better compatibility with system commands and multi-turn dialogues.
Module
Function
ModelCache
GPTCache
Basic Interface
Data query interface
β
β
Data writing interface
β
β
Embedding
Embedding model configuration
β
β
Large model embedding layer
β
BERT model long text processing
β
Large model invocation
Decoupling from large models
β
Local loading of embedding model
β
Data isolation
Model data isolation
β
β
Hyperparameter isolation
Databases
MySQL
β
β
Milvus
β
β
OceanBase
β
Session management
Single-turn dialogue
β
β
System commands
β
Multi-turn dialogue
β
Data management
Data persistence
β
β
One-click cache clearance
β
Tenant management
Support for multi-tenancy
β
Milvus multi-collection capability
β
Other
Long-short dialogue distinction
β
## Features
In ModelCache, we incorporated the core principles of GPTCache. ModelCache has four modules: adapter, embedding, similarity, and data_manager.
- The adapter module orchestrates the business logic for various tasks, integrate the embedding, similarity, and data_manager modules.
- The embedding module converts text into semantic vector representations, and transforms user queries into vectors.
- The rank module ranks and evaluate the similarity of recalled vectors.
- The data_manager module manages the databases.
To make ModelCache more suitable for industrial use, we made several improvements to its architecture and functionality:
- [x] Architectural adjustment (lightweight integration):
- Embedded into LLM products using a Redis-like caching mode
- Provided semantic caching without interfering with LLM calls, security audits, and other functions
- Compatible with all LLM services
- [x] Multiple model loading:
- Supported local embedding model loading, and resolved Hugging Face network connectivity issues
- Supported loading embedding layers from various pre-trained models
- [x] Data isolation
- Environment isolation: Read different database configurations based on the environment. Isolate development, staging, and production environments.
- Multi-tenant data isolation: Dynamically create collections based on models for data isolation, addressing data separation issues in multi-model/service scenarios within large language model products
- [x] Supported system instruction: Adopted a concatenation approach to resolve issues with system instructions in the prompt paradigm.
- [x] Long and short text differentiation: Long texts bring more challenges for similarity assessment. Added differentiation between long and short texts, allowing for separate threshold configurations.
- [x] Milvus performance optimization: Adjusted Milvus consistency level to "Session" level for better performance.
- [x] Data management:
- One-click cache clearing to enable easy data management after model upgrades.
- Recall of hit queries for subsequent data analysis and model iteration reference.
- Asynchronous log write-back for data analysis and statistics
- Added model field and data statistics field to enhance features
## Todo List
### Adapter
- [ ] Register adapter for MilvusοΌBased on the "model" parameter in the scope, initialize the corresponding Collection and perform the load operation.
### Embedding model&inference
- [ ] Inference Optimization: Optimizing the speed of embedding inference, compatible with inference engines such as FasterTransformer, TurboTransformers, and ByteTransformer.
- [ ] Compatibility with Hugging Face models and ModelScope models, offering more methods for model loading.
### Scalar Storage
- [ ] Support MongoDB
- [ ] Support ElasticSearch
### Vector Storage
- [ ] Adapts Faiss storage in multimodal scenarios.
### Ranking
- [ ] Add ranking model to refine the order of data after embedding recall.
### Service
- [ ] Supports FastAPI.
- [ ] Add visual interface to offer a more direct user experience.
## Acknowledgements
This project has referenced the following open-source projects. We would like to express our gratitude to the projects and their developers for their contributions and research.
[GPTCache](https://github.com/zilliztech/GPTCache)
## Contributing
ModelCache is a captivating and invaluable project, whether you are an experienced developer or a novice just starting out, your contributions to this project are warmly welcomed. Your involvement in this project, be it through raising issues, providing suggestions, writing code, or documenting and creating examples, will enhance the project's quality and make a significant contribution to the open-source community.