Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/jd-opensource/dtc

High performance Distributed Table Cache system designed by JD.com, Offering hotspot data cache from databases in order to reduce pressure of database and improve QPS.
https://github.com/jd-opensource/dtc

cache cache-storage distributed-storage distributed-systems storage

Last synced: 5 days ago
JSON representation

High performance Distributed Table Cache system designed by JD.com, Offering hotspot data cache from databases in order to reduce pressure of database and improve QPS.

Host: GitHub
URL: https://github.com/jd-opensource/dtc
Owner: jd-opensource
License: apache-2.0
Created: 2021-08-22T13:52:39.000Z (about 3 years ago)
Default Branch: master
Last Pushed: 2024-09-10T09:45:27.000Z (2 months ago)
Last Synced: 2024-09-10T11:10:24.899Z (2 months ago)
Topics: cache, cache-storage, distributed-storage, distributed-systems, storage
Language: C++
Homepage:
Size: 26.6 MB
Stars: 30
Watchers: 4
Forks: 19
Open Issues: 19
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

![DTC](http://storage.360buyimg.com/bq-install/release/dtc_logo.png)
# DTC - Distributed Table Cache
[![ubuntu&gcc-4.9](https://github.com/jd-opensource/dtc/actions/workflows/ubuntu-20.04&gcc-4.9.yml/badge.svg)](https://github.com/jd-opensource/dtc/actions/workflows/ubuntu-20.04&gcc-4.9.yml)
## Overview
DTC is a high performance Distributed Table Cache system designed by JD.com that offering hotspot data cache for databases in order to reduce pressure of database and improve QPS.

![](http://storage.360buyimg.com/bq-install/release/architecture.png)

The DTC system consists of the following components:
* **Agent** - Provides key consistent hash routing in order to reduce connections and improve performance.
* **Dtcd** - Provides hot data caching service.
* **Connector** - Provides connection and communication between cache and persistent storage database such as MYSQL.

## Feature
* Database Protection
- protection for null node, prevent cache breakdown.
- provide long-term data caching, and prevent cache penetration.
- data source thread available, protect the database with a limited number of connections.
- Estimated timeout policy to reduce invalid database requests.
* Data consistency
- write-through policy, ensure cache and database data consistent.
- barrier policy to prevent update requests lost while concurrcy.
* Performance
- integrated memroy allocation policy to avoid frequent system calls.
- I/O multiplexing to handle concurrcy requests.
- multiple data structure models to improve memory performance.
* Scalability
- cache node expands horizontally to enhance cache capacity.
- cache node expands vertically, supports slave reading, and solve the bottleneck of hot keys.
- provide sharding, supports for persistent storage scalable.
## Performance
* DTC can process 90,000 QPS of query requests at single-core cpu & single dtc instance.
* DTC can provide above 3,000,000 QPS query capability with above 99.9% hit rate and less than 200 μs response time in actual distributed scenarios.
* Layered Storage is able to provide about 1,000 QPS write capability with above 99.9% per single instance.
## How to Build
DTC provides docker images for quick start:
* Start server docker:

```shell
docker pull dtc8/server:latest
docker run -i -t --name dtc-server -p 127.0.0.1:20015:20015 dtc8/server:latest
```
Depending on 3rd-party sql parsing engine [hsql](https://github.com/DTC8/sql-parser) .
For more compile information, click [Building](docs/building.md).

Trying a demo, visit [QuickStart](docs/quickstart.md).