https://github.com/jd-opensource/dtc

High performance Distributed Table Cache system designed by JD.com, Offering hotspot data cache from databases in order to reduce pressure of database and improve QPS.
https://github.com/jd-opensource/dtc

cache cache-storage distributed-storage distributed-systems storage

Last synced: 7 months ago
JSON representation

High performance Distributed Table Cache system designed by JD.com, Offering hotspot data cache from databases in order to reduce pressure of database and improve QPS.

• Host: GitHub
• URL: https://github.com/jd-opensource/dtc
• Owner: jd-opensource
• License: apache-2.0
• Created: 2021-08-22T13:52:39.000Z (almost 4 years ago)
• Default Branch: master
• Last Pushed: 2024-09-10T09:45:27.000Z (10 months ago)
• Last Synced: 2024-12-10T14:22:32.969Z (7 months ago)
• Topics: cache, cache-storage, distributed-storage, distributed-systems, storage
• Language: C++
• Homepage:
• Size: 26.6 MB
• Stars: 31
• Watchers: 4
• Forks: 19
• Open Issues: 19
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
![DTC](http://storage.360buyimg.com/bq-install/release/dtc_logo.png)

# DTC - Distributed Table Cache

[![ubuntu&gcc-4.9](https://github.com/jd-opensource/dtc/actions/workflows/ubuntu-20.04&gcc-4.9.yml/badge.svg)](https://github.com/jd-opensource/dtc/actions/workflows/ubuntu-20.04&gcc-4.9.yml)

## Overview

DTC is a high performance Distributed Table Cache system designed by JD.com that offering hotspot data cache for databases in order to reduce pressure of database and improve QPS.

![](http://storage.360buyimg.com/bq-install/release/architecture.png)

The DTC system consists of the following components:

* **Agent** - Provides key consistent hash routing in order to reduce connections and improve performance.

* **Dtcd** - Provides hot data caching service.

* **Connector** - Provides connection and communication between cache and persistent storage database such as MYSQL.

## Feature

* Database Protection

  - protection for null node, prevent cache breakdown.

  - provide long-term data caching, and prevent cache penetration.

  - data source thread available, protect the database with a limited number of connections.

  - Estimated timeout policy to reduce invalid database requests.

* Data consistency

  - write-through policy, ensure cache and database data consistent.

  - barrier policy to prevent update requests lost while concurrcy.

* Performance

  - integrated memroy allocation policy to avoid frequent system calls.

  - I/O multiplexing to handle concurrcy requests.

  - multiple data structure models to improve memory performance.

* Scalability

  - cache node expands horizontally to enhance cache capacity.

  - cache node expands vertically, supports slave reading, and solve the bottleneck of hot keys.

  - provide sharding, supports for persistent storage scalable.

## Performance

* DTC can process 90,000 QPS of query requests at single-core cpu & single dtc instance.

* DTC can provide above 3,000,000 QPS query capability with above 99.9% hit rate and less than 200 μs response time in actual distributed scenarios.

* Layered Storage is able to provide about 1,000 QPS write capability with above 99.9% per single instance.

## How to Build

DTC provides docker images for quick start:

* Start server docker:


```shell

docker pull dtc8/server:latest

docker run -i -t --name dtc-server -p 127.0.0.1:20015:20015 dtc8/server:latest

```

Depending on 3rd-party sql parsing engine [hsql](https://github.com/DTC8/sql-parser) .

For more compile information, click [Building](docs/building.md).


Trying a demo, visit [QuickStart](docs/quickstart.md).

## License

JD.com © Copyright 2021-2024 [JD.com](https://ir.jd.com/), Inc.


[Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0). Visit [LICENSE](./LICENSE) for more details.