{"id":13461296,"url":"https://github.com/pingcap/tiflash","last_synced_at":"2026-01-16T15:56:30.639Z","repository":{"id":36953825,"uuid":"167499157","full_name":"pingcap/tiflash","owner":"pingcap","description":"The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial","archived":false,"fork":false,"pushed_at":"2025-12-25T07:09:35.000Z","size":157788,"stargazers_count":1002,"open_issues_count":643,"forks_count":412,"subscribers_count":51,"default_branch":"master","last_synced_at":"2025-12-26T15:32:59.081Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://docs.pingcap.com/tidb/stable/tiflash-overview","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/pingcap.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":"NOTICE","maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2019-01-25T06:48:07.000Z","updated_at":"2025-12-25T10:10:23.000Z","dependencies_parsed_at":"2024-02-04T10:42:08.864Z","dependency_job_id":"bb290c6a-f22f-481b-9ebb-401b8b149682","html_url":"https://github.com/pingcap/tiflash","commit_stats":{"total_commits":16543,"total_committers":277,"mean_commits":59.72202166064982,"dds":0.5838723327086985,"last_synced_commit":"f41da70fef02f98450317265ee443a649c40922f"},"previous_names":["pingcap/tics"],"tags_count":195,"template":false,"template_full_name":null,"purl":"pkg:github/pingcap/tiflash","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftiflash","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftiflash/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftiflash/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftiflash/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/pingcap","download_url":"https://codeload.github.com/pingcap/tiflash/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/pingcap%2Ftiflash/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28331253,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-12T00:36:25.062Z","status":"ssl_error","status_checked_at":"2026-01-12T00:36:15.229Z","response_time":60,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T11:00:32.943Z","updated_at":"2026-01-12T01:46:35.542Z","avatar_url":"https://github.com/pingcap.png","language":"C++","readme":"# TiFlash\n\n![tiflash-architecture](tiflash-architecture.png)\n\n[TiFlash](https://docs.pingcap.com/tidb/stable/tiflash-overview) is a columnar storage component of [TiDB](https://docs.pingcap.com/tidb/stable) and [TiDB Cloud](https://en.pingcap.com/tidb-cloud/), the fully-managed service of TiDB. It mainly plays the role of Analytical Processing (AP) in the Hybrid Transactional/Analytical Processing (HTAP) architecture of TiDB.\n\nTiFlash stores data in columnar format and synchronizes data updates in real-time from [TiKV](https://github.com/tikv/tikv) by Raft logs with sub-second latency. Reads in TiFlash are guaranteed transactionally consistent with Snapshot Isolation level. TiFlash utilizes Massively Parallel Processing (MPP) computing architecture to accelerate the analytical workloads.\n\nTiFlash repository is based on [ClickHouse](https://github.com/ClickHouse/ClickHouse). We appreciate the excellent work of the ClickHouse team.\n\n## Quick Start\n\n### Start with TiDB Cloud\n\nQuickly explore TiFlash with [a free trial of TiDB Cloud](https://tidbcloud.com/free-trial).\n\nSee [TiDB Cloud Quick Start Guide](https://docs.pingcap.com/tidbcloud/tidb-cloud-quickstart).\n\n### Start with TiDB\n\nSee [Quick Start with HTAP](https://docs.pingcap.com/tidb/stable/quick-start-with-htap) and [Use TiFlash](https://docs.pingcap.com/tidb/stable/use-tiflash).\n\n## Build TiFlash\n\nTiFlash can be built on the following hardware architectures:\n\n- x86-64 / amd64\n- aarch64\n\nAnd the following operating systems:\n\n- Linux\n- MacOS\n\n### 1. Prepare Prerequisites\n\nThe following packages are required:\n\n- CMake 3.23.0+\n- Clang 17.0.0+\n- Rust\n- Python 3.0+\n- Ninja-Build or GNU Make\n- Ccache (not necessary but highly recommended to reduce rebuild time)\n\nDetailed steps for each platform are listed below.\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eUbuntu / Debian\u003c/b\u003e\u003c/summary\u003e\n\n```shell\nsudo apt update\n\n# Install Rust toolchain, see https://rustup.rs for details\ncurl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none\nsource $HOME/.cargo/env\n\n# Install LLVM, see https://apt.llvm.org for details\n# Clang will be available as /usr/bin/clang++-17\nwget https://apt.llvm.org/llvm.sh\nchmod +x llvm.sh\nsudo ./llvm.sh 17 all\n\n# Install other dependencies\nsudo apt install -y cmake ninja-build zlib1g-dev libcurl4-openssl-dev ccache\n```\n\nThen, expose the LLVM17 toolchain as the default one in order to use it later in compile:\n\n```shell\nexport CC=\"/usr/bin/clang-17\"\nexport CXX=\"/usr/bin/clang++-17\"\n```\n\n**Note for Ubuntu 18.04 and Ubuntu 20.04:**\n\nThe default installed cmake may be not recent enough. You can install a newer cmake from the [Kitware APT Repository](https://apt.kitware.com):\n\n```shell\nsudo apt install -y software-properties-common lsb-release\nwget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc 2\u003e/dev/null | gpg --dearmor - | sudo tee /etc/apt/trusted.gpg.d/kitware.gpg \u003e/dev/null\nsudo apt-add-repository \"deb https://apt.kitware.com/ubuntu/ $(lsb_release -cs) main\"\nsudo apt update\nsudo apt install -y cmake\n```\n\n**If you are facing \"ld.lld: error: duplicate symbol: ssl3_cbc_digest_record\":**\n\nIt is likely because you have a pre-installed libssl3 where TiFlash prefers libssl1. TiFlash has vendored libssl1, so that you can simply remove the one in the system to make compiling work:\n\n```shell\nsudo apt remove libssl-dev\n```\n\nIf this doesn't work, please [file an issue](https://github.com/pingcap/tiflash/issues/new?assignees=\u0026labels=type%2Fquestion\u0026template=general-question.md).\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eArchlinux\u003c/b\u003e\u003c/summary\u003e\n\n```shell\n# Install Rust toolchain, see https://rustup.rs for details\ncurl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none\nsource $HOME/.cargo/env\n\n# Install compilers and dependencies\nsudo pacman -S clang lld libc++ libc++abi compiler-rt openmp lcov cmake ninja curl openssl zlib llvm ccache\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eRocky Linux 8\u003c/b\u003e\u003c/summary\u003e\n\nPlease refer to [release-linux-llvm/env/prepare-sysroot.sh](./release-linux-llvm/env/prepare-sysroot.sh)\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\u003csummary\u003e\u003cb\u003eMacOS\u003c/b\u003e\u003c/summary\u003e\n\n```shell\n# Install Rust toolchain, see https://rustup.rs for details\ncurl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain none\nsource $HOME/.cargo/env\n\n# Install compilers\nxcode-select --install\n\n# Install other dependencies\nbrew install ninja cmake openssl@1.1 ccache\n\nbrew install llvm@17\n```\n\nThen, expose the LLVM17 toolchain as the default one in order to use it later in compile:\n\n```shell\nexport PATH=\"$(brew --prefix)/opt/llvm@17/bin:$PATH\"\nexport CC=\"$(brew --prefix)/opt/llvm@17/bin/clang\"\nexport CXX=\"$(brew --prefix)/opt/llvm@17/bin/clang++\"\n```\n\n\u003c/details\u003e\n\n### 2. Checkout Source Code\n\n```shell\ngit clone https://github.com/pingcap/tiflash.git --recursive -j 20\ncd tiflash\n```\n\n### 3. Build\n\nTo build TiFlash for development:\n\n```shell\n# In the TiFlash repository root:\ncmake --workflow --preset dev\n```\n\nOr if you don't like the preset:\n\n```shell\n# In the TiFlash repository root:\nmkdir cmake-build-debug\ncd cmake-build-debug\ncmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG\nninja tiflash\n```\n\n### Build Options\n\nTiFlash has several CMake build options to tweak for development purposes. These options SHOULD NOT be changed for production usage, as they may introduce unexpected build errors and unpredictable runtime behaviors.\n\nTo tweak options, pass one or multiple `-D...=...` args when invoking CMake, for example:\n\n```shell\ncd cmake-build-debug\ncmake .. -GNinja -DCMAKE_BUILD_TYPE=DEBUG -DFOO=BAR\n                                          ^^^^^^^^^\n```\n\n- **Build Type**:\n\n  - `-DCMAKE_BUILD_TYPE=RELWITHDEBINFO`: Release build with debug info (default)\n\n  - `-DCMAKE_BUILD_TYPE=DEBUG`: Debug build\n\n  - `-DCMAKE_BUILD_TYPE=RELEASE`: Release build\n\n  Usually you may want to use different build directories for different build types, e.g. a new build directory named `cmake-build-release` for the release build, so that compile unit cache will not be invalidated when you switch between different build types.\n\n- **Build with Unit Tests**:\n\n  - `-DENABLE_TESTS=ON`: Enable unit tests (enabled by default in debug profile)\n\n  - `-DENABLE_TESTS=OFF`: Disable unit tests (default in release profile)\n\n- **Build using GNU Make instead of ninja-build**:\n\n  \u003cdetails\u003e\n  \u003csummary\u003eClick to expand instructions\u003c/summary\u003e\n\n  To use GNU Make, simply don't pass `-GNinja` to cmake:\n\n  ```shell\n  cd cmake-build-debug\n  cmake .. -DCMAKE_BUILD_TYPE=DEBUG\n  make tiflash -j\n  ```\n\n  \u003e **NOTE**: Option `-j` (defaults to your system CPU core count, otherwise you can optionally specify a number) is used to control the build parallelism. Higher parallelism consumes more memory. If you encounter compiler OOM or hang, try to lower the parallelism by specifying a reasonable number, e.g., half of your system CPU core count or even smaller, after `-j`, depending on the available memory in your system.\n\n  \u003c/details\u003e\n\n- **Build with System Libraries**:\n\n  \u003cdetails\u003e\n  \u003csummary\u003eClick to expand instructions\u003c/summary\u003e\n\n  For local development, it is sometimes handy to use pre-installed third-party libraries in the system, rather than to compile them from sources of the bundled (internal) submodules.\n\n  Options are supplied to control whether to use internal third-party libraries (bundled in TiFlash) or to try using the pre-installed system ones.\n\n  \u003e **WARNING**: It is NOT guaranteed that TiFlash would still build if any of the system libraries are used.\n  \u003e Build errors are very likely to happen, almost all the time.\n\n  You can view these options along with their descriptions by running:\n\n  ```shell\n  cd cmake-build-debug\n  cmake -LH | grep \"USE_INTERNAL\" -A3\n  ```\n\n  All of these options are default as `ON`, as the names tell, using the internal libraries and build from sources.\n\n  There is another option to append extra paths for CMake to find system libraries:\n\n  - `PREBUILT_LIBS_ROOT`: Default as empty, can be specified with multiple values, separated by `;`\n\n  \u003c/details\u003e\n\n- **Build for AMD64 Architecture**:\n\n  \u003cdetails\u003e\n  \u003csummary\u003eClick to expand instructions\u003c/summary\u003e\n\n  To deploy TiFlash under the Linux AMD64 architecture, the CPU must support the `AVX2` instruction set. Ensure that `cat /proc/cpuinfo | grep avx2` has output.\n\n  If need to build TiFlash for AMD64 architecture without such instruction set, please use cmake option `-DNO_AVX_OR_HIGHER=ON`.\n\n  \u003c/details\u003e\n\n## Run Unit Tests\n\nUnit tests are automatically enabled in debug profile. To build these unit tests:\n\n```shell\n# In the TiFlash repository root:\ncmake --workflow --preset unit-tests-all\n```\n\nThen, to run these unit tests:\n\n```shell\ncmake-build-debug/dbms/gtests_dbms\ncmake-build-debug/libs/libdaemon/src/tests/gtests_libdaemon\ncmake-build-debug/libs/libcommon/src/tests/gtests_libcommon\n```\n\nUnit tests take time, because tests are run one by one. You can use our parallel test runner instead:\n\n```shell\npython3 tests/gtest_10x.py \\\n  cmake-build-debug/dbms/gtests_dbms \\\n  cmake-build-debug/libs/libdaemon/src/tests/gtests_libdaemon \\\n  cmake-build-debug/libs/libcommon/src/tests/gtests_libcommon\n```\n\nMore usages are available via `./dbms/gtests_dbms --help`.\n\n## Run Sanitizer Tests\n\nTiFlash supports testing with thread sanitizer and address sanitizer.\n\nTo build unit test executables with sanitizer enabled:\n\n```shell\n# In the TiFlash repository root:\ncmake --workflow --preset asan-tests-all # or tsan-tests-all\n```\n\nThere are known false positives reported from leak sanitizer (which is included in address sanitizer). To suppress these errors, set the following environment variables before running the executables:\n\n```shell\nLSAN_OPTIONS=\"suppressions=../tests/sanitize/asan.suppression\" ./dbms/gtests_dbms ...\n# or\nTSAN_OPTIONS=\"suppressions=../tests/sanitize/tsan.suppression\" ./dbms/gtests_dbms ...\n```\n\n## Run Integration Tests\n\nCheck out the [Integration Test Guide](/tests/README.md) for more details.\n\n## Run MicroBenchmark Tests\n\nTo build micro benchmark tests, you need release profile and tests enabled:\n\n```shell\n# In the TiFlash repository root:\ncmake --workflow --preset benchmarks\n```\n\nThen, to run these micro benchmarks:\n\n```shell\ncd cmake-build-release\n./dbms/bench_dbms\n\n# Or run with filter:\n# ./dbms/bench_dbms --benchmark_filter=xxx\n```\n\nMore usages are available via `./dbms/bench_dbms --help`.\n\n## Generate LLVM Coverage Report\n\nTo build coverage report, run the script under `release-linux-llvm`\n\n```shell\ncd release-linux-llvm\n./gen_coverage.sh\n# Or run with filter:\n# FILTER='*DMFile*:*DeltaMerge*:*Segment*' ./gen_coverage.sh\n\n# After the script finished, it will output the directory of code coverage report, you can check out the files by webbrowser\npython3 -m http.server --directory \"${REPORT_DIR}\" \"${REPORT_HTTP_PORT}\"\n```\n\n## Contributing\n\nHere is the overview of TiFlash architecture [The architecture of TiFlash's distributed storage engine and transaction layer](/docs/design/0000-00-00-architecture-of-distributed-storage-and-transaction.md).\n\nSee [TiFlash Development Guide](/docs/DEVELOPMENT.md) and [TiFlash Design documents](/docs/design).\n\nBefore submitting a pull request, please resolve clang-tidy errors and use [format-diff.py](format-diff.py) to format source code, otherwise CI build may raise error.\n\n\u003e **NOTE**: It is required to use clang-format 17.0.0+.\n\n```shell\n# In the TiFlash repository root:\nmerge_base=$(git merge-base upstream/master HEAD)\npython3 release-linux-llvm/scripts/run-clang-tidy.py -p cmake-build-debug -j 20 --files `git diff $merge_base --name-only`\n# if there are too much errors, you can try to run the script again with `-fix`\npython3 format-diff.py --diff_from $merge_base\n```\n\n## License\n\nTiFlash is under the Apache 2.0 license. See the [LICENSE](./LICENSE) file for details.\n","funding_links":[],"categories":["C++"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpingcap%2Ftiflash","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpingcap%2Ftiflash","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpingcap%2Ftiflash/lists"}