https://github.com/tidesdb/tidesql
A space and write optimized pluggable MariaDB engine built on TidesDB
https://github.com/tidesdb/tidesql
btree cloud-native lsmtree mariadb mvcc object-storage olap oltp optimistic pessimistic-concurrency sql storage-engine transactional
Last synced: 8 days ago
JSON representation
A space and write optimized pluggable MariaDB engine built on TidesDB
- Host: GitHub
- URL: https://github.com/tidesdb/tidesql
- Owner: tidesdb
- License: other
- Created: 2026-01-24T17:13:20.000Z (5 months ago)
- Default Branch: master
- Last Pushed: 2026-06-02T06:36:47.000Z (12 days ago)
- Last Synced: 2026-06-02T07:25:05.346Z (12 days ago)
- Topics: btree, cloud-native, lsmtree, mariadb, mvcc, object-storage, olap, oltp, optimistic, pessimistic-concurrency, sql, storage-engine, transactional
- Language: C++
- Homepage: https://tidesdb.com/reference/tidesql/
- Size: 56.2 MB
- Stars: 18
- Watchers: 1
- Forks: 3
- Open Issues: 6
-
Metadata Files:
- Readme: README
- Contributing: CONTRIBUTING
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT
- Security: SECURITY
- Cla: CLA
Awesome Lists containing this project
README
TIDESQL
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
A pluggable write and space optimized storage engine for MariaDB using TidesDB.
To find compatibility table with MariaDB go to https://tidesdb.com/reference/tidesql/#mariadb-compatibility
INSTALLATION
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
You can choose to run install.sh, for instructions run ./install.sh --help
To install in a specific location(e.g /data being NVMe disk)
./install.sh --mariadb-prefix /data/mariadb --tidesdb-prefix /data/tidesdb --build-dir /data/tidesql-build
The install script will build TidesDB and MariaDB from source.
You can specify the versions if you want. The script will install everything
and make TidesDB available as an engine to utilize.
To skip storage engines you don't need (saves build time and reduces warnings):
./install.sh --list-engines # see what can be skipped
./install.sh --skip-engines mroonga,rocksdb,connect,spider,oqgraph,columnstore
To use specific MariaDB version you use:
--mariadb-version mariadb-12.3.1
mariadb-12.3.1 aligning to https://github.com/MariaDB/server/releases/tag/mariadb-12.3.1
When using install script you are installing the latest version of TideSQL at the time of clone or download.
To link libtidesdb against a non-default allocator (jemalloc is a common
pick for write-heavy workloads):
./install.sh --allocator jemalloc
./install.sh --allocator mimalloc
./install.sh --allocator tcmalloc
The flag only affects libtidesdb.so's internal allocations; mariadbd's
allocator is unchanged. For a process-wide swap also LD_PRELOAD the
allocator at mariadbd startup:
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 \
/usr/local/mariadb/bin/mariadbd-safe \
--defaults-file=/usr/local/mariadb/my.cnf &
Note that --rebuild-plugin does not rebuild libtidesdb, so changing the
allocator requires a full install run (omit --rebuild-plugin) to take
effect. Verify the linkage with:
ldd /usr/local/lib/libtidesdb.so | grep -E 'jemalloc|mimalloc|tcmalloc'
Skippable engines:
archive Archive storage engine (read-only row-format tables)
blackhole Blackhole engine (accepts writes, stores nothing)
columnstore MariaDB ColumnStore (columnar analytics engine)
connect CONNECT engine (access external data sources)
example Example storage engine (test/demo only)
federated Legacy Federated engine (MODULE_ONLY)
federatedx FederatedX engine (query remote MySQL/MariaDB tables)
mroonga Mroonga full-text search engine (requires Groonga)
oqgraph Open Query GRAPH engine (graph computation)
rocksdb MyRocks / RocksDB LSM-tree engine
sequence Sequence engine (virtual auto-increment sequences)
sphinx SphinxSE engine (Sphinx full-text search integration)
spider Spider engine (sharding / federation)
InnoDB, Aria, MyISAM, and CSV cannot be skipped (server depends on them).
Below are manual install instructions for those who want to configure everything
themselves, though with the install script you can modify configuration files
after the fact.
LINUX (Ubuntu/Debian)
░░░░░░░░░░░░░░░░░░░░░░░░
1. Install dependencies:
sudo apt update
sudo apt install -y build-essential cmake ninja-build bison flex \
libzstd-dev liblz4-dev libsnappy-dev libncurses-dev libssl-dev \
libxml2-dev libevent-dev libcurl4-openssl-dev pkg-config
2. Install TidesDB library:
git clone --depth 1 https://github.com/tidesdb/tidesdb.git tidesdb-lib
cd tidesdb-lib && mkdir build && cd build
cmake ..
make -j$(nproc)
sudo make install
sudo ldconfig
3. Clone MariaDB and copy TidesDB storage engine:
git clone --depth 1 --branch mariadb-11.8.6 https://github.com/MariaDB/server.git mariadb-server
cd mariadb-server
git submodule update --init --recursive
cp -r /path/to/tidesql/tidesdb storage/
cp -r /path/to/tidesql/mysql-test/suite/tidesdb mysql-test/suite/
4. Build MariaDB:
mkdir build && cd build
cmake ..
make -j$(nproc)
5. Run tests (from the build directory):
cd mysql-test
perl mtr --suite=tidesdb --parallel=4 --force
MACOS
░░░░░░░░░░░░░░░░░░░░░░░░
1. Install dependencies:
brew install cmake ninja bison snappy lz4 zstd openssl@3
2. Install TidesDB library:
# Ensure Xcode SDK is used (not CommandLineTools)
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
export SDKROOT=$(xcrun --show-sdk-path)
git clone --depth 1 https://github.com/tidesdb/tidesdb.git tidesdb-lib
cd tidesdb-lib && mkdir build && cd build
cmake .. -DCMAKE_OSX_SYSROOT=${SDKROOT} -DOPENSSL_ROOT_DIR=$(brew --prefix openssl@3)
make -j$(sysctl -n hw.ncpu)
sudo make install
3. Clone MariaDB and copy TidesDB storage engine:
git clone --depth 1 --branch mariadb-11.8.6 https://github.com/MariaDB/server.git mariadb-server
cd mariadb-server
git submodule update --init --recursive
cp -r /path/to/tidesql/tidesdb storage/
cp -r /path/to/tidesql/mysql-test/suite/tidesdb mysql-test/suite/
4. Build MariaDB:
# Remove CommandLineTools SDK to prevent header conflicts
sudo rm -rf /Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk
export SDKROOT=$(xcrun --show-sdk-path)
export CC=$(xcrun -find clang)
export CXX=$(xcrun -find clang++)
mkdir build && cd build
cmake .. \
-DCMAKE_C_COMPILER=${CC} \
-DCMAKE_CXX_COMPILER=${CXX} \
-DCMAKE_OSX_SYSROOT=${SDKROOT} \
-DCMAKE_C_FLAGS="-Wno-nullability-completeness" \
-DCMAKE_CXX_FLAGS="-Wno-nullability-completeness" \
-DWITH_SSL=bundled \
-DWITH_PCRE=bundled \
-G Ninja
cmake --build . --parallel $(sysctl -n hw.ncpu)
Note -- The CommandLineTools SDK removal is needed because CMake may find
headers in /Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk which
causes C/C++ header path conflicts with libc++.
5. Run tests (from the build directory):
cd mysql-test
perl mtr --suite=tidesdb --parallel=4 --force
WINDOWS
░░░░░░░░░░░░░░░░░░░░░░░░
1. Install prerequisites:
- Visual Studio 2022 with C++ workload
- CMake (choco install cmake)
- Strawberry Perl (choco install strawberryperl)
- WinFlexBison (download from GitHub releases)
2. Install vcpkg dependencies:
cd C:\vcpkg
git pull
.\vcpkg.exe install zstd:x64-windows lz4:x64-windows snappy:x64-windows pthreads:x64-windows
3. Install TidesDB library:
git clone --depth 1 https://github.com/tidesdb/tidesdb.git tidesdb-lib
cd tidesdb-lib
mkdir build && cd build
cmake .. -G "Visual Studio 17 2022" -A x64 ^
-DCMAKE_TOOLCHAIN_FILE=C:/vcpkg/scripts/buildsystems/vcpkg.cmake ^
-DCMAKE_INSTALL_PREFIX=C:/tidesdb
cmake --build . --config Release
cmake --install . --config Release
4. Clone MariaDB and copy TidesDB storage engine:
git clone --depth 1 --branch mariadb-11.8.6 https://github.com/MariaDB/server.git mariadb-server
cd mariadb-server
git submodule update --init --recursive
xcopy /E /I path\to\tidesql\tidesdb storage\tidesdb
xcopy /E /I path\to\tidesql\mysql-test\suite\tidesdb mysql-test\suite\tidesdb
5. Build MariaDB:
mkdir build && cd build
cmake .. -G "Visual Studio 17 2022" -A x64 ^
-DCMAKE_PREFIX_PATH="C:/tidesdb;C:/vcpkg/installed/x64-windows" ^
-DCONNECT_DYNAMIC=NO
cmake --build . --config RelWithDebInfo --parallel
Note -- CONNECT plugin is disabled (-DCONNECT_DYNAMIC=NO) because it
requires libxml2 headers that may conflict with vcpkg installations.
6. Run tests:
cd mysql-test
perl mtr --suite=tidesdb --parallel=4 --force
ENABLE PLUGIN
░░░░░░░░░░░░░░░░░░░░░░░░
After building, enable the plugin in MariaDB:
INSTALL SONAME 'ha_tidesdb'; -- works on all platforms
FEATURES
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Core:
- MVCC transactions with per-table isolation (autocommit uses READ_COMMITTED;
multi-statement transactions use the table's configured level)
- SQL savepoints (SAVEPOINT / ROLLBACK TO / RELEASE SAVEPOINT)
- START TRANSACTION WITH CONSISTENT SNAPSHOT
- Lock-free concurrency (no THR_LOCK, TidesDB handles isolation internally)
- Pessimistic row locking (tidesdb_pessimistic_locking, ON by default).
Two-mode lock manager with shared (S) and exclusive (X) locks over a
cache-line aligned hash-partitioned table sized at init from CPU
count (8 * hardware_concurrency, clamped to 128..65536 partitions).
Multiple S holders coexist on the same row, X conflicts with any
other holder, and new S requests block while an X is queued so
writers are never starved. Writes acquire X on the actual mutation
target inside write_row, update_row and delete_row; range scans
under UPDATE/DELETE do not lock rows that ICP filters away. SELECT
FOR UPDATE keeps the per-row lock on every cursor-visible row per
the SQL contract. Plain reads acquire S under REPEATABLE-READ or
SERIALIZABLE, no lock under READ-COMMITTED or SNAPSHOT (MVCC
snapshot suffices). DFS wait-for-graph deadlock detection bounded
by DEADLOCK_MAX_DEPTH walks every conflicting holder per hop;
lock-wait timeout configurable via tidesdb_lock_wait_timeout_ms
(default 50 s, mirrors innodb_lock_wait_timeout). Lock entries
recycle in place once empty so per-partition memory is bounded by
peak concurrent locks. Status counters tidesdb_lock_waits,
tidesdb_lock_wait_us, tidesdb_lock_deadlocks, tidesdb_lock_timeouts,
tidesdb_lock_held, tidesdb_lock_entries, tidesdb_lock_entry_recycles
surface contention for observability
- LSM-tree storage with optional B+tree SSTable format
- Compression (NONE, LZ4, LZ4_FAST, ZSTD, Snappy)
- Bloom filters for fast key lookups
- Block cache for frequently accessed data
- Primary key (single and composite) and secondary index support
- Index Condition Pushdown (ICP) for secondary index scans
- Multi-Range Read (MRR) with sorted point-lookup batching for queries of
the shape WHERE col IN (v1, v2, ..., vN). Accepted only when every
range is a full-key point equality and there are 2+ ranges; keys are
sorted by comparable bytes so the LSM sees a monotone stream of seeks.
- Bulk UPDATE / DELETE batching -- multi-row UPDATE and DELETE statements
share the same mid-txn commit plumbing as bulk INSERT, keeping long
statements' txn memory bounded regardless of statement size
- Single-delete with pair cancellation at compaction -- every secondary
index delete (regular, FTS, spatial) uses tidesdb_txn_single_delete
unconditionally, safe by construction because each (col_values, pk)
composite is put once and deleted once per row lifetime. Primary CF
single-delete is opt-in via the tidesdb_single_delete_primary session
variable (default OFF) for sessions that only INSERT and DELETE
- Tombstone-density compaction trigger -- per-table TOMBSTONE_DENSITY_TRIGGER
and TOMBSTONE_DENSITY_MIN_ENTRIES options arm a post-flush check that
escalates compaction for any single SSTable whose tombstone density
crosses the configured ratio. Aggregates surface as
tidesdb_total_tombstones, tidesdb_tombstone_ratio, and
tidesdb_max_sst_tombstone_density status vars
- Auto compact-after-range-delete -- session variable
tidesdb_compact_after_range_delete_min_rows (default 0) lets a multi-row
DELETE that touches at least N rows synchronously call
tidesdb_compact_range over the touched primary-key range at end of
statement, physically reclaiming tombstoned space without waiting for
a structural trigger
- REPLACE INTO and INSERT ... ON DUPLICATE KEY UPDATE
- AUTO_INCREMENT with O(1) atomic counter (no iterator per INSERT).
TRUNCATE TABLE and ALTER TABLE ... AUTO_INCREMENT=N both correctly
reset the counter via the reset_auto_increment handler hook.
- TTL (time-to-live) per-row and per-table expiration
- Virtual/generated columns
- Online backup (SET GLOBAL tidesdb_backup_dir = '/path')
- Hard-link checkpoint (SET GLOBAL tidesdb_checkpoint_dir = '/path')
- OPTIMIZE TABLE (synchronous purge + compact via tidesdb_purge_cf)
- CHECK TABLE (verifies SSTable metadata and block integrity)
- REPAIR TABLE (full purge + compaction, rebuilds LSM tree)
- FLUSH TABLES FOR EXPORT (HA_CAN_EXPORT, consistent file copy)
- SHOW ENGINE TIDESDB STATUS (DB stats, memory, cache, conflict info,
unified-memtable block with active bytes / immutable count / WAL
generation / next-CF index, and a write-amplification block with
cumulative user/WAL/flush/compaction bytes plus a derived total WA
ratio. Object-store builds also report the last successfully
uploaded unified-WAL generation.)
- SHOW GLOBAL STATUS LIKE 'tidesdb%' monitoring variables for tools like
Prometheus, PMM, Datadog -- column-family and memtable bytes, cache hit
rate, tombstone density aggregates, lock-manager observability
(tidesdb_lock_waits, tidesdb_lock_wait_us, tidesdb_lock_deadlocks,
tidesdb_lock_timeouts, tidesdb_lock_held, tidesdb_lock_entries,
tidesdb_lock_entry_recycles), back-pressure wait counters
(tidesdb_backpressure_waits, tidesdb_backpressure_wait_us), unified
memtable state (tidesdb_unified_memtable_enabled,
tidesdb_unified_memtable_bytes, tidesdb_unified_immutable_count,
tidesdb_unified_is_flushing, tidesdb_unified_wal_generation),
object-store progress (tidesdb_object_store_enabled,
tidesdb_replica_mode_active, tidesdb_local_cache_bytes,
tidesdb_local_cache_files, tidesdb_upload_queue_depth,
tidesdb_total_uploads, tidesdb_upload_failures,
tidesdb_last_uploaded_generation), and write-amplification counters
(tidesdb_uwal_bytes_written, tidesdb_wal_bytes_written,
tidesdb_flush_bytes_written, tidesdb_compaction_bytes_written,
tidesdb_compaction_bytes_read, tidesdb_user_bytes_written,
tidesdb_flush_count, tidesdb_compaction_count)
- Handlerton lifecycle + administrative hooks wired:
drop_database DROP DATABASE removes all TidesDB CFs for the db
flush_logs FLUSH LOGS syncs the TidesDB WAL (safe before backup)
panic orderly close on signal-driven shutdown
pre_shutdown background thread quiesce before deinit
kill_query KILL QUERY wakes waiters in row_lock_acquire and is
checked inside rnd_next / index_next / spatial /
ft_read so long scans return HA_ERR_ABORTED_BY_USER
- Partitioning (RANGE, LIST, HASH, KEY)
- Data-at-rest encryption (MariaDB encryption plugin integration; fail-closed
on missing key so a rotated-out version cannot silently mis-encrypt or
return zeroed plaintext)
- Online DDL (instant metadata, inplace add/drop secondary index, force-copy
for FULLTEXT/SPATIAL adds so row back-fill goes through write_row, copy
for column-type changes and PK rewrites)
- TRUNCATE TABLE as O(1) drop+recreate (instant regardless of table size)
- ANALYZE TABLE with detailed CF statistics (levels, keys, sizes, cache
hit rate, plus a per-CF write-amplification note that reports user
bytes ingested versus WAL / flush / compaction bytes and the derived
ratio, so a single ANALYZE makes the WA of an individual table
legible without having to compute it from global counters)
- Cached optimizer statistics (refreshed every 2 seconds from TidesDB)
- Full-text search (FULLTEXT indexes with BM25 ranking, natural language
and boolean mode, charset-aware tokenizer, configurable BM25 parameters,
stop words matching InnoDB defaults, blend_chars for Romance language
elision support, extended FT API via HA_CAN_FULLTEXT_EXT)
- Vector search (VECTOR indexes via MariaDB's MHNSW approximate nearest
neighbor, Euclidean and cosine distance, INSERT/UPDATE/DELETE support)
- Spatial indexes (SPATIAL KEY with Hilbert curve encoding on LSM,
MBRIntersects/MBRContains/MBRWithin/MBREquals/MBRDisjoint predicates)
Cloud / Object Store:
- S3-compatible object store backend (AWS S3, MinIO, GCS)
- Four-tier hot/warm/cold/frozen storage hierarchy
- Block-level range_get for point lookups on frozen SSTables (single HTTP request)
- Parallel SSTable prefetch for iterators
- Read-only replica mode with MANIFEST sync and WAL replay
- Primary promotion via SET GLOBAL tidesdb_promote_primary = ON
- Configurable WAL sync (per-commit for RPO=0, or bytes-threshold)
- LRU local file cache with paired klog/vlog eviction
- Upload retry with verification, download retry with backoff
- Kubernetes deployment with automated failover (see k8s/ directory)
CONFIGURATION
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
TidesDB stores its data as a sibling of the MariaDB data directory:
/tidesdb_data
SYSTEM VARIABLES (SET GLOBAL tidesdb_...)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Read-only (set at startup):
flush_threads Background flush threads (default: 4)
compaction_threads Background compaction threads (default: 4)
log_level DEBUG/INFO/WARN/ERROR/FATAL/NONE (default: DEBUG)
block_cache_size Global block cache in bytes (default: 256MB)
max_open_sstables Max cached SSTable files (default: 256)
max_memory_usage Global memory limit in bytes; 0 = auto (default: 0)
data_home_dir Override TidesDB data directory (default: auto)
log_to_file Write logs to file vs stderr (default: ON)
log_truncation_at Log file truncation size (default: 24MB; 0 = off)
unified_memtable Single shared WAL+memtable across all CFs (default: ON)
unified_memtable_write_buffer_size Write buffer for unified memtable (default: 256MB)
unified_memtable_sync_mode NONE/INTERVAL/FULL for unified WAL (default: FULL)
unified_memtable_sync_interval Sync interval in µs for INTERVAL mode (default: 128000)
unified_memtable_skip_list_max_level Skip-list max level for the unified
memtable; 0 keeps the library default
unified_memtable_skip_list_probability Skip-list level promotion probability
for the unified memtable; 0.0 keeps
the library default
object_store_backend LOCAL or S3 (default: LOCAL)
s3_endpoint S3 endpoint (e.g. s3.amazonaws.com)
s3_bucket S3 bucket name
s3_prefix S3 key prefix (e.g. production/db1/)
s3_access_key S3 access key ID
s3_secret_key S3 secret access key
s3_region S3 region (NULL for MinIO)
s3_use_ssl Use HTTPS (default: ON)
s3_path_style Path-style URLs for MinIO (default: OFF)
s3_tls_ca_path Custom CA bundle path for the S3 TLS handshake;
NULL uses the system bundle
s3_tls_insecure_skip_verify Disable TLS peer/host verification (INSECURE,
test endpoints only; default: OFF)
s3_multipart_threshold Object size in bytes that triggers S3 multipart
upload; 0 keeps the library default
s3_multipart_part_size S3 multipart chunk size in bytes; 0 keeps the
library default
objstore_local_cache_max Local cache size limit (default: 0 = unlimited)
objstore_wal_sync_threshold WAL sync byte threshold (default: 1MB)
objstore_wal_sync_on_commit Upload WAL on every commit (default: OFF)
objstore_cache_on_read Cache downloaded objects locally (default: ON)
objstore_cache_on_write Cache uploaded objects locally (default: ON)
objstore_max_concurrent_uploads
Concurrent upload threads; 0 keeps the library
default
objstore_max_concurrent_downloads
Concurrent download threads; 0 keeps the library
default
objstore_multipart_threshold
Object size in bytes that triggers multipart
upload at the objstore layer; 0 keeps the library
default
objstore_multipart_part_size
Multipart chunk size in bytes at the objstore
layer; 0 keeps the library default
objstore_sync_manifest_to_object
Upload MANIFEST after each compaction (default: ON)
objstore_wal_upload_sync Block memtable flush on WAL upload
(default: OFF for background WAL upload)
objstore_replicate_wal Upload WAL segments for replica recovery
(default: ON)
objstore_replica_replay_wal Replay WAL on replicas for near-real-time
visibility (default: ON)
replica_mode Read-only replica (default: OFF)
replica_sync_interval MANIFEST poll interval in us (default: 5000000)
fast_shutdown When ON, deinit calls tidesdb_cancel_background_work
before tidesdb_close so in-flight compactions bail
at their next checkpoint and shutdown returns
quickly (default: OFF). Default OFF lets close
drain naturally, which is safer for object-store
and replica setups where a mid-compaction cancel
can leave S3 with referenced-but-not-yet-uploaded
SSTables that confuse a syncing replica.
Dynamic (SET GLOBAL at runtime):
backup_dir Set to a path to trigger online backup
checkpoint_dir Set to a path to trigger hard-link checkpoint
print_all_conflicts Log all TDB_ERR_CONFLICT events (default: OFF)
pessimistic_locking Enable plugin-level row locks for SELECT...FOR UPDATE,
UPDATE, DELETE, and INSERT (default: ON). Two-mode
S/X lock manager over a cache-line aligned
hash-partitioned table sized at init from CPU
count (8 * hardware_concurrency, clamped to
128..65536 partitions). Writes acquire X on the
actual mutation target inside write_row,
update_row and delete_row; range scans under
UPDATE/DELETE do not lock rows that ICP filters
away. Plain reads under REPEATABLE-READ or
SERIALIZABLE acquire S (multiple S holders
coexist), while READ-COMMITTED and SNAPSHOT
reads take no lock and rely on MVCC. A new S
blocks while an X is queued so writers cannot
be starved. DFS wait-for-graph deadlock
detection with bounded depth, lock-wait timeout
controlled by tidesdb_lock_wait_timeout_ms. Locks
can cover non-existing keys (SFU on a missing row
blocks INSERT of that key). OFF reverts to pure
optimistic MVCC, where conflicts surface as
HA_ERR_LOCK_DEADLOCK at COMMIT and the application
must retry.
promote_primary Set to ON to promote a replica to primary
(trigger variable, resets to OFF after promotion)
fts_min_word_len Minimum word length for FTS indexing (default: 3)
fts_max_word_len Maximum word length for FTS indexing (default: 84)
fts_bm25_k1 BM25 k1 term-frequency saturation (default: 1.2)
fts_bm25_b BM25 b document-length normalization (default: 0.75)
ft_stopword_table Custom stop word table 'db/table' (default: NULL = InnoDB defaults)
fts_blend_chars Blend characters for Romance elision (default: empty)
Session (SET SESSION tidesdb_...):
ttl Per-session TTL override in seconds (default: 0)
skip_unique_check Skip PK/unique checks on INSERT (default: OFF)
lock_wait_timeout_ms Milliseconds a pessimistic row-lock acquire
waits before returning HA_ERR_LOCK_WAIT_TIMEOUT
(default: 50000, mirrors innodb_lock_wait_timeout).
0 disables the bound and the wait is interrupted
only by KILL QUERY.
backpressure_wait_timeout_ms
Milliseconds the plugin blocks a writer on
TidesDB back-pressure (memtable, flush queue,
or L0 backlog at soft cap) before surfacing
the condition to the SQL layer as
HA_ERR_LOCK_WAIT_TIMEOUT (default: 60000).
The budget is anchored once per statement at
external_lock(F_WRLCK) and shared across every
backpressure call inside that statement, so a
multi-row INSERT respects the configured ms
end-to-end instead of multiplying it by row
count. 0 disables blocking and the
back-pressure signal returns immediately.
single_delete_primary Route primary-CF DELETEs through
tidesdb_txn_single_delete (default: OFF).
Caller promises the session does no UPDATE
on non-PK columns, no REPLACE INTO, and no
INSERT ... ON DUPLICATE KEY UPDATE on tables
without secondary indexes. Violating the
contract may re-expose older row versions
after compaction. Safe for INSERT-only +
DELETE-only workloads (e.g. log rotation,
iibench l.i1 shape).
compact_after_range_delete_min_rows
If non-zero, after a multi-row DELETE that
touches at least this many rows, the engine
calls tidesdb_compact_range over the
touched primary-key range to physically
reclaim tombstoned space synchronously
instead of waiting for a structural
compaction trigger (default: 0 = disabled).
Useful for sliding-window expiry, tenant
eviction, or any DELETE-by-PK-range op.
default_compression Default compression for new tables
default_write_buffer_size Default write buffer for new tables (64MB)
default_sync_mode Default sync mode for new tables (FULL)
default_tombstone_density_trigger
Default tombstone-density compaction
trigger ratio for new tables, in parts
per 10000 (5000 = 0.50 ratio). When set
non-zero, after each flush the engine
inspects level-1 SSTables and escalates
compaction for any single SST whose
tombstone count divided by entry count
exceeds the ratio (default: 0 = disabled).
default_tombstone_density_min_entries
Minimum entry count for an SSTable to be
considered by the tombstone-density
trigger; smaller SSTables are ignored
(default: 1024).
(and other default_* variables for all table options)
Logging -- TidesDB writes to /LOG by default with automatic
truncation at 24 MB. Set log_level to WARN or higher in production to
reduce log volume.
TABLE OPTIONS (CREATE TABLE ... ENGINE=TidesDB =)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
These are per-table options set at creation time and applied to the column
family configuration. ALTER TABLE ... = updates both the .frm
and the live column family via tidesdb_cf_update_runtime_config().
Storage:
WRITE_BUFFER_SIZE Memtable size before flush (default: 64MB PER TABLE/CF)
MIN_DISK_SPACE Minimum free disk space (default: 100MB)
KLOG_VALUE_THRESHOLD Values larger than this go to vlog (default: 512)
Compression:
COMPRESSION NONE/SNAPPY/LZ4/ZSTD/LZ4_FAST (default: LZ4)
Bloom Filters:
BLOOM_FILTER Enable bloom filters (default: ON)
BLOOM_FPR FPR in parts per 10000; 100 = 1% (default: 100)
Durability:
SYNC_MODE NONE/INTERVAL/FULL (default: FULL)
SYNC_INTERVAL_US Sync interval in microseconds (default: 128000)
Isolation:
ISOLATION_LEVEL READ_UNCOMMITTED/READ_COMMITTED/REPEATABLE_READ/
SNAPSHOT/SERIALIZABLE (default: REPEATABLE_READ)
LSM Tree:
USE_BTREE Use B+tree SSTable format (default: OFF)
LEVEL_SIZE_RATIO Level size multiplier (default: 10)
MIN_LEVELS Minimum LSM levels (default: 1, matches library)
DIVIDING_LEVEL_OFFSET Compaction dividing level offset (default: 1, matches library)
L1_FILE_COUNT_TRIGGER L1 file count trigger for compaction (default: 4)
L0_QUEUE_STALL_THRESHOLD L0 queue stall threshold (default: 10, matches library)
TOMBSTONE_DENSITY_TRIGGER Ratio in parts per 10000 above which compaction
is escalated for a single SSTable when the
tombstone-density check fires after a flush
(e.g. 5000 = 0.50; default: 0 = disabled).
TOMBSTONE_DENSITY_MIN_ENTRIES
Minimum entry count for an SSTable to be
considered by the density trigger; smaller
SSTables are skipped (default: 1024).
Skip List:
SKIP_LIST_MAX_LEVEL Max skip list level (default: 12)
SKIP_LIST_PROBABILITY Percentage; 25 = 0.25 (default: 25)
Block Indexes:
BLOCK_INDEXES Enable block indexes (default: ON)
INDEX_SAMPLE_RATIO Sample ratio for block index (default: 1)
BLOCK_INDEX_PREFIX_LEN Block index prefix length (default: 16)
TTL:
TTL Default TTL in seconds, 0 = none (default: 0)
Object Store:
OBJECT_LAZY_COMPACTION Double L1 compaction trigger in S3 mode (default: OFF)
OBJECT_PREFETCH_COMPACTION Prefetch inputs before merge in S3 mode (default: ON)
Encryption:
ENCRYPTED Enable data-at-rest encryption (default: OFF)
ENCRYPTION_KEY_ID Encryption key ID (default: 1)
DEFAULTS ALIGNMENT WITH THE LIBRARY
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
The column-family option defaults track tidesdb_default_column_family_config
in the underlying TidesDB library 1:1 except for two deliberate
deviations chosen to fit SQL semantics. tidesdb_defaults_alignment in
the MTR suite pins every default so a future change has to be made
deliberately.
SYNC_MODE plugin FULL, library NONE. SQL clients expect
committed writes to be durable. Only the per CF
SSTable sync is affected since the shared WAL is
governed by tidesdb_unified_memtable_sync_mode.
ISOLATION_LEVEL plugin REPEATABLE_READ, library READ_COMMITTED.
Matches MariaDB's default session isolation so a
plain CREATE TABLE mirrors what an InnoDB table
would do.
Every other CF default (write_buffer_size 64MB, level_size_ratio,
min_levels, dividing_level_offset, klog_value_threshold, compression
LZ4, bloom filter on with fpr 0.01, block indexes, index sample ratio,
block_index_prefix_len, skip list shape, l1 trigger, l0 stall
threshold, tombstone density triggers, min_disk_space, use_btree,
object_lazy_compaction, object_prefetch_compaction) matches the
library exactly so a table created in the plugin and one created with
a bare libtidesdb application share the same on disk shape.
tidesdb_unified_memtable_write_buffer_size defaults to 256MB which is a
DB-level shared WAL+memtable knob and intentionally larger than a per
CF buffer since one pool serves every table on the instance.
FIELD OPTIONS (per-column)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
TTL Marks column as per-row TTL source (seconds)
Example:
CREATE TABLE t (
id INT PRIMARY KEY,
data VARCHAR(100),
expires INT TTL=1
) ENGINE=TIDESDB TTL=3600 SYNC_MODE='NONE' COMPRESSION='ZSTD';
TESTING
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
Run all TidesDB tests (from the build directory):
cd mysql-test
perl mtr --suite=tidesdb
Run specific test:
perl mtr --suite=tidesdb tidesdb_crud
Run with verbose output:
perl mtr --suite=tidesdb --verbose
QUICK TEST
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
CREATE TABLE t (id INT PRIMARY KEY, data VARCHAR(100)) ENGINE=TidesDB;
INSERT INTO t VALUES (1, 'hello'), (2, 'world');
SELECT * FROM t;
DROP TABLE t;
SANITIZER BUILDS
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
The TidesDB plugin supports plugin-level UBSAN/ASAN without rebuilding the
entire MariaDB server:
cmake .. -DTIDESDB_WITH_UBSAN=ON # UBSAN only
cmake .. -DTIDESDB_WITH_ASAN=ON # ASAN only
cmake .. -DTIDESDB_WITH_ASAN=ON -DTIDESDB_WITH_UBSAN=ON # both
make -j$(nproc) tidesdb
# Run tests with sanitizer error logging:
UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=0 \
perl ./mtr --suite=tidesdb --parallel=4
For full ASAN coverage (requires full server rebuild):
cmake .. -DWITH_ASAN=ON -DWITH_UBSAN=ON -DCMAKE_BUILD_TYPE=Debug
make -j$(nproc)
LICENSE
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░
GNU General Public License v2