{"id":13449719,"url":"https://github.com/0b01/tectonicdb","last_synced_at":"2025-12-12T14:48:16.324Z","repository":{"id":25737519,"uuid":"105599508","full_name":"0b01/tectonicdb","owner":"0b01","description":"Database for L2 orderbook","archived":false,"fork":false,"pushed_at":"2024-01-25T09:15:02.000Z","size":72647,"stargazers_count":701,"open_issues_count":17,"forks_count":93,"subscribers_count":20,"default_branch":"master","last_synced_at":"2025-04-28T11:51:00.841Z","etag":null,"topics":["cryptocurrency","hft","high-frequency-trading","orderbook","trading"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/0b01.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-10-03T00:34:01.000Z","updated_at":"2025-04-21T07:14:25.000Z","dependencies_parsed_at":"2024-06-21T18:07:49.417Z","dependency_job_id":null,"html_url":"https://github.com/0b01/tectonicdb","commit_stats":{"total_commits":497,"total_committers":15,"mean_commits":33.13333333333333,"dds":"0.33601609657947684","last_synced_commit":"c8fc836e2dd6266b63a187fa1371e59dd96c65db"},"previous_names":["rickyhan/tectonicdb"],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2Ftectonicdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2Ftectonicdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2Ftectonicdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/0b01%2Ftectonicdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/0b01","download_url":"https://codeload.github.com/0b01/tectonicdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254319719,"owners_count":22051073,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cryptocurrency","hft","high-frequency-trading","orderbook","trading"],"created_at":"2024-07-31T06:00:52.382Z","updated_at":"2025-12-12T14:48:11.283Z","avatar_url":"https://github.com/0b01.png","language":"Rust","funding_links":[],"categories":["Databases","Rust","trading"],"sub_categories":["Cryptocurrencies","TimeSeries Analysis"],"readme":"# tectonicdb\n\n[![build](https://github.com/0b01/tectonicdb/workflows/Rust/badge.svg)](https://github.com/0b01/tectonicdb/actions?query=workflow%3ARust)\n[![crate.io](https://img.shields.io/crates/v/tdb-core.svg)](https://crates.io/crates/tdb-core)\n[![doc.rs](https://docs.rs/tdb_core/badge.svg)](https://docs.rs/crate/tdb_core)\n![Minimum Rust version](https://img.shields.io/badge/rustc-1.40+-yellow.svg)\n![Rust stable](https://img.shields.io/badge/rust-stable-green.svg)\n\n| crate | docs.rs | crate.io |\n| - | - | ------ |\n| tectonicdb  | [![doc.rs](https://docs.rs/tectonicdb/badge.svg)](https://docs.rs/crate/tectonicdb) | [![crate.io](https://img.shields.io/crates/v/tectonicdb.svg)](https://crates.io/crates/tectonicdb)  |\n| tdb-core  | [![doc.rs](https://docs.rs/tdb_core/badge.svg)](https://docs.rs/crate/tdb_core) | [![crate.io](https://img.shields.io/crates/v/tdb-core.svg)](https://crates.io/crates/tdb-core)  |\n| tdb-server-core  | [![doc.rs](https://docs.rs/tdb_server_core/badge.svg)](https://docs.rs/crate/tdb_server_core) | [![crate.io](https://img.shields.io/crates/v/tdb-server-core.svg)](https://crates.io/crates/tdb-server-core)  |\n| tdb-cli  | [![doc.rs](https://docs.rs/tdb_cli/badge.svg)](https://docs.rs/crate/tdb_cli) | [![crate.io](https://img.shields.io/crates/v/tdb-cli.svg)](https://crates.io/crates/tdb-cli)  |\n\ntectonicdb is a fast, highly compressed standalone database and streaming protocol for order book ticks.\n\n## Why\n\n* Uses a simple and efficient binary file format: Dense Tick Format(DTF)\n\n* Stores order book tick data tuple of shape: `(timestamp, seq, is_trade, is_bid, price, size)`.\n\n* Sorted by timestamp + seq\n\n* 12 bytes per orderbook event\n\n* 600,000 inserts per thread second\n\n## Installation\n\nThere are several ways to install tectonicdb.\n\n1.  **Binaries**\n\nBinaries are available for [download](https://github.com/0b01/tectonicdb/releases). Make sure to put the path to the binary into your PATH. Currently only build is for Linux x86_64.\n\n2.  **Crates**\n\n```\ncargo install tectonicdb\n```\n\nThis command will download `tdb`, `tdb-server`, `dtftools` binaries from crates.io and build locally.\n\n3.  **GitHub**\n\nTo contribute you will need the copy of the source code on your local machine.\n\n    git clone https://github.com/0b01/tectonicdb\n    cd tectonicdb\n    cargo build --release\n    cargo run --release tdb-server\n\nThe binaries can be found under `target/release`.\n\n## How to use\n\nIt's very easy to setup.\n\n```\n./tdb-server --help\n```\n\nFor example:\n\n```bash\n./tdb-server -vv -a -i 10000\n# run the server on INFO verbosity\n# turn on autoflush for every 10000 inserts per orderbook\n```\n\n### Configuration\n\nTo config the Google Cloud Storage and Data Collection Backend integration, the following environment variables are used:\n\n| Variable Name                 | Default      | Description                                                                                                                                   |\n| ----------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------- |\n| `TDB_HOST`             | 0.0.0.0      | The host to which the database will bind                                                                                                      |\n| `TDB_PORT`             | 9001         | The port that the database will listen on                                                                                                     |\n| `TDB_DTF_FOLDER`       | db           | Name of the directory in which DTF files will be stored                                                                                       |\n| `TDB_AUTOFLUSH`        | false        | If `true`, recorded orderbook data will automatically be flushed to DTF files every `interval` inserts.                                       |\n| `TDB_FLUSH_INTERVAL`   | 1000         | Every `interval` inserts, if `autoflush` is enabled, DTF files will be written from memory to disk.                                           |\n| `TDB_GRANULARITY`      | 0            | Record history granularity level                                                                                                              |\n| `TDB_LOG_FILE_NAME`    | tdb.log      | Filename of the log file for the database                                                                                                     |\n| `TDB_Q_CAPACITY`       | 300          | Capacity of the circular queue for recording history                                                                                          |\n\n## Client API\n\n| Command | Description |\n| :--- | :--- |\n| HELP | Prints help |\n| PING | Responds PONG |\n| INFO | Returns info about table schemas |\n| PERF | Returns the answercount of items over time |\n| LOAD \\[orderbook\\] | Load orderbook from disk to memory |\n| USE \\[orderbook\\] | Switch the current orderbook |\n| CREATE \\[orderbook\\] | Create orderbook |\n| GET \\[n\\] FROM \\[orderbook\\] | Returns items |\n| GET \\[n\\] | Returns n items from current orderbook |\n| COUNT | Count of items in current orderbook |\n| COUNT ALL | Returns total count from all orderbooks |\n| CLEAR | Deletes everything in current orderbook |\n| CLEAR ALL | Drops everything in memory |\n| FLUSH | Flush current orderbook to \"Howdisk can|\n| FLUSHALL | Flush everything from memory to disk |\n| SUBSCRIBE \\[orderbook\\] | Subscribe to updates from orderbook |\n| EXISTS \\[orderbook\\] | Checks if orderbook exists |\n| SUBSCRIBE \\[orderbook\\] | Subscribe to orderbook |\n\n### Data commands\n\n```\nUSE [dbname]\nADD [ts], [seq], [is_trade], [is_bid], [price], [size];\nINSERT 1505177459.685, 139010, t, f, 0.0703620, 7.65064240; INTO dbname\n```\n\n## Monitoring\n\nTectonicDB supports monitoring/alerting by periodically sending its usage info to an InfluxDB instance:\n\n```bash\n    --influx-db \u003cinflux_db\u003e                        influxdb db\n    --influx-host \u003cinflux_host\u003e                    influxdb host\n    --influx-log-interval \u003cinflux_log_interval\u003e    influxdb log interval in seconds (default is 60)\n```\n\nAs a concrete example,\n\n```bash\n...\n$ influx\n\u003e CREATE DATABASE market_data;\n\u003e ^D\n$ tdb --influx-db market_data --influx-host http://localhost:8086 --influx-log-interval 20\n...\n```\n\nTectonicDB will send field values `disk={COUNT_DISK},size={COUNT_MEM}` with tag `ob={ORDERBOOK}` to `market_data` measurement which is the same as the dbname.\n\nAdditionally, you can query usage information directly with `INFO` and `PERF` commands:\n\n1. `INFO` reports the current tick count in memory and on disk.\n\n2. `PERF` returns recorded tick count history whose granularity can be configured.\n\n## Logging\n\nLog file defaults to `tdb.log`.\n\n## Testing\n\n```bash\nexport RUST_TEST_THREADS=1\ncargo test\n```\n\nTests must be run sequentially because some tests depend on dtf files that other tests generate.\n\n## Benchmark\n\ntdb client comes with a benchmark mode. This command inserts 1M records into the tdb.\n\n```bash\ntdb -b 1000000\n```\n## Using dtf files\n\nTectonic comes with a commandline tool `dtfcat` to inspect the file metadata and all the stored events into either JSON or CSV.\n\nOptions:\n\n```\nUSAGE:\n    dtfcat [FLAGS] --input \u003cINPUT\u003e\n\nFLAGS:\n    -c, --csv         output csv\n    -h, --help        Prints help information\n    -m, --metadata    read only the metadata\n    -V, --version     Prints version information\n\nOPTIONS:\n    -i, --input \u003cINPUT\u003e    file to read\n```\n\n## As a library\n\nIt is possible to use the Dense Tick Format streaming protocol / file format in a different application. Works nicely with any buffer implementing the `Write` trait.\n\n## Requirements\n\nTectonicDB is a standalone service.\n\n* Linux\n\n* macOS\n\nLanguage bindings:\n\n* [x] TypeScript\n\n* [x] Rust\n\n* [x] Python\n\n* [x] JavaScript\n\n## Additional Features\n\n* [x] Usage statistics like Cloud SQL\n\n* [x] Commandline inspection tool for dtf file format\n\n* [x] Logging\n\n* [x] Query by timestamp\n\n# Changelog\n\n* 0.5.0: InfluxDB monitoring plugin and improved command line arguments\n* 0.4.0: iterator-based APIs for handling DTF files and various quality of life improvements\n* 0.3.0: Refactor to async\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0b01%2Ftectonicdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2F0b01%2Ftectonicdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2F0b01%2Ftectonicdb/lists"}