{"id":49458387,"url":"https://github.com/ne0ekspert/driftts","last_synced_at":"2026-04-30T08:05:00.677Z","repository":{"id":352552528,"uuid":"1212164771","full_name":"ne0ekspert/driftTS","owner":"ne0ekspert","description":"Lightweight time series DB","archived":false,"fork":false,"pushed_at":"2026-04-20T04:31:19.000Z","size":81,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-20T06:29:08.373Z","etag":null,"topics":["database","time-series","time-series-database"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ne0ekspert.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-16T05:56:08.000Z","updated_at":"2026-04-20T04:31:08.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ne0ekspert/driftTS","commit_stats":null,"previous_names":["ne0ekspert/driftts"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ne0ekspert/driftTS","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ne0ekspert%2FdriftTS","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ne0ekspert%2FdriftTS/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ne0ekspert%2FdriftTS/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ne0ekspert%2FdriftTS/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ne0ekspert","download_url":"https://codeload.github.com/ne0ekspert/driftTS/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ne0ekspert%2FdriftTS/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32458256,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T22:27:22.272Z","status":"online","status_checked_at":"2026-04-30T02:00:05.929Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["database","time-series","time-series-database"],"created_at":"2026-04-30T08:04:27.156Z","updated_at":"2026-04-30T08:05:00.669Z","avatar_url":"https://github.com/ne0ekspert.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# drift-ts\n\n`drift-ts` is a small Rust time-series database with a gRPC interface. It stores typed series (`f64`, `i64`, `bool`), buffers incoming samples in memory, flushes them into immutable segment files, and serves range queries over both flushed and buffered data.\n\nThe crate ships as both:\n\n- a library crate: `drift_ts`\n- a server binary: `drift-ts`\n- a benchmark binary: `throughput`\n\n## Features\n\n- Typed series registration by `data_id`\n- Batched append path with per-sample rejection details\n- Range queries by `data_id` and timestamp window\n- Manifest-backed recovery on startup\n- CRC32-protected segment files\n- Optional `none|zstd` compression for flushed segment files\n- Configurable flush threshold and per-series retention cap\n- Automatic oldest-segment eviction when a series exceeds its retention budget\n- gRPC services generated from [`proto/driftts/v1/driftts.proto`](/home/ne0ekspert/drifTS/proto/driftts/v1/driftts.proto)\n\n## Project Layout\n\n- [`src/main.rs`](/home/ne0ekspert/drifTS/src/main.rs) starts the gRPC server\n- [`src/core/engine.rs`](/home/ne0ekspert/drifTS/src/core/engine.rs) implements ingestion, flushing, querying, recovery, and eviction\n- [`src/core/segment.rs`](/home/ne0ekspert/drifTS/src/core/segment.rs) defines the on-disk segment format\n- [`src/core/manifest.rs`](/home/ne0ekspert/drifTS/src/core/manifest.rs) tracks registered series and flushed segments\n- [`src/config.rs`](/home/ne0ekspert/drifTS/src/config.rs) loads the TOML config\n- [`src/bin/throughput.rs`](/home/ne0ekspert/drifTS/src/bin/throughput.rs) runs a local ingest/query benchmark\n- [`tests/grpc_smoke.rs`](/home/ne0ekspert/drifTS/tests/grpc_smoke.rs) covers the basic register -\u003e append -\u003e query -\u003e stats flow\n\n## Requirements\n\n- Rust toolchain with Cargo\n- No system `protoc` is required; the build uses `protoc-bin-vendored`\n\n## Quick Start\n\nBuild the project:\n\n```bash\ncargo build\n```\n\nUse the sample config in [`drift-ts.toml`](/home/ne0ekspert/drifTS/drift-ts.toml):\n\n```toml\n# TCP listener:\nlisten_addr = \"127.0.0.1\"\nlisten_port = 50051\ndata_dir = \"data\"\nflush_threshold_count = 1000\ndefault_series_max_bytes = 104857600\nsegment_compression = \"zstd\"\n```\n\n`listen_addr` can also be a Unix domain socket path on Unix systems, for example:\n\n```toml\n# Unix socket listener:\nlisten_addr = \"/tmp/drift-ts.sock\"\n```\n\nRun the server:\n\n```bash\ncargo run --bin drift-ts -- --config drift-ts.toml\n```\n\nIf `--config` is omitted, the server looks for `drift-ts.toml` in the current working directory.\n\n## Configuration\n\nThe server accepts exactly one CLI flag:\n\n```text\n--config \u003cpath\u003e\n```\n\nConfig fields:\n\n- `listen_addr`: TCP host/IP such as `127.0.0.1` or `localhost`, or a Unix socket path such as `/tmp/drift-ts.sock`\n- `listen_port`: optional TCP port. When set, the server binds TCP using `listen_addr` + `listen_port`. When omitted, `listen_addr` is treated as a Unix socket path\n- `data_dir`: root directory for `manifest.json` and segment files\n- `flush_threshold_count`: number of buffered samples per series before an automatic flush\n- `default_series_max_bytes`: optional flushed-segment retention ceiling applied to newly registered series when they do not provide an explicit limit\n- `segment_compression`: codec for newly written segment files: `none` or `zstd` (defaults to `zstd` if omitted)\n\nValidation rules:\n\n- `listen_addr` must be non-empty\n- when `listen_port` is set, `listen_addr` must be a TCP host/IP, not a path\n- when `listen_port` is omitted, `listen_addr` must be a Unix socket path\n- `flush_threshold_count` must be greater than zero\n- `default_series_max_bytes` must be greater than zero when set\n\n## gRPC API\n\nThe protobuf definition lives at [`proto/driftts/v1/driftts.proto`](/home/ne0ekspert/drifTS/proto/driftts/v1/driftts.proto).\n\nServices:\n\n- `IngestService`\n- `QueryService`\n- `AdminService`\n\nRPCs:\n\n- `RegisterSeries(data_id, series_type, max_storage_bytes?)`\n- `AppendBatch(samples[])`\n- `FlushSeries(data_id)`\n- `FlushAll()`\n- `RangeQuery(data_id, start_ts_ms, end_ts_ms, limit?)`\n- `Stats()`\n- `Health()`\n\n### Example With `grpcurl`\n\nRegister an `i64` series:\n\n```bash\ngrpcurl \\\n  -plaintext \\\n  -import-path proto \\\n  -proto proto/driftts/v1/driftts.proto \\\n  -d '{\"data_id\":42,\"series_type\":\"SERIES_TYPE_I64\",\"max_storage_bytes\":104857600}' \\\n  127.0.0.1:50051 \\\n  driftts.v1.IngestService/RegisterSeries\n```\n\nAppend two samples:\n\n```bash\ngrpcurl \\\n  -plaintext \\\n  -import-path proto \\\n  -proto proto/driftts/v1/driftts.proto \\\n  -d '{\n    \"samples\": [\n      {\"data_id\":42,\"timestamp_ms\":100,\"i64_value\":1},\n      {\"data_id\":42,\"timestamp_ms\":200,\"i64_value\":2}\n    ]\n  }' \\\n  127.0.0.1:50051 \\\n  driftts.v1.IngestService/AppendBatch\n```\n\nQuery a range:\n\n```bash\ngrpcurl \\\n  -plaintext \\\n  -import-path proto \\\n  -proto proto/driftts/v1/driftts.proto \\\n  -d '{\"data_id\":42,\"start_ts_ms\":0,\"end_ts_ms\":1000}' \\\n  127.0.0.1:50051 \\\n  driftts.v1.QueryService/RangeQuery\n```\n\nRead stats:\n\n```bash\ngrpcurl \\\n  -plaintext \\\n  -import-path proto \\\n  -proto proto/driftts/v1/driftts.proto \\\n  -d '{}' \\\n  127.0.0.1:50051 \\\n  driftts.v1.AdminService/Stats\n```\n\n## Storage Model\n\nOn disk, the database stores:\n\n- `manifest.json`: registered series, segment metadata, and storage accounting\n- `segments/\u003cdata_id\u003e/\u003csegment_id\u003e.seg`: immutable flushed segment files\n\nSegment files contain:\n\n- a fixed header with series metadata and min/max timestamps\n- encoded sample records, optionally compressed with `zstd`\n- a CRC32 checksum over the uncompressed segment body\n\nStartup recovery:\n\n- removes stale `*.tmp` files\n- reloads `manifest.json` if present\n- rescans segment files on disk\n- restores missing manifest entries for valid segment files\n- drops mismatched segment files if their type conflicts with the registered series\n- supports mixed legacy uncompressed segments and new compressed segments in the same data directory\n\n## Semantics and Caveats\n\n- Series must be registered before they can accept samples.\n- Each series has a fixed type. Type mismatches are rejected.\n- Samples older than or equal to the latest flushed timestamp for a series are rejected.\n- Duplicate timestamps inside the in-memory buffer are deduplicated on flush; the last write wins.\n- Query results merge flushed data with buffered data and also keep the last value for duplicate timestamps.\n- Per-series `max_storage_bytes` applies to flushed segments only. When exceeded, the engine deletes the oldest flushed segment in that series until usage is back under the limit.\n- `default_series_max_bytes` is copied into a series when it is registered without an explicit limit, including older manifests upgraded during recovery.\n- `storage_bytes` and eviction use the actual on-disk size of segment files, so compressed segments count by their compressed byte size.\n- The engine does not enforce a node-wide disk ceiling. Total disk usage is the sum of each series' retained flushed segments.\n- Buffered, unflushed samples are not persisted across process restarts.\n- `Health()` currently returns a static `ok=true` / `\"ok\"` response.\n- There is no authentication, TLS, or compaction layer in the current server.\n\n## Benchmark Binary\n\nThe repository includes a local throughput benchmark:\n\n```bash\ncargo run --release --bin throughput -- --help\n```\n\nTypical run:\n\n```bash\ncargo run --release --bin throughput -- \\\n  --ingest-mode append-batch \\\n  --samples 1000000 \\\n  --series 8 \\\n  --flush-threshold 4096 \\\n  --batch-size 1024 \\\n  --queries 2000 \\\n  --query-span 2048 \\\n  --segment-compression zstd\n```\n\nSupported flags:\n\n- `--ingest-mode append-one|append-batch`\n- `--samples N`\n- `--series N`\n- `--flush-threshold N`\n- `--batch-size N`\n- `--queries N`\n- `--query-span N`\n- `--default-series-max-bytes N`\n- `--segment-compression none|zstd`\n- `--data-dir PATH`\n\nIf `--data-dir` is omitted, the benchmark uses a temporary directory and removes it after completion.\n\n## Testing\n\nRun the test suite:\n\n```bash\ncargo test\n```\n\nThe existing smoke test verifies:\n\n- series registration\n- batched append\n- range query correctness\n- admin stats\n- health endpoint behavior\n\n## License\n\nThis project is licensed under the terms in [`LICENSE`](/home/ne0ekspert/drifTS/LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fne0ekspert%2Fdriftts","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fne0ekspert%2Fdriftts","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fne0ekspert%2Fdriftts/lists"}