{"id":51778777,"url":"https://github.com/ydb-platform/ydb-janusgraph-storage-backend","last_synced_at":"2026-07-20T09:31:15.322Z","repository":{"id":372111781,"uuid":"1304804378","full_name":"ydb-platform/ydb-janusgraph-storage-backend","owner":"ydb-platform","description":null,"archived":false,"fork":false,"pushed_at":"2026-07-18T23:00:49.000Z","size":91,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-19T01:04:51.545Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ydb-platform.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-07-18T10:20:23.000Z","updated_at":"2026-07-18T23:00:53.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/ydb-platform/ydb-janusgraph-storage-backend","commit_stats":null,"previous_names":["ydb-platform/ydb-janusgraph-storage-backend"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/ydb-platform/ydb-janusgraph-storage-backend","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-janusgraph-storage-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-janusgraph-storage-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-janusgraph-storage-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-janusgraph-storage-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ydb-platform","download_url":"https://codeload.github.com/ydb-platform/ydb-janusgraph-storage-backend/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ydb-platform%2Fydb-janusgraph-storage-backend/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35682455,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-07-20T02:08:10.276Z","status":"ssl_error","status_checked_at":"2026-07-20T02:08:09.736Z","response_time":111,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-07-20T09:31:13.887Z","updated_at":"2026-07-20T09:31:15.315Z","avatar_url":"https://github.com/ydb-platform.png","language":"Java","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JanusGraph YDB Storage Backend\n\nA [JanusGraph](https://janusgraph.org) storage backend that keeps graph data in\n[YDB](https://ydb.tech) — a distributed, strongly consistent NewSQL database.\nThe result is a horizontally scalable, transactional TinkerPop/Gremlin graph.\n\nThe backend implements JanusGraph's **ordered key-value SPI**\n(`OrderedKeyValueStoreManager`), the same integration path used by the\nBerkeleyDB and FoundationDB backends. YDB is a natural fit for this SPI: it is\nan ordered (range-sharded by primary key), transactional, distributed store —\narchitecturally the closest open-source relative of FoundationDB.\n\n## Compatibility\n\n| Component | Version |\n|---|---|\n| JanusGraph | 1.1.0 |\n| YDB Java SDK | 2.4.7 |\n| YDB server | any version supported by SDK 2.4.x (tested with `ydbplatform/local-ydb` 26.x) |\n| Java | 8+ (module is built with target 1.8) |\n\n## Quick start\n\nRun a local YDB:\n\n```bash\ndocker run -d --rm --name ydb-local -h localhost \\\n  -p 2136:2136 -p 8765:8765 \\\n  -e GRPC_PORT=2136 -e MON_PORT=8765 -e YDB_USE_IN_MEMORY_PDISKS=true \\\n  ydbplatform/local-ydb:latest\n```\n\nPut the `janusgraph-ydb` jar and its dependencies on the JanusGraph classpath\n(e.g. into `lib/` or `ext/` of a JanusGraph distribution) and configure:\n\n```properties\ngremlin.graph=org.janusgraph.core.JanusGraphFactory\nstorage.backend=org.janusgraph.diskstorage.ydb.YdbStoreManager\nstorage.ydb.endpoint=grpc://localhost:2136\nstorage.ydb.database=/local\nstorage.ydb.directory=janusgraph\n# activates batched slice reads (recommended)\nquery.batch.enabled=true\n```\n\n\u003e **Why the full class name?** JanusGraph resolves `storage.backend` against a\n\u003e hard-coded enum of built-in backends and otherwise treats the value as a fully\n\u003e qualified class name. There is no registration mechanism (ServiceLoader or\n\u003e similar) that would let an external module claim a shorthand like `ydb`, so\n\u003e the FQCN is the only way to wire in any third-party backend — the\n\u003e FoundationDB backend is configured the same way.\n\nThen use JanusGraph as usual:\n\n```java\nJanusGraph graph = JanusGraphFactory.open(\"janusgraph-ydb.properties\");\nGraphTraversalSource g = graph.traversal();\n```\n\n## Configuration reference (`storage.ydb.*`)\n\n| Option | Default | Description |\n|---|---|---|\n| `storage.ydb.endpoint` | `grpc://localhost:2136` | YDB endpoint; use `grpcs://` for TLS |\n| `storage.ydb.database` | `/local` | YDB database path |\n| `storage.ydb.directory` | `janusgraph` | Directory inside the database holding all store tables; falls back to `graph.graphname` |\n| `storage.ydb.read-tx-mode` | `snapshot_ro` | YDB transaction mode for reads: `snapshot_ro`, `stale_ro`, `online_ro` (lock-free pooled reads, buffered writes) or `serializable_rw` (one interactive transaction) |\n| `storage.ydb.auth-mode` | `anonymous` | `anonymous`, `token`, `static` or `environ` (standard `YDB_*` env vars) |\n| `storage.ydb.auth-token` | — | Access token for `auth-mode=token` |\n| `storage.ydb.auth-username` / `auth-password` | — | Credentials for `auth-mode=static` |\n| `storage.ydb.session-pool-max` | `50` | Max YDB sessions; bounds the number of concurrently open JanusGraph transactions |\n| `storage.ydb.session-acquire-timeout` | `10 s` | Wait for a free session before failing (retryably) |\n| `storage.ydb.scan-page-size` | `10000` | Max rows per round trip during range scans |\n| `storage.ydb.slice-batch-size` | `10` | Max ranges combined into one multi-statement query by batched slice reads |\n| `storage.ydb.auto-partitioning` | `true` | Enable auto-partitioning by size and load for store tables |\n| `storage.ydb.presplit-partitions` | `0` | Pre-split new store tables into N shards at uniform key boundaries (aligned with JanusGraph's virtual partitions); avoids the single-hot-shard warm-up phase under ingest |\n\n## Data model\n\nEach JanusGraph store (`edgestore`, `graphindex`, `janusgraph_ids`,\n`system_properties`, `txlog`, `systemlog`) becomes one YDB row table under\n`\u003cdatabase\u003e/\u003cdirectory\u003e/`:\n\n```sql\nCREATE TABLE \u003cstore\u003e (\n    key   String NOT NULL,   -- raw bytes\n    value String,            -- raw bytes\n    PRIMARY KEY (key)\n);\n```\n\nYDB orders `String` primary keys by unsigned lexicographic byte comparison\n(`memcmp`, shorter prefix first) — exactly the order JanusGraph requires from an\nordered key-value store; a dedicated test (`YdbKeyOrderTest`) guards this\nproperty. Tables are created idempotently on startup (`CREATE TABLE IF NOT\nEXISTS`); existing tables are never touched.\n\n## Transactions and consistency\n\nThe backend has two shapes, selected by `storage.ydb.read-tx-mode`.\n\n**Read-only read modes** (`snapshot_ro`, default; also `stale_ro` / `online_ro`\nfor lower-latency, less-fresh reads):\n\n* Reads execute as independent **lock-free** read-only queries through the\n  session pool: no session pinning, no read locks, no transaction aborts for\n  long traversals, fully parallel batched slice reads.\n* Writes are buffered client-side; a transaction always observes its own writes\n  (reads merge the buffer). On commit the whole buffer is applied atomically in\n  one `SERIALIZABLE_RW` batch driven through YDB's retry machinery.\n* There is **no locking at all, by design** (last-write-wins): concurrent\n  writers to the same element race, and unique-index constraints are not\n  enforced under concurrent insertion of the same key. This is the intended\n  trade-off for read-heavy workloads such as agent memory; use\n  `serializable_rw` when you need enforced invariants.\n* Visibility is read-committed-style: each read sees the latest committed data\n  (`snapshot_ro` — a consistent per-query snapshot).\n\n**`serializable_rw`**: one interactive serializable YDB transaction per\nJanusGraph transaction, pinned to a pooled session. Reads take optimistic\nlocks; conflicting commits fail with `ABORTED` (surfaced as retryable\n`TemporaryBackendException`). The transaction keeps one consistent snapshot for\nits whole lifetime. See `YdbReadModeSemanticsTest` for both semantics.\n\n**Common to both modes**:\n\n* Retryable YDB statuses map to `TemporaryBackendException`, which JanusGraph's\n  own machinery retries (e.g. id-block allocation — safe in both modes: it\n  relies on consistent reads, not locking).\n* Mutations of one flush are applied as a **single multi-statement query**\n  (`DELETE … ON SELECT FROM AS_TABLE($keys)` + `UPSERT … SELECT FROM\n  AS_TABLE($rows)` per store) — one round trip regardless of batch size.\n* Batched slice reads (`getSlices`, activated by `query.batch.enabled=true`)\n  combine up to `slice-batch-size` range reads into one multi-statement query;\n  in the read-only read modes the batches additionally execute **in parallel**.\n* With `storage.batch-loading=true` writes bypass transactions entirely:\n  additions stream through YDB's **BulkUpsert** API, deletions run as\n  auto-committed batches — the fastest ingestion path (not atomic, not\n  rollbackable, as usual for batch loading).\n\n## Vector search (mixed indexes)\n\nThe module ships a mixed-index backend (`IndexProvider`) that keeps index\ndocuments — including **vector embeddings** — in the same YDB database:\n\n```properties\nindex.vec.backend=org.janusgraph.diskstorage.ydb.index.YdbIndexProvider\nindex.vec.ydb.endpoint=grpc://localhost:2136\nindex.vec.ydb.database=/local\n```\n\n```java\n// schema: float[] is a built-in JanusGraph property type\nPropertyKey embedding = mgmt.makePropertyKey(\"embedding\").dataType(float[].class).make();\nmgmt.buildIndex(\"memoryVectors\", Vertex.class)\n    .addKey(embedding, Parameter.of(ParameterType.customParameterName(\"dimension\"), 1536))\n    .addKey(mgmt.makePropertyKey(\"kind\").dataType(String.class).make(), Mapping.STRING.asParameter())\n    .buildMixedIndex(\"vec\");\n\n// write: the index follows graph commits automatically (updates and deletions too)\ngraph.addVertex(\"kind\", \"fact\", \"embedding\", vector);\n\n// kNN: nearest neighbours with scores, no Gremlin required\ngraph.indexQuery(\"memoryVectors\", YdbVectors.nearest(\"embedding\", queryVector))\n     .limit(20).vertexStream();\n```\n\nCapabilities and behavior:\n\n* Scalar fields (string/number/date/boolean/UUID) support exact and range\n  predicates (`Cmp`), so `has()` lookups route through the mixed index.\n* kNN runs as an **exact scan** (always correct, O(n)) until\n  `YdbIndexProvider#buildVectorIndex(store, field)` builds a\n  `vector_kmeans_tree` index — built online under a temporary name and swapped\n  in atomically; after that queries use ANN through the index. The index is\n  mutable (inserts/updates/deletes are immediately visible), but YDB never\n  recalculates centroids — rebuild periodically as the data drifts.\n* Distance strategies: `cosine` (default), `euclidean`, `manhattan`,\n  `inner_product` — via `index.[X].ydb.vector-distance` or the per-key custom\n  parameter `distance`. Scores returned to JanusGraph are \"higher is closer\".\n* `REINDEX` (`mgmt.updateIndex(...)`) is supported and rebuilds documents\n  through the provider's restore path.\n* Not supported: full-text and geo predicates (YDB has neither), LIST/SET\n  cardinality. The standard `IndexProviderTest` TCK hard-requires full-text and\n  geo, so coverage lives in dedicated tests (`YdbIndexProviderVectorTest`,\n  `YdbMixedIndexGraphTest`).\n\n## Limitations\n\n* `storage.backend=ydb` shorthand is impossible (see above) — use the FQCN.\n* No TTL support (`cellTTL`/`storeTTL` are disabled); JanusGraph TTL features\n  are unavailable with this backend.\n* In the read-only read modes there is no locking: concurrent writes to the\n  same element are last-write-wins and unique indexes are not enforced against\n  concurrent duplicate insertion. Switch to `read-tx-mode=serializable_rw` for\n  enforced invariants.\n* In `serializable_rw` mode every open transaction pins one pooled session\n  (size `storage.ydb.session-pool-max` accordingly) and a single transaction\n  must not be used from multiple threads concurrently.\n* Very large single transactions are bounded by YDB's transaction limits;\n  prefer `storage.batch-loading=true` (BulkUpsert) for bulk ingestion.\n\n## Development and testing\n\nA [devcontainer](.devcontainer/) ships a JDK 11 + Maven toolchain and a\n`ydbplatform/local-ydb` sidecar — nothing needs to be installed on the host:\n\n```bash\ndocker compose -f .devcontainer/docker-compose.yml up -d\ndocker compose -f .devcontainer/docker-compose.yml exec dev mvn test\n```\n\nIntegration tests resolve YDB as follows:\n\n1. If `YDB_ENDPOINT` and `YDB_DATABASE` env vars are set (the devcontainer sets\n   them to the sidecar), that instance is used directly.\n2. Otherwise a `ydbplatform/local-ydb` container is started via Testcontainers.\n\nThe suite includes the standard JanusGraph TCK: `KeyValueStoreTest` (both read\nmodes), `KeyColumnValueStoreTest` (fixed- and variable-length key packing),\n`MultiWriteKeyColumnValueStoreTest`, `IDAuthorityTest`, `KCVSLogTest`,\n`JanusGraphTest`, plus a byte-order guard test, a read-mode semantics test\n(`YdbReadModeSemanticsTest`), a batch-loading (BulkUpsert) test and a\n10k-vertex traversal smoke test. The TCK runs against the default\n`snapshot_ro` read mode.\n\nThree TCK methods are disabled, each with an in-code justification:\n\n* `testConcurrentGetSliceAndMutate` — the TCK helper races rollback/replace of\n  a shared transaction handle across 64 threads while writing through it,\n  which is inherently racy for any backend with stateful transactions;\n  BerkeleyJE and FoundationDB disable it too. The read-only variant\n  (`testConcurrentGetSlice`) stays enabled: pooled reads are stateless.\n* `testConsistencyEnforcement` / `testConcurrentConsistencyEnforcement` —\n  assert the exception type of JanusGraph's built-in locker, which this\n  backend never installs: it is deliberately lock-free in the read-only read\n  modes and relies on YDB's own conflict detection in `serializable_rw`;\n  BerkeleyJE and FoundationDB disable the same pair.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fydb-platform%2Fydb-janusgraph-storage-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fydb-platform%2Fydb-janusgraph-storage-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fydb-platform%2Fydb-janusgraph-storage-backend/lists"}