{"id":13565235,"url":"https://github.com/FasterXML/TransiStore","last_synced_at":"2025-04-03T22:30:52.890Z","repository":{"id":5813243,"uuid":"7028243","full_name":"FasterXML/TransiStore","owner":"FasterXML","description":"Distributed data store for transient (temporary, time-bound) data. Based on ClusterMate/StoreMate","archived":false,"fork":false,"pushed_at":"2022-06-22T03:14:36.000Z","size":1037,"stargazers_count":21,"open_issues_count":1,"forks_count":7,"subscribers_count":8,"default_branch":"master","last_synced_at":"2025-03-18T23:45:52.209Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/FasterXML.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-12-06T02:03:51.000Z","updated_at":"2022-06-22T03:14:39.000Z","dependencies_parsed_at":"2022-08-26T10:00:35.331Z","dependency_job_id":null,"html_url":"https://github.com/FasterXML/TransiStore","commit_stats":null,"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FasterXML%2FTransiStore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FasterXML%2FTransiStore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FasterXML%2FTransiStore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/FasterXML%2FTransiStore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/FasterXML","download_url":"https://codeload.github.com/FasterXML/TransiStore/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247089918,"owners_count":20881882,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T13:01:42.918Z","updated_at":"2025-04-03T22:30:51.781Z","avatar_url":"https://github.com/FasterXML.png","language":"Java","readme":"# TransiStore\n\nTransiStore is a distributed data store for temporary (time-bound, possibly but not necessarily transient) data, such as intermediate processing results (for Map/Reduce, Hadoop), staging area between high-volume producers and consumers (log processing and aggregation), or just as general-purpose store for data exchange.\n\nTransiStore is built on [ClusterMate](https://github.com/cowtowncoder/ClusterMate) platform, and serves as a sample system.\nNote that implementing storage system with different behavior, using `ClusterMate` (possibly using `TransiStore` as sample code) is highly encouraged: it is not meant as \"The\" storage system; although if it works as-is for your use case, all the better.\n\n## Basics\n\n### License\n\n'TransiStore` (and all its dependencies) are licensed under [Apache 2.0](http://www.apache.org/licenses/LICENSE-2.0.html).\n\n### Functionality\n\n`TransiStore` can be viewed as a distributed key/value (\"BLOB\") store which explicitly supports Key Range queries for primary keys.\nStored entries have explicit time-to-live setting and service removes expired entries if they have not been explicitly deleted earlier.\nClusters are horizontally scalable and allow addition/removal of storage nodes, without restarting other nodes.\n\nThe main limitation -- as of now, at least -- is that content is Write-Once; that is, entries are immutable after being added: they may however. be explicitly deleted (and if not, will expire as per time-to-live settings). While this is a limit that use cases must conform to, it greatly simplifies implementation and improves handling performance, as conflict resolution is simple to handle (due to minimal number of cases to resolve).\n\n### Underlying (per-node) storage\n\nStorage layer comes from [ClusterMate](https://github.com/cowtowncoder/ClusterMate) (which in turn builds on [StoreMate](https://github.com/cowtowncoder/StoreMate) project.\n\nHere are some highlights (for more refer to `StoreMate` project) of storage system:\n\n* Pluggable backends: default implementation uses `BDB-JE` for local storage, but there is also experimental `LevelDB` backend.\n* Automatic on-the-fly (de)compression; negotiated using standard HTTP; supports multiple compression methods (client can pre-compress instead of server, or defer decompression)\n* Partial content queries (HTTP Range supported)\n* Key-range queries.\n\nand additional features that `ClusterMate` provides are:\n\n* Peer-to-peer content synchronization used for on-going content synchronization, recovery, and bootstrapping of newly added nodes\n* Configurable redundancy (number of copies to store), with different client-controlled minimal required writes.\n* Client-configurable data expiration rates (per-entry time-to-live) to ensure that content will not live forever even if no explicit deletions are performed\n* Key partitioning to support cluster-wide key-range queries (routing by partition; queries within single partition) -- note: key structure fully configurable at `ClusterMate` level; TransiStore uses a simple `partition + path` key structure.\n\n### Configuration\n\nConfiguration is simple: it consists of a single JSON configuration file. Sample configuration\nfiles can be found from under `sample/`.\n\n## Why not X?\n\n(where X may be \"HBase\", \"Cassandra\", \"Voldemort\", \"Riak\" or any of dozens of distributed key/value stores)\n\nFeatures that separate TransiStore for most other \"noSQL\" key/value stores are ones listed earlier:\n\n1. Can store LARGE entries (similar to S3): authors use it for storing 10 megabyte sized entries. It is designed for such usage -- you can even do partial GETs (with HTTP Range header)\n2. Key-range queries: some noSQL systems (like HBase) support them, but most don't (Cassandra, Voldemort), at least with recommended set ups. TransiStore is designed to  fully support them.\n3. Automatic expiration of data: useful for intermediate results (Hadoop processing? Log aggregation?), weekly reports, or even some types of caching. You just specify maximum life-time for entries and system purges them as necessary\n4. Simple configuration: single configuration file, used for all nodes; and even clients if you want.\n5. Automatic node recovery: once a node has its configuration file, it is ready to start up and synchronize entries it needs -- no manual process or delicate coordination needed (NOTE: there will be need to build dashboards, tools to help track progress -- but foundational functionality works for actual syncing)\n\nThere are many more advanced features; but most of them are common within this class of servers\n(recovery, configurable number of copies required, statistics/metrics).\n\n## Documentation\n\nHere are things you will probably want to read first:\n\n* [Installing](../../wiki/Install) (my first TransiStore cluster): Build or download jar, find a config, start up!\n* [Configuration](../../wiki/Configuration): what do those sample configs contain?\n* [Tools](../../wiki/Tools): Yes, we have small set of command-line tools too; accessed with `tstore.sh` wrapper (invoking `ts-client` jar)\n\nand here's some more stuff:\n\n* [Dependencies](../../wiki/Dependencies) -- what is TransiStore built of?\n","funding_links":[],"categories":["Java"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFasterXML%2FTransiStore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FFasterXML%2FTransiStore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FFasterXML%2FTransiStore/lists"}