{"id":23418543,"url":"https://github.com/rustshop/loglog","last_synced_at":"2025-04-12T11:33:18.063Z","repository":{"id":48149218,"uuid":"514996439","full_name":"rustshop/loglog","owner":"rustshop","description":"Distributed, fault tolerant, strongly consistent, performant (event)log in Rust ","archived":false,"fork":false,"pushed_at":"2023-05-30T06:52:41.000Z","size":492,"stargazers_count":18,"open_issues_count":5,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-26T06:11:48.608Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rustshop.png","metadata":{"files":{"readme":"README.design.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-07-18T01:20:50.000Z","updated_at":"2025-03-23T15:38:14.000Z","dependencies_parsed_at":"2024-12-23T00:20:47.199Z","dependency_job_id":"275f03ba-f30c-43fa-ae86-c60b9339aeb7","html_url":"https://github.com/rustshop/loglog","commit_stats":{"total_commits":36,"total_committers":1,"mean_commits":36.0,"dds":0.0,"last_synced_commit":"7e83e4318ecd44e86335cf40546250fcba48e9d0"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustshop%2Floglog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustshop%2Floglog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustshop%2Floglog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rustshop%2Floglog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rustshop","download_url":"https://codeload.github.com/rustshop/loglog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248560167,"owners_count":21124605,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-12-23T00:20:14.214Z","updated_at":"2025-04-12T11:33:18.038Z","avatar_url":"https://github.com/rustshop.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# LogLog's design\n\n## Log\n\nLogically the core and only responsibility of LogLog is maintaining\nan ordered append-only binary *Log*.\n\nLogLog allows appending new *entries* (bunch of bytes) to this binary log.\nUnlike most other similiar solutions it does not interpret\nthe content of each entry. It will only add a very short prefix and suffix\nto each entry for the purpose of local and cluster-wise consistency and\ncrash resistance and append new data to the existing log.\n\nThe content of the log (including prefix and suffix of each entry) is\ndirectly accessible to the clients. Clients request binary data from log in chunks\nby log position and limit on the amount of data they would like to receive.\nLogLog does not maintain any index of entries stored in a log.\n\nIn normal circumstances LogLog server will not return parts of the\nlog that are not yet \"commited\". Client can request to block\nand wait for new data if desired, to avoid any extra polling latency.\n\n## Supported operations\n\n* Append entry to the log\n* Read log directly in pages by offset and size limit\n* Upload the entry data to arbitrary node (to help data replication)\n\n## Segments\n\nInternally log data is stored in segment files. LogLog\nwill create a new segment file after current one crosses\ncertain limit.\n\nEach segment file starts with a short preffix, mostly containing\nthe Log offset position of the data it contains.\n\n## Supported client-side functionality\n\nIt might seem like the core functionality supported by LogLog\nis very minimal and might be limiting, but it is actually possible to build out\neverything on top of it, client side.\n\n### Adding multiple log events at the time\n\nIf clients wish to be able to add multiple sub-entries to the\nlog in one \"append\" operations, they need to serialize and deserialize\nentries to include multiple sub-entries (by e.g. prefixing them with\ntheir length).\n\n### Data integrity\n\nLogLog does not perform any data consistency checks, neither\non read, nor write side. Applications that need consistency\nguarantees need to append their own checksums before appending\nentries to the log, and check them when reading entries.\nIf inconsistency was detected, data can be re-requested from\na different node. When appending entries clients upload\nentry content multiple nodes directly (more on it later),\nthere should always be enough independent copies to restore from.\n\n### Higher level data structures\n\nAlthough LogLog maintains only log, it is expected that\nit's users will build state machines following entries\nin the log and maintaining any higher level data structures\nas they see fit. There's no need for a built in key-value\nstores or any other functionality.\n\n\n## Rationale for extreme functional minimalism\n\nThe main rationale is performance. Since ordering a log\nis ultimately a sequential operation, the log is always going\nto be a bottleneck of any system, and scaling it horizontally\nwill require sacrifices in terms of consistency.\n\nBy offloading absolutely every unncessary logic to the client\nside, a LogLog cluster can (in theory at least) handle\nhigher workloads before becoming a bottleneck.\n\nThe extreme transparency of the internal log storage, and 1:1\nmapping between how data is persisted and how it is exposed and\naccessible by the clients, enables minimal overhead read and write operations.\n\n\n## HA Cluster\n\n### Independent node uploads\n\nSince all the writes will ultimately have to be handled by\nthe Cluster leader node (due to consistency), LogLog will allow\n(and even require) each writting client to upload the data to all the\ncluster nodes before commiting the new entry to the log.\nThis is to take as much load from the leader node as possible,\nand also to avoid possibility of one corrupted copy of the entry.\n\n### Raft\n\nOtherwise LogLog works is a standard Raft way: there are elections,\nleader, folloers, log entries contain Term, and consistency and HA is\nmaintained this way.\n\n## Modes of applicability\n\n### Independent Cluster\n\nLogLog can be deployed as an independent cluster, and accessed remotely\nusing Client libraries.\n\n### Built-in\n\nSince every distributed database is built on top of distributed log\nin a Raft cluster, it should be possible to use LogLog nodes locally:\neither in a separate process, or as a library. This should allow re-using\nall the functionality while avoiding extra network overheads.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustshop%2Floglog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frustshop%2Floglog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frustshop%2Floglog/lists"}