{"id":18976314,"url":"https://github.com/quickwit-oss/recordlog","last_synced_at":"2025-07-05T09:05:55.559Z","repository":{"id":44796233,"uuid":"223956492","full_name":"quickwit-oss/recordlog","owner":"quickwit-oss","description":null,"archived":false,"fork":false,"pushed_at":"2022-09-12T14:15:18.000Z","size":95,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":12,"default_branch":"master","last_synced_at":"2025-01-01T09:25:51.820Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quickwit-oss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-11-25T13:34:27.000Z","updated_at":"2023-09-26T14:49:06.000Z","dependencies_parsed_at":"2023-01-18T05:00:38.925Z","dependency_job_id":null,"html_url":"https://github.com/quickwit-oss/recordlog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Frecordlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Frecordlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Frecordlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Frecordlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quickwit-oss","download_url":"https://codeload.github.com/quickwit-oss/recordlog/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239978057,"owners_count":19728271,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T15:23:32.164Z","updated_at":"2025-02-21T07:43:25.612Z","avatar_url":"https://github.com/quickwit-oss.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# What is it?\n\n\nThat record log uses log rotation.\n\nEach recordlog has its own \"local\" notion of position.\nIt is possible to truncate each of the queues individually.\n\n# Goals\n\n- be durable, offer some flexibility on `fsync` strategies.\n- offer a way to truncate a queue after a specific position\n- handle an arbitrary number of queues\n- have limited IO\n- be fast\n- offer the possibility to implement push back\n\n```rust\npub struct MultiRecordLog {\n    pub async fn open(directory_path: \u0026Path) -\u003e Result\u003cSelf, ReadRecordError\u003e;\n    pub async fn append_record(\u0026mut self,\n        queue: \u0026str,\n        local_position: Option\u003cPosition\u003e,\n        payload: \u0026[u8]) -\u003e Result\u003cOption\u003cPosition\u003e\u003e;\n    pub fn iter_from\u003c'a\u003e(\n        \u0026'a self,\n        queue: \u0026str,\n        position: Position,\n    ) -\u003e Option\u003cimpl Iterator\u003cItem = (Position, \u0026'a [u8])\u003e + 'a\u003e;\n    pub async fn truncate(\u0026mut self, queue: \u0026str, position: Position) -\u003e io::Result\u003c()\u003e\n}\n```\n# Non-goals\n\nThis is not Kafka. That recordlog is designed for a \"small amount of data\".\nAll retained data can fit in RAM.\n\nIn the context of quickwit, that queue is used in the PushAPI and is meant to contain\n1 worth of data. (At 60MB/s, means 3.6 GB of RAM)\n\n\n# Implementation details.\n\n`mrecordlog` is multiplexing several independent queues into the same record log.\nThis approach has the merit of limiting the number of file descriptor necessary,\nand more importantly, to limit the number of `fsync`.\n\nIt also offers the possibility to truncate the queue for a given record log.\nThe actual deletion of the data happens when a file only contains deleted records.\nThen, and only then, the entire file is deleted.\n\nThat recordlog emits a new file every 1GB.\nA recordlog file is deleted, once all queues have been truncated after the\nlast record of a  of a file.\n\n.\n\nThere are no compaction logic.\n\n\n\nTO BE DONE\n\nIt also makes it possible to know the size of the recordlog, to make backpressure possible.\n\n# Implementation\n\nThe implementation works by stacking different level of abstraction.\nAt the lowest level, the frame reader splits the data into frames.\nFrames have a fixed size and a checksum. The combination makes the recordlog resilient to corruption.\n\nThe record reader implements a protocol to build records over the frame reader.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquickwit-oss%2Frecordlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquickwit-oss%2Frecordlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquickwit-oss%2Frecordlog/lists"}