{"id":18976307,"url":"https://github.com/quickwit-oss/mrecordlog","last_synced_at":"2025-04-19T17:12:57.029Z","repository":{"id":61618767,"uuid":"536871914","full_name":"quickwit-oss/mrecordlog","owner":"quickwit-oss","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-30T11:16:17.000Z","size":222,"stargazers_count":25,"open_issues_count":9,"forks_count":7,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-04-16T12:18:45.618Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/quickwit-oss.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2022-09-15T05:06:54.000Z","updated_at":"2025-03-17T14:09:28.000Z","dependencies_parsed_at":"2025-02-03T19:23:02.971Z","dependency_job_id":"19598adc-4d87-45ed-8360-ed7d2530d176","html_url":"https://github.com/quickwit-oss/mrecordlog","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Fmrecordlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Fmrecordlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Fmrecordlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/quickwit-oss%2Fmrecordlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/quickwit-oss","download_url":"https://codeload.github.com/quickwit-oss/mrecordlog/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249746042,"owners_count":21319581,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T15:23:31.095Z","updated_at":"2025-04-19T17:12:57.005Z","avatar_url":"https://github.com/quickwit-oss.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# What is it?\n\nThis crate implements a solution to efficiently handle several record logs.\nEach recordlog has its own \"local\" notion of position.\nIt is possible to truncate each of the queues individually.\n\n# Goals\n\n- be durable, offer some flexibility on `fsync` strategies.\n- offer a way to truncate a queue after a specific position\n- handle an arbitrary number of queues\n- have limited IO\n- be fast\n- offer the possibility to implement push back\n\n```rust\npub struct MultiRecordLog {\n    pub fn create_queue(\u0026mut self, queue: \u0026str) -\u003e Result\u003c(), CreateQueueError\u003e;\n    pub fn delete_queue(\u0026mut self, queue: \u0026str) -\u003e Result\u003c(), DeleteQueueError\u003e;\n    pub fn queue_exists(\u0026self, queue: \u0026str) -\u003e bool;\n    pub fn list_queues(\u0026self) -\u003e impl Iterator\u003cItem = \u0026str\u003e {\n    pub fn append_record(\n        \u0026mut self,\n        queue: \u0026str,\n        position_opt: Option\u003cu64\u003e,\n        payload: \u0026[u8],\n    );\n    pub fn truncate(\u0026mut self, queue: \u0026str, position: u64) -\u003e Result\u003c(), TruncateError\u003e;\n    pub fn range\u003cR\u003e(\n        \u0026self,\n        queue: \u0026str,\n        range: R,\n    ) -\u003e Option\u003cimpl Iterator\u003cItem = (u64, \u0026[u8])\u003e + '_\u003e;\n}\n```\n\n# Non-goals\n\nThis is not Kafka. This recordlog is designed for a \"small amount of data\".\nAll retained data can fit in RAM.\n\nIn the context of Quickwit, this queue is used in the ingest API and is meant to contain\n1 minute worth of data. (At 60MB/s, means 3.6 GB of RAM)\n\nReading the recordlog files only happens on startup.\nHigh-performance when reading the recordlog files is not a goal.\nWriting fast on the other hand is important.\n\n# Implementation details.\n\n`mrecordlog` is multiplexing several independent queues into the same record log.\nThis approach has the merit of limiting the number of file descriptors necessary,\nand more importantly, to limit the number of `fsync`.\n\nIt also offers the possibility to truncate the queue for a given record log.\nThe actual deletion of the data happens when a file only contains deleted records.\nThen, and only then, the entire file is deleted.\n\nThat recordlog emits a new file every 1GB.\nA recordlog file is deleted once all queues have been truncated after the\nlast record of a  of a file.\n\nThere is no compaction logic.\n\n# TODO\n\n- add backpressure.\n- add fsync policy\n- better testing.\n- non auto-inc position\n- less Arc\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquickwit-oss%2Fmrecordlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fquickwit-oss%2Fmrecordlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fquickwit-oss%2Fmrecordlog/lists"}