{"id":51029289,"url":"https://github.com/sumant1122/ringlog","last_synced_at":"2026-06-21T22:31:02.974Z","repository":{"id":359538171,"uuid":"1246543554","full_name":"sumant1122/ringlog","owner":"sumant1122","description":"A highly optimized, thread-per-core message broker built from scratch in Rust, utilizing io_uring for zero-copy, low-latency commit log streaming.","archived":false,"fork":false,"pushed_at":"2026-05-22T10:11:44.000Z","size":22,"stargazers_count":2,"open_issues_count":1,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-05-22T16:26:35.429Z","etag":null,"topics":["commit-log","high-performance","io-uring","low-latency","message-broker","rust","systems-programming","thread-per-core","write-ahead-log","zero-copy"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sumant1122.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-05-22T09:47:46.000Z","updated_at":"2026-05-22T16:21:57.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/sumant1122/ringlog","commit_stats":null,"previous_names":["sumant1122/ringlog"],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/sumant1122/ringlog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumant1122%2Fringlog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumant1122%2Fringlog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumant1122%2Fringlog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumant1122%2Fringlog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sumant1122","download_url":"https://codeload.github.com/sumant1122/ringlog/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sumant1122%2Fringlog/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34628453,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-21T02:00:05.568Z","response_time":54,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["commit-log","high-performance","io-uring","low-latency","message-broker","rust","systems-programming","thread-per-core","write-ahead-log","zero-copy"],"created_at":"2026-06-21T22:31:01.289Z","updated_at":"2026-06-21T22:31:02.967Z","avatar_url":"https://github.com/sumant1122.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# RingLog: Thread-Per-Core io_uring Message Broker in Rust\n\nA highly-optimized, single-node, mini-Kafka broker built from scratch in Rust. This system leverages a **thread-per-core** architecture powered by `io_uring` via the `tokio-uring` framework, bypassing traditional OS thread context switches and synchronization overhead.\n\n---\n\n## ⚡ Key Architectural Features\n\n1. **`io_uring` Asynchronous System Calls**: Uses direct Linux kernel ring buffers for both Disk WAL I/O (`File::write_at`/`File::read_at`) and network I/O (`TcpStream::read`/`TcpStream::write`).\n2. **Kernel-User Ownership Passing**: Implements strict data ownership transfer where memory buffers (`Vec\u003cu8\u003e`) are passed to the kernel and reaped asynchronously, achieving zero-copy properties.\n3. **Single-Thread Hot Path**: The server runs entirely inside a single CPU thread event loop. Cross-client synchronization and tail-log notifications are achieved using thread-local asynchronous coordination (`tokio::sync::watch` and `tokio::sync::Mutex`), avoiding standard thread-locking contention.\n4. **Append-Only Write-Ahead Log (WAL)**: Messages are sequentially packed with a 4-byte length header and flushed directly to disk.\n\n---\n\n## 📊 Wire Protocol Specifications\n\nThe broker communicates over TCP using a simplified binary framing protocol:\n\n### 1. Connection Handshake\nOn initial connection, a client must send a single byte designating its role:\n* **`0x01`**: Producer Client\n* **`0x02`**: Consumer Client\n\n### 2. Producer Protocol\nOnce handshaking as a Producer, the client continuously streams records:\n* **Request Format**: `[4-Byte Big-Endian Length] [Payload Bytes]`\n* **Response ACK**: The server appends the record to the WAL and immediately returns an `[8-Byte Big-Endian Physical Offset]` representing the exact byte location of the message.\n\n### 3. Consumer Protocol\nOnce handshaking as a Consumer, the client requests a stream of records starting from a specific point:\n* **Request Format**: `[8-Byte Big-Endian Starting Offset]`\n* **Response Stream**: The server tails the WAL and streams back matches: `[4-Byte Big-Endian Length] [Payload Bytes]`. When it reaches the end of the log, it yields and awaits notifications of new writes.\n\n---\n\n## 🛠️ Prerequisites\n\n* **OS**: Linux (Kernel **5.1** or newer is required for basic `io_uring` support, **5.6+** is highly recommended for full socket support).\n* **Rust**: Modern stable toolchain (`edition = \"2021\"`).\n\n---\n\n## 🚀 How to Run\n\nThe single binary supports four distinct operational modes via command-line arguments.\n\n### 📦 Compiling and Running the Release Binary (Recommended)\nTo build the highly-optimized release binary:\n```bash\ncargo build --release\n```\nThis produces a standalone executable at `./target/release/ringlog`. You can copy this binary and run it anywhere on a compatible Linux environment:\n```bash\n# To run the self-contained simulation demo:\n./target/release/ringlog\n\n# To run the standalone broker server:\n./target/release/ringlog server\n\n# To run the interactive producer client:\n./target/release/ringlog producer\n\n# To run the real-time consumer client starting from physical offset 0:\n./target/release/ringlog consumer 0\n```\n\n---\n\n### 🛠️ Running with Cargo (Development)\n\nFor quick development and testing, you can use cargo to compile and run:\n\n### 1. Run the Real-Time Simulation Demo\nThis runs a fully self-contained simulation of the broker, producer, and consumer all running concurrently inside the same thread-local event loop:\n\n```bash\ncargo run\n```\n\n### 2. Run the Dedicated Broker Server\nTo start the standalone broker server listening on port `12000`:\n\n```bash\ncargo run -- server\n```\n*Creates/appends to a persistent file `commit.log` in your working directory, automatically preserving/restoring the write offsets.*\n\n### 3. Run the Interactive TCP Producer Client\nIn a new terminal window, connect to a running broker server and dynamically type messages to publish them to the WAL:\n\n```bash\ncargo run -- producer\n```\n*Simply type your message in the console and hit Enter. The client will print the assigned physical byte offset ACK returned from the server.*\n\n### 4. Run the Real-Time TCP Consumer Client\nIn a new terminal window, connect to a running broker server to stream all historical and newly landing messages:\n\n```bash\ncargo run -- consumer [starting_offset]\n```\n*If `starting_offset` is omitted, it defaults to `0` (replaying the entire log).*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumant1122%2Fringlog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsumant1122%2Fringlog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsumant1122%2Fringlog/lists"}