{"id":19441026,"url":"https://github.com/rosedblabs/wal","last_synced_at":"2025-05-16T02:08:27.453Z","repository":{"id":160783807,"uuid":"635597667","full_name":"rosedblabs/wal","owner":"rosedblabs","description":"Write Ahead Log for LSM or bitcask storage(or any append-only write).","archived":false,"fork":false,"pushed_at":"2025-01-26T13:36:50.000Z","size":107,"stargazers_count":257,"open_issues_count":2,"forks_count":45,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-08T12:12:25.791Z","etag":null,"topics":["bitcask","database","kv-store","lsm-tree","storage","write-ahead-log"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rosedblabs.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-05-03T03:24:01.000Z","updated_at":"2025-04-05T09:03:49.000Z","dependencies_parsed_at":null,"dependency_job_id":"c5f87024-5feb-4d76-9994-85f4d52f8cdf","html_url":"https://github.com/rosedblabs/wal","commit_stats":{"total_commits":94,"total_committers":11,"mean_commits":8.545454545454545,"dds":"0.19148936170212771","last_synced_commit":"d663aa64fb1209b394160514ab8fad6710f6ac97"},"previous_names":["rosedblabs/wal"],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosedblabs%2Fwal","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosedblabs%2Fwal/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosedblabs%2Fwal/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rosedblabs%2Fwal/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rosedblabs","download_url":"https://codeload.github.com/rosedblabs/wal/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254453655,"owners_count":22073617,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bitcask","database","kv-store","lsm-tree","storage","write-ahead-log"],"created_at":"2024-11-10T15:34:06.825Z","updated_at":"2025-05-16T02:08:27.379Z","avatar_url":"https://github.com/rosedblabs.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# wal\nWrite Ahead Log for LSM or bitcask storage.\n\n## Key Features\n* Disk based, support large data volume\n* Append only write, high performance\n* Fast read, one disk seek to retrieve any value\n* Support batch write, all data in a batch will be written in a single disk seek\n* Iterate all data in wal with `NewReader` function\n* Extremely fast read\n* Support concurrent write and read, all functions are thread safe\n\n## Design Overview\n\n![wal-logo.png](https://s2.loli.net/2025/01/12/SF9vThRkAObm4WD.png)\n\n## Format\n\n**Format of a single segment file:**\n\n```\n       +-----+-------------+--+----+----------+------+-- ... ----+\n File  | r0  |      r1     |P | r2 |    r3    |  r4  |           |\n       +-----+-------------+--+----+----------+------+-- ... ----+\n       |\u003c---- BlockSize -----\u003e|\u003c---- BlockSize -----\u003e|\n\n  rn = variable size records\n  P = Padding\n  BlockSize = 32KB\n```\n\n**Format of a single record:**\n\n```\n+----------+-------------+-----------+--- ... ---+\n| CRC (4B) | Length (2B) | Type (1B) |  Payload  |\n+----------+-------------+-----------+--- ... ---+\n\nCRC = 32-bit hash computed over the payload using CRC\nLength = Length of the payload data\nType = Type of record\n       (FullType, FirstType, MiddleType, LastType)\n       The type is used to group a bunch of records together to represent\n       blocks that are larger than BlockSize\nPayload = Byte stream as long as specified by the payload size\n```\n\n## Getting Started\n\n```go\nfunc main() {\n\twal, _ := wal.Open(wal.DefaultOptions)\n\t// write some data\n\tchunkPosition, _ := wal.Write([]byte(\"some data 1\"))\n\t// read by the position\n\tval, _ := wal.Read(chunkPosition)\n\tfmt.Println(string(val))\n\n\twal.Write([]byte(\"some data 2\"))\n\twal.Write([]byte(\"some data 3\"))\n\n\t// iterate all data in wal\n\treader := wal.NewReader()\n\tfor {\n\t\tval, pos, err := reader.Next()\n\t\tif err == io.EOF {\n\t\t\tbreak\n\t\t}\n\t\tfmt.Println(string(val))\n\t\tfmt.Println(pos) // get position of the data for next read\n\t}\n}\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frosedblabs%2Fwal","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frosedblabs%2Fwal","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frosedblabs%2Fwal/lists"}