{"id":28396218,"url":"https://github.com/ssbc/async-append-only-log","last_synced_at":"2025-07-21T13:32:28.462Z","repository":{"id":37902253,"uuid":"296949940","full_name":"ssbc/async-append-only-log","owner":"ssbc","description":"A new append-only-log for SSB purposes","archived":false,"fork":false,"pushed_at":"2023-03-15T08:53:52.000Z","size":275,"stargazers_count":16,"open_issues_count":7,"forks_count":6,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-07-03T21:32:45.199Z","etag":null,"topics":["ssb-db2"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ssbc.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSES/CC0-1.0.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2020-09-19T20:53:21.000Z","updated_at":"2024-05-30T20:13:41.000Z","dependencies_parsed_at":"2024-03-17T02:05:41.795Z","dependency_job_id":"232fe91e-fecd-4ea2-a1d4-e69bac46f610","html_url":"https://github.com/ssbc/async-append-only-log","commit_stats":{"total_commits":266,"total_committers":5,"mean_commits":53.2,"dds":"0.43609022556390975","last_synced_commit":"0a97b8966441f922dd36d75ad676121e28cb6998"},"previous_names":["ssb-ngi-pointer/async-append-only-log","flumedb/async-flumelog"],"tags_count":30,"template":false,"template_full_name":null,"purl":"pkg:github/ssbc/async-append-only-log","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fasync-append-only-log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fasync-append-only-log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fasync-append-only-log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fasync-append-only-log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ssbc","download_url":"https://codeload.github.com/ssbc/async-append-only-log/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ssbc%2Fasync-append-only-log/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":264230184,"owners_count":23576383,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ssb-db2"],"created_at":"2025-05-31T21:37:26.333Z","updated_at":"2025-07-21T13:32:28.417Z","avatar_url":"https://github.com/ssbc.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003c!--\nSPDX-FileCopyrightText: 2021 Anders Rune Jensen\n\nSPDX-License-Identifier: CC0-1.0\n--\u003e\n\n# Async append only log\n\nThis module is heavily inspired by [flumelog-aligned-offset]. It is an\nattempt to implement the same concept but in a simpler fashion, making\nit easier to reason about the code. A log is the lowest part of the\nSSB stack, so it should extremly stable while still maintaining good\nperformance.\n\nA log consists of a number of `blocks`, that contain a number of\n`record`s. A `record` is simply it's `length`, as a 16-bit unsigned\ninteger, followed by the `data` bytes. A record must be in one and\nonly one block, which means there probably will be some empty space at\nthe end of a block. Blocks are always written in full.\n\n```\n\u003cblock\u003e\n  \u003crecord\n    \u003clength: UInt16LE\u003e\n    \u003cdata: Bytes\u003e\n  \u003c/record\u003e*\n\u003c/block\u003e*\n```\n\nIn contrast to flumelog-aligned-offset there is no additional `length`\nafter the `data` in a `record` and no pointer at the end of a\n`block`. These were there to be able to iterate over the log in\nreverse, but I have never seen the need for that.\n\nWriting to the log is always async. Note this is different from\n[flumelog-offset] and [flumelog-aligned-offset]. The `since`\nobservable will be updated once the data is written. The `onDrain`\ncallback can be used to know when data has been written if\nneeded. Streaming will only emit values that have been written to\nstorage. This is to ensure that a view will never get ahead of the\nmain log and thus end up in a bad state if the system crashes before\ndata is written. `get` will return values that have not been written\nto disk yet.\n\nThis module is not compatible with flume without a wrapper around\nstream as it uses the same terminology as [JITDB] and [ssb-db2] of\nusing offset for the byte position of a record instead of seq.\n\n## API\n\n### Open the log\n\n```js\nconst OffsetLog = require('async-append-only-log')\n\nconst log = OffsetLog('/path/to/log.file', {\n  /**\n   * Size of the block, in bytes.\n   *\n   * DEFAULT: 65536\n   */\n  blockSize: 1024,\n\n  /**\n   * Conversion layer as an object of the shape `{encode, decode}`,\n   * where `encode` defines a function (item)=\u003ebuffer when writing to disk\n   * and `decode` defines a function (buffer)=\u003eitem, where `item` is what\n   * you will directly interact with using async-append-only-log's APIs.\n   * For JSON, use `flumecodec/json`.\n   *\n   * DEFAULT: `{encode: x =\u003e x, decode: x =\u003e x}`\n   */\n  codec: { encode, decode },\n\n  /**\n   * Amount of time to wait between writes, in milliseconds.\n   *\n   * DEFAULT: 250\n   */\n  writeTimeout: 100,\n\n  /**\n   * A function that takes a record's buffer and should return a boolean\n   * indicating whether the record is \"valid\". Implement this to ensure the\n   * record is not corrupted. When the log is loaded, all records in the latest\n   * block will be checked using this.\n   *\n   * DEFAULT: (recordBuffer) =\u003e true\n   */\n  validateRecord: (recordBuffer) =\u003e {\n    // ...\n  },\n})\n```\n\n### Write a single record\n\n```js\nlog.append(item, (err, offset) =\u003e {\n  // ...\n})\n```\n\n### Write several records\n\n```js\nlog.append([item1, item2, item3], (err, offset3) =\u003e {\n  // ...\n})\n```\n\n### Write several records, either all fail or all succeed\n\n```js\nlog.appendTransaction([item1, item2, item3], (err, offset3) =\u003e {\n  // ...\n})\n```\n\n### Wait for all ongoing appends to be flushed to disk\n\n```js\nlog.onDrain(() =\u003e {\n  // ...\n})\n```\n\n### Scan all records as a `push-stream`\n\n```js\nlog.stream(opts).pipe(sink)\n```\n\nWhere\n\n```js\nopts = { live, offsets, values, limit, gte, gt }\n```\n\n- `live` is a boolean indicating that you're interested only in records added\nafter streaming began. DEFAULT: `false`\n- `offsets` is a boolean indicating you're interested in knowing the offset for each record streamed to the sink. DEFAULT: `true`\n- `values` is a boolean indicating you're interested in getting the data buffer for each record streamed to the sink. DEFAULT: `true`\n- `limit` is a number indicating how many records you want from the stream, after which the stream will close. DEFAULT: `0` which **means unlimited**\n- `gte` and `gt` and other opts are specific to [ltgt]\n\n```js\nsink = { paused, write, end }\n```\n\n`sink` is from [push-stream]\n\n### Read a record\n\n```js\nlog.get(offset, (err, item) =\u003e {\n  // ...\n})\n```\n\n### Delete a record\n\nIn practice, this will just overwrite the record with zero bytes.\n\n```js\nlog.del(offset, (err) =\u003e {\n  // ...\n})\n```\n\n### Wait for all ongoing deletes to be flushed to disk\n\n```js\nlog.onDeletesFlushed(() =\u003e {\n  // ...\n})\n```\n\n### Keep track of the most recent record\n\nAs an [obz] observable:\n\n```js\nlog.since((offset) =\u003e {\n  // ...\n})\n```\n\n### Get statistics on deleted records\n\nAmong other things, this is useful for knowing how much storage space you could\nsave by running compaction, to eliminate deleted records.\n\n```js\nlog.stats((err, stats) =\u003e {\n  console.log(stats)\n  // { totalBytes, deletedBytes }\n})\n```\n\n### Compact the log (remove deleted records)\n\n```js\nlog.compact((err) =\u003e {\n  // This callback will be called once, when the compaction is done.\n})\n```\n\nNote, this functionality is currently not available when running in a\nbrowser.\n\n### Track progress of compactions\n\nAs an [obz] observable:\n\n```js\nlog.compactionProgress((progress) =\u003e {\n  console.log(progress)\n  // {\n  //   startOffset,\n  //   compactedOffset,\n  //   unshiftedOffset,\n  //   percent,\n  //   done,\n  //   sizeDiff,\n  //   holesFound,\n  // }\n})\n```\n\nWhere\n\n- `startOffset`: the starting point for compaction. All offsets smaller than\n  this have been left untouched by the compaction algorithm.\n- `compactedOffset`: all records up until this point have been compacted so far.\n- `unshiftedOffset`: offset for the first record that hasn't yet been \"moved\"\n  to previous slots. Tracking this allows you to see the algorithm proceeding.\n- `percent`: a number between 0 and 1 to indicate the progress of compaction.\n- `done`: a boolean indicating whether compaction is ongoing (`false`) or done\n  (`true`).\n- `sizeDiff`: number of bytes freed after compaction is finished. Only available\n  if `done` is `true`.\n- `holesFound`: number of deleted records that were found while compaction was\n  ongoing. Only available if `done` is `true`.\n\n### Close the log\n\n```js\nlog.close((err) =\u003e {\n  // ...\n})\n```\n\n## Benchmarks\n\nRunning [bench-flumelog] reveals the following numbers. Async flumelog\nis faster that regular flumelog-offset in all categories. The most\nimportant numbers are append (used for onboarding) and stream (used\nfor building indexes). Flumelog-aligned-offset is not included in the\nbenchmarks, as it writes every message synchronously rendering the\nresults invalid.\n\n```\n\nasync-append-only-log:\n\nname, ops/second, mb/second, ops, total-mb, seconds\nappend, 923964.807, 138.002, 4620748, 690.149, 5.001\nstream, 1059075.865, 158.182, 4620748, 690.149, 4.363\nstream no cache, 1102803.818, 164.713, 4620748, 690.149, 4.19\nstream10, 2540947.641, 379.51, 12714902, 1899.068, 5.004\nrandom, 39715.656, 5.931, 198618, 29.664, 5.001\n\nflumelog offset:\n\nname, ops/second, mb/second, ops, total-mb, seconds\nappend, 306180.037, 45.74, 3064556, 457.817, 10.009\nstream, 294511.348, 43.997, 2945408, 440.017, 10.001\nstream no cache, 327724.949, 48.959, 3064556, 457.817, 9.351\nstream10, 452973.302, 67.67, 4530186, 676.776, 10.001\nrandom, 28774.712, 4.298, 287891, 43.008, 10.005\n\n```\n\nTo run the benchmarks the small `bench-flumelog.patch` needs to be\napplied.\n\n[JITDB] results for more real world benchmarks are available as [jitdb-results].\n\n[push-stream]: https://github.com/push-stream/push-stream\n[flumelog-aligned-offset]: https://github.com/flumedb/flumelog-aligned-offset/\n[flumelog-offset]: https://github.com/flumedb/flumelog-offset/\n[bench-flumelog]: https://github.com/flumedb/bench-flumelog\n[jitdb]: https://github.com/ssb-ngi-pointer/jitdb/\n[ltgt]: https://github.com/dominictarr/ltgt\n[jitdb-results]: https://github.com/arj03/jitdb/blob/master/bench.txt\n[ssb-db2]: https://github.com/ssb-ngi-pointer/ssb-db2/\n[obz]: https://www.npmjs.com/package/obz\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbc%2Fasync-append-only-log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fssbc%2Fasync-append-only-log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fssbc%2Fasync-append-only-log/lists"}