{"id":20367541,"url":"https://github.com/earthstar-project/willow-js","last_synced_at":"2025-04-12T05:29:24.854Z","repository":{"id":171657866,"uuid":"616672122","full_name":"earthstar-project/willow-js","owner":"earthstar-project","description":"A Willow Data Model, Sideloading, and General Purpose Sync protocol implementation in TypeScript.","archived":false,"fork":false,"pushed_at":"2025-03-26T22:22:52.000Z","size":607,"stargazers_count":66,"open_issues_count":7,"forks_count":7,"subscribers_count":7,"default_branch":"main","last_synced_at":"2025-04-11T06:31:19.907Z","etag":null,"topics":["deno","eventually-consistent","local-first","sync","typescript","willow"],"latest_commit_sha":null,"homepage":"https://willowprotocol.org","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/earthstar-project.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["sgwilym"],"open_collective":"earthstar"}},"created_at":"2023-03-20T21:08:25.000Z","updated_at":"2025-03-26T22:22:57.000Z","dependencies_parsed_at":null,"dependency_job_id":"52ceff35-ce00-4fe7-b05c-65236074d893","html_url":"https://github.com/earthstar-project/willow-js","commit_stats":{"total_commits":185,"total_committers":3,"mean_commits":"61.666666666666664","dds":0.09189189189189184,"last_synced_commit":"55ad89f642ddf578378ee6669083517c0f7db1c9"},"previous_names":["earthstar-project/willow-js"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fwillow-js","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fwillow-js/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fwillow-js/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fwillow-js/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/earthstar-project","download_url":"https://codeload.github.com/earthstar-project/willow-js/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248522738,"owners_count":21118372,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["deno","eventually-consistent","local-first","sync","typescript","willow"],"created_at":"2024-11-15T00:33:07.733Z","updated_at":"2025-04-12T05:29:24.823Z","avatar_url":"https://github.com/earthstar-project.png","language":"TypeScript","funding_links":["https://github.com/sponsors/sgwilym","https://opencollective.com/earthstar"],"categories":[],"sub_categories":[],"readme":"# willow-js\n\nThis is a implementation of the [Willow protocol](https://willowprotocol.org)\nwritten in TypeScript.\n\n## API\n\nThis module is published on JSR. Please see the\n[@earthstar/willow page](https://jsr.io/@earthstar/willow) for installation\ninstructions.\n\nAPI documentation can be found [here](https://jsr.io/@earthstar/willow/doc).\n\n## Overview\n\n`willow-js` includes:\n\n- A `Store` compliant with the\n  [Willow Data model](https://willowprotocol.org/specs/data-model/index.html#data_model),\n- A `WgpsMessenger` which syncs data between two stores via the\n  [Willow General Purpose Sync Protocol](https://willowprotocol.org/specs/sync/index.html#sync),\n- and `createDrop` and `ingestDrop` compliant with the\n  [Willow Sideloading protocol](https://willowprotocol.org/specs/sideloading/index.html#sideloading).\n\nThis is a low-level module for people to build their own protocols on top of,\nlike [Earthstar](https://earthstar-project.org). It is an _extremely_ generic\nand modular codebase, with many parameters to configure.\n\n### Parameters\n\nThese parameters have been abstracted as various `Scheme` interfaces, e.g.\n`NamespaceScheme`, `SubspaceScheme`, `FingerprintScheme`. At\n`src/test/test_schemes.ts` you can find a full suite of schemes used to\nconfigure willow-js during tests. Use these to experiment with willow-js. When\nyou want to configure these parameter schemes yourself, please see each scheme's\naccompanying documentation.\n\nUsing `Store` requires these schemes:\n\n- `NamespaceScheme`\n- `SubspaceScheme`\n- `PayloadScheme`\n- `PathScheme`\n- `AuthorisationScheme`\n- `FingerprintScheme`\n\nUsing `WgpsMessenger` requires all prior schemes, as well as the following:\n\n- `AccessControlScheme`\n- `SubspaceCapScheme`\n- `PaiScheme`\n- `AuthorisationTokenScheme`\n\n### Interfaces\n\nIn addition to these parameters, it's possible to configure willow-js to use\ndifferent drivers for entry and payload storage, transports during sync, or even\ndifferent data structures to write and retrieve data from.\n\nMost of the interfaces are geared around changing the way `Store` works:\n\n- `KvDriver` - an interface for writing and reading data from a key value store.\n  We've chosen key-value stores as the lowest common denominator for data\n  storage, and these drivers can be used by many different client. See our\n  KvDriverMemory, KvDriverDeno, and KvDriverIndexedDB. This is the quickest way\n  to adding support for new runtimes to willow-js.\n- `SummarisableStorage` - a data structure capable of summarising ranges of\n  stored data as a `PreFingerprint` via a `LiftingMonoid`. See `Skiplist` for an\n  implementation which reads and writes data using given `KvDriver`.\n- `Storage3d` - a data structure to write and read entries from a 3d data\n  structure. See `TripleStore` for our only extant implementation of this\n  interface, which uses three differently ordered `SummarisableStorage`.\n- `EntryDriver` - An interface encompassing all of the above to be directly\n  provided to `Store`.\n- `PrefixIterator` - provides a means to tell if one path is prefixed by another\n  path, and store a bit of arbitrary data with it e.g. the timestamp. See\n  `RadixTree` for an example.\n\n`WgpsMessenger` can communicate with peers via different transports using the\n`Transport` interface. Currently we have transports for WebSockets and\nin-memory.\n\n## `willow-js`' Most Wanted\n\nHere is a list of features we want to implement in willow-js. We welcome all\ncontributions.\n\nThe number of 🌶 emoji next to each item indicates a scientifically measured\nestimated challenge level for each of these undertakings, three peppers being\nthe maximum.\n\n### Storage\n\n- 🌶 A LevelDB driver conforming to `KvDriver`\n- 🌶🌶 A payload driver which intelligently stores small payloads in a database,\n  and larger payloads on the filesystem, conforming to `PayloadDriver`\n- 🌶🌶🌶 A Z-ordered skiplist conforming to `Storage3d` (CLAIMED)\n\n### Sync\n\nWgpsMessenger is currently compliant with the WGPS. However there are a number\nof optional enhancements yet to be implemented:\n\n- 🌶 Make WgpsMessenger send payloads below `maximum_payload_size` during\n  reconciliation via `ReconciliationSendPayload`\n- 🌶 Make it possible to configure an upper byte length limit over which payloads\n  are not requested.\n- 🌶 Make the threshold at which 3d range-based reconciliation stop comparing\n  fingerprints and just return entries user-configurable.\n- 🌶 A WebRTC `Transport`.\n- 🌶🌶 Make WgpsMessenger track the progress of reconciliation using the `covers`\n  field of `ReconciliationSendFingerprint` and `ReconciliationAnnounceEntries`\n- 🌶🌶 Make `WgpsMessenger`'s resources user configurable (currently guarantees\n  effectively infinite memory to the other peer).\n- 🌶🌶 Add events to `WgpsMessenger` so that the progress of a sync session can be\n  tracked.\n- 🌶🌶 Make `WgpsMessenger` able to add and remove `ReadAuthorisation` during a\n  sync session.\n- 🌶🌶🌶 Post-reconciliation forwarding of messages using a push-lazy-push\n  multicast tree (plumtree), `DataSetMetadata`, and `DataSendEntry`.\n- 🌶🌶🌶 Make WgpsMessenger intelligently free handles no longer in use via\n  `ControlFree`\n\n## Dev\n\nDeno is used the development runtime. Run `deno task test` to run tests.\n\n---\n\nWant to follow along with development, ask questions, or get involved yourself?\nCome and join us on the\n[Earthstar Project Discord](https://discord.gg/6NtYzQC2G4).\n\n---\n\nThis project was funded through the NGI Assure Fund, a fund established by NLnet\nwith financial support from the European Commission’s Next Generation Internet\nprogramme, under the aegis of DG Communications Networks, Content and Technology\nunder grant agreement № 957073.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearthstar-project%2Fwillow-js","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fearthstar-project%2Fwillow-js","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearthstar-project%2Fwillow-js/lists"}