{"id":13521172,"url":"https://github.com/earthstar-project/earthstar","last_synced_at":"2025-04-06T14:10:10.043Z","repository":{"id":39709335,"uuid":"266444815","full_name":"earthstar-project/earthstar","owner":"earthstar-project","description":"Storage for private, distributed, offline-first applications.","archived":false,"fork":false,"pushed_at":"2024-09-10T15:49:50.000Z","size":26729,"stargazers_count":638,"open_issues_count":48,"forks_count":20,"subscribers_count":20,"default_branch":"main","last_synced_at":"2024-10-29T15:21:48.568Z","etag":null,"topics":["browser","deno","earthstar","eventual-consistency","local-first","offline-first","p2p","sync","willow"],"latest_commit_sha":null,"homepage":"https://earthstar-project.org","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"lgpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/earthstar-project.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["sgwilym"],"open_collective":"earthstar"}},"created_at":"2020-05-24T00:49:12.000Z","updated_at":"2024-10-22T21:56:45.000Z","dependencies_parsed_at":"2022-07-13T12:10:49.521Z","dependency_job_id":"729cef74-b824-4fc8-b7e1-d58b347ec7f4","html_url":"https://github.com/earthstar-project/earthstar","commit_stats":{"total_commits":1156,"total_committers":13,"mean_commits":88.92307692307692,"dds":"0.40830449826989623","last_synced_commit":"3694adbdfd0fcb5978be7b4b71ce865d2ace42ce"},"previous_names":["cinnamon-bun/keywing"],"tags_count":129,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fearthstar","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fearthstar/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fearthstar/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/earthstar-project%2Fearthstar/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/earthstar-project","download_url":"https://codeload.github.com/earthstar-project/earthstar/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247492513,"owners_count":20947544,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["browser","deno","earthstar","eventual-consistency","local-first","offline-first","p2p","sync","willow"],"created_at":"2024-08-01T06:00:29.840Z","updated_at":"2025-04-06T14:10:10.027Z","avatar_url":"https://github.com/earthstar-project.png","language":"TypeScript","funding_links":["https://github.com/sponsors/sgwilym","https://opencollective.com/earthstar"],"categories":["Protocols and Specifications","TypeScript","deno","browser","Protocols and Technologies"],"sub_categories":["General","Data"],"readme":"# Earthstar\n\n\u003e v11 using [Willow](https://willowprotocol.org) is now available in\n\u003e [beta](https://jsr.io/@earthstar/earthstar/versions)!\n\u003e\n\u003e Development is happening on the\n\u003e [`willow branch`](https://github.com/earthstar-project/earthstar/tree/willow).\n\n[Earthstar](https://earthstar-project.org) is a small and resilient distributed\nstorage protocol designed with a strong focus on simplicity and versatility,\nwith the social realities of peer-to-peer computing kept in mind.\n\nThis is a reference implementation written in Typescript. You can use it to add\nEarthstar functionality to applications running on servers, browsers, the\ncommand line, or anywhere else JavaScript can be run.\n\n[Detailed API documentation for this module can be found here](https://doc.deno.land/https://deno.land/x/earthstar@v10.0.1/mod.ts).\n\nThis document is concerned with the usage of this module's APIs. To learn more\nabout what Earthstar is, please see these links:\n\n- [What is Earthstar?](https://earthstar-project.org/docs/what-is-it)\n- [How does Earthstar work?](https://earthstar-project.org/docs/how-it-works)\n\nTo learn more about running Earthstar servers, see\n[README_SERVERS](README_SERVERS.md)\n\nTo learn more about this codebase, please see [ARCHITECTURE](ARCHITECTURE.md).\n\nTo learn about contributing to this codebase, please see\n[CONTRIBUTING](CONTRIBUTING.md).\n\n## Table of contents\n\n## Importing the module\n\nIt can be imported via URL into a browser:\n\n```html\n\u003cscript type=\"module\"\u003e\n  import * as Earthstar from \"https://cdn.earthstar-project.org/js/earthstar.web.v10.0.1.js\";\n\u003c/script\u003e\n```\n\nOr Deno:\n\n```ts\nimport * as Earthstar from \"https://deno.land/x/earthstar/mod.ts\";\n```\n\n\u003e Earthstar's web syncing does not work with version of Deno between 1.28.0 -\n\u003e 1.29.3 (inclusive) due to a regression in these versions' WebSocket\n\u003e implementation. **Use Deno 1.30.0. or later**.\n\nor installed with NPM:\n\n```bash\n{`npm install earthstar`}\n```\n\n```ts\nimport * as Earthstar from \"earthstar\";\n\n// Node and browser APIs are namespaced in the NPM distribution:\nimport { ReplicaDriverWeb } from \"earthstar/browser\";\nimport { ReplicaDriverSqlite } from \"earthstar/node\";\n```\n\nWe recommend the browser and Deno versions. This module has been built with many\nstandard web APIs that have need to be polyfilled to work in Node.\n\n## Instantiating a replica\n\n`Replica` is the central API of this module. It is used to write and read data\nto a locally persisted copy of a share's data, and much more besides.\n\nTo instantiate a replica, you will need knowledge of a share's public address.\n\n```ts\nimport { Replica, ReplicaDriverMemory } from \"earthstar\";\n\nconst replica = new Replica({\n  driver: ReplicaDriverMemory(YOUR_SHARE_ADDRESS),\n  shareSecret: YOUR_SHARE_SECRET,\n});\n```\n\nThe `shareSecret` property is optional. If we omit it, the replica will be\nread-only.\n\n### Generating share keypairs\n\nYou can create new shares whenever you want.\n\n```ts\nimport { Crypto } from \"earthstar\";\n\nconst shareKeypair = await Crypto.generateShareKeypair(\"gardening\");\n```\n\nThe result of this operation will either be a `ShareKeypair` object with\n`shareAddress` and `secret` properties, or a `ValidationError`.\n\n### Persisting data with drivers\n\n`Replica` must always be instantiated with a driver. These drivers tell the\n`Replica` how to store and retrieve data, with different drivers using different\nstorage mechanisms.\n\nHere are the available drivers:\n\n- `ReplicaDriverMemory` (works in all environments, but only stores data in\n  memory)\n- `ReplicaDriverWeb` (works in the browser, stores data with IndexedDB)\n- `ReplicaDriverFs` (works on runtimes with filesystem access, stores data with\n  Sqlite and the filesystem)\n\nDrivers are made of two sub-drivers: one for documents, and one for attachments\n(arbitrary binary data).\n\nThere are some extra document drivers not used in the default drivers:\n\n- `DocDriverLocalStorage` (works in runtimes supporting the WebStorage API)\n- `DocDriverSqliteFFI` (works in Deno, stores data with a FFI implementation of\n  Sqlite, requires using the `--unstable` flag, and is faster than the default\n  driver in `ReplicaDriverFs`)\n\nThese document drivers can be used like this:\n\n```ts\nconst driver: IReplicaDriver = {\n  docDriver: new DocDriverSqliteFfi(SHARE_ADDR, FS_PATH),\n  attachmentDriver: new AttachmentDriverFs(FS_ATTACHMENTS_PATH),\n};\n\nconst replica = new Replica({ driver });\n```\n\n## Writing data\n\nWriting data requires two things:\n\n- A replica configured with a valid share secret\n- An author keypair\n\nAuthor keypairs can be generated like this:\n\n```ts\nimport { Crypto } from \"earthstar\";\n\nconst authorKeypair = await Crypto.generateAuthorKeypair(\"suzy\");\n```\n\nThe result will be a new `AuthorKeypair` object with `address` and `secret`\nproperties, or a `ValidationError`.\n\nWith a valid author keypair you can write data using `Replica.set`:\n\n```ts\nconst setResult = await replica.set(authorKeypair, {\n  path: \"/my-note\",\n  text: \"Saw seven magpies today\",\n});\n```\n\nThe result of this operation is either an `IngestEvent` describing the\noperation's success (or failure, if one of the parameters was invalid in some\nway).\n\n## Wiping data\n\nOnce written, data can be removed by overwriting it:\n\n```ts\nawait replica.set(authorKeypair, {\n  path: \"/my-note\",\n  text: \"\",\n});\n```\n\nOr with the convenience method:\n\n```ts\nawait replica.wipeDocAtPath(authorKeypair, \"/my-note\");\n```\n\n### Creating ephemeral documents\n\nThere is another way to remove written data without leaving any trace of it.\n\nEphemeral documents are held by replicas until a specified time, until at which\npoint they are deleted.\n\n```ts\nawait replica.set(authorKeypair, {\n  path: \"/my-temporary-note!\",\n  text: \"I accidentally stepped on the strawberries.\",\n  deleteAfter: TIME_IN_MICROSECONDS,\n});\n```\n\nTo set an ephemeral document, the path _must_ contain a `!`, and the\n`deleteAfter` property must be set with a timestamp in _microseconds_.\n\n## Querying data\n\nThere are many ways to get data back out of a Replica. The simplest one is\n`Replica.getAllDocs`:\n\n```ts\nconst everything = await replica.getAllDocs();\n```\n\nThe most powerful is `Replica.queryDocs`:\n\n```ts\nconst mostRecentlyEditedWikiPageDocs = await replica.queryDocs({\n  historyMode: \"latest\",\n  filter: {\n    pathStartsWith: \"/wiki\",\n  },\n  limit: 10,\n});\n```\n\nHere are all the querying methods on `Replica`:\n\n- `getAllDocs`\n- `getLatestDocs`\n- `getAllDocsAtPath`\n- `getLatestDocAtPath`\n- `queryDocs`\n- `queryPaths`\n- `queryAuthors`\n\nDetailed API documentation for all of them can be found\n[here](https://doc.deno.land/https://deno.land/x/earthstar/mod.ts).\n\n## Using document contents\n\nThe documents returned by queries are plain objects with the following shape:\n\n```ts\ntype Doc = {\n  /** Which document format the doc adheres to, e.g. `es.5`. */\n  format: \"es.5\";\n  author: AuthorAddress;\n  text: string;\n  textHash: string;\n  /** When the document should be deleted, as a UNIX timestamp in microseconds. */\n  deleteAfter?: number;\n  path: Path;\n  /** Used to verify the authorship of the document. */\n  signature: Signature;\n  /** Used to verify the author knows the share's secret */\n  shareSignature: Signature;\n  /** When the document was written, as a UNIX timestamp in microseconds (millionths of a second, e.g. `Date.now() * 1000`).*/\n  timestamp: Timestamp;\n  /** The share this document is from. */\n  share: ShareAddress;\n  /** The size of the associated attachment in bytes, if any. */\n  attachmentSize?: number;\n  /** The sha256 hash of the associated attachment, if any. */\n  attachmentHash?: string;\n};\n```\n\nThough most applications will probably only use the `author`, `text`, and\n`timestamp` properties.\n\n## Syncing with other peers\n\nSyncing data with other peers requires adding your replica(s) to an instance of\n`Peer`:\n\n```ts\nimport { Peer } from \"earthstar\";\n\nconst peer = new Peer();\n\n// Pretend myReplica is an instance of `Replica`\npeer.addReplica(myReplica);\n\npeer.sync(\"https://my.server\");\n```\n\n`Peer.sync` can be passed another instance of `Peer` or a valid URL of an\nEarthstar server to sync with.\n\nThe two peers will only sync the replicas with shares they have in common.\n\nThe result of `Peer.sync` can be assigned and used to monitor the progress of\nthe sync operation:\n\n```ts\nconst syncer = peer.sync(\"https://my.server\");\n\nsyncer.onStatusChange((newStatus) =\u003e {\n  console.log(newStatus);\n});\n\nsyncer.isDone().then(() =\u003e {\n  console.log(\"Sync complete\");\n}).catch((err) =\u003e {\n  console.error(\"Sync failed\", err);\n});\n```\n\n## Using document attachments\n\nDocuments can be written along with some arbitrary data which is persisted as an\n'attachment'. Whereas a document's `text` field can hold a UTF-8 string of 8kb,\nattachments can be of any kind of data and of any size.\n\n```ts\n// Here we use Deno.readFile to get a file's contents as a Uint8Array\nconst imageData = await Deno.readFile(\"/Desktop/leaf.jpg\");\n\nawait replica.set(authorKeypair, {\n  path: \"/images/pear-leaf.jpg\",\n  text: \"A close-up of a leaf of a pear tree\",\n  attachment: imageData,\n});\n```\n\nThe path _must_ have a file extension e.g. `.jpg`, `.mp3` if it also has an\nattachment.\n\nIf we were attaching a large amount of data, we would use a `ReadableStream`\ninstead:\n\n```ts\n// Here we use Deno.readFile to get a file's contents as a ReadableStream\u003cUint8Array\u003e\nconst videoFile = await Deno.open(\"/Desktop/little-mole.mp4\");\n\nawait replica.set(authorKeypair, {\n  path: \"/videos/little-mole.mp4\",\n  text: \"A close-up of a leaf of a pear tree\",\n  attachment: videoFile.readable,\n});\n```\n\n### Retrieving attachments\n\nIf you already have a document with an attachment, you can use\n`Replica.getAttachment`:\n\n```ts\nconst attachment = await replica.getAttachment(docWithAttachment);\n```\n\nThe result of this operation will be a `DocAttachment` with `getBytes` and\n`getStream` methods, undefined (if our replica has not received a copy of this\nattachment from other peers), or a `ValidationError` in case `getAttachment` was\npassed a document which can't have an attachment.\n\nIt's also possible to add attachments to many documents at once:\n\n```ts\nconst allDocs = await replica.getAllDocs();\n\nconst allDocsWithAttachments = await replica.addAttachments(allDocs);\n```\n\n`allDocsWithAttachments` will be an array of all documents with an added\n`attachment` property. The type of this property will either be `DocAttachment`,\n`undefined`, or `ValidationError`.\n\n## Discovering other peers on the local network\n\nEarthstar is able to automatically discover other Earthstar peers on the same\nnetwork, and in turn advertise itself to others.\n\n```ts\nconst discoveryLan = new DiscoveryLAN({ name: \"My Computer\" });\n\nfor await (const discoveryEvent of peer.discover(discoveryLan)) {\n  if (discoveryEvent.kind === \"PEER DISCOVERED\") {\n    discoveryEvent.sync();\n  }\n}\n```\n\n\u003e On Node, the `DiscoveryLAN` API is exported from \"earthstar/node\".\n\u003e\n\u003e On Deno, this feature uses unstable APIs and is not included with the default\n\u003e module exports. It can be imported from\n\u003e `https://deno.land/x/earthstar/src/discovery/discovery_lan.ts`. Usage also\n\u003e requires the `--unstable` flag.\n\n## Subscribing to replica changes\n\nThere are many ways to subscribe to the many events a replica generates during\nits lifetime.\n\nIf you want to subscribe to updates in order to update a UI, the most ergonomic\nAPI is `ReplicaCache`:\n\n```ts\nimport { ReplicaCache } from \"earthstar\";\n\nconst replicaCache = new ReplicaCache(myReplica);\n\nconst allDocs = replicaCache.getAllDocs();\n```\n\nThe caveat is that the first time a query method is called, it _always_ returns\nan empty result. To get new updates, you must subscribe to changes. In the\nfollowing example, we build a UI with a fictitious `renderDocListUI` function\nwith the results of `ReplicaCache.getAllDocs`:\n\n```ts\nfunction triggerUIRender() {\n  const allDocs = replicaCache.getAllDocs();\n\n  renderDocListUI(allDocs);\n}\n\nreplicaCache.onCacheUpdated(() =\u003e {\n  triggerUIRender();\n});\n\ntriggerUIRender();\n```\n\nThe important thing to remember is that the callback to `onCacheUpdated` will\nnever trigger until the cache has been queried at least once.\n\nIf you're not tracking changes for a UI, `Replica.getQueryStream` returns a\n`ReadableStream` of `QuerySourceEvent`, for documents matching a specific query.\nThis API is great for creating indexes.\n\n```ts\n// Create a query stream for all docs with paths starting with /chat\n// And include all existing documents and all newly created or synced documents\nconst chatMessagesStream = replica.getQueryStream({\n  filter: { pathStartsWith: \"/chat\" },\n}, \"everything\");\n\nchatMessagesStream.pipeTo(\n  new WritableStream({\n    write(event) {\n      if (event.kind === \"success\" || event.kind === \"existing\") {\n        console.log(event.doc.text);\n      }\n    },\n  }),\n);\n```\n\nFinally, `Replica.getEventStream` returns a `ReadableStream` of `ReplicaEvent`,\nwhich includes events for new document ingestions, document expirations,\nattachment ingestions, attachment prunes, and events for when the replica is\nabout to close (and has closed).\n\n## Using common settings between clients\n\nThere are a number of configurations which most Earthstar applications will want\nto persist between runs:\n\n- The shares used\n- An author to using for signing documents\n- Servers to sync with\n\nEarthstar offers a `SharedSettings` API which persists settings for these\nbetween sessions in all runtimes supporting the WebStorage APIs:\n\n```ts\nconst settings = new SharedSettings();\n\nsettings.author = authorKeypair;\nsettings.addShare(myShareAddress);\nconsole.log(settings.servers);\n```\n\nIt also offers a method which instantiates a new `Peer` with replicas for all\nshares already added:\n\n```ts\nconst settings = new SharedSettings();\n\n// Create a peer with all saved shares and sync once with all saved servers.\nconst peer = settings.getPeer({\n  sync: \"once\",\n  onCreateReplica: (addr, secret) =\u003e {\n    return new Replica({\n      driver: new ReplicaDriverMemory(addr),\n      shareSecret: secret,\n    });\n  },\n});\n```\n\n## Checking for errors\n\nMany functions in Earthstar return errors like `ValidationError`. These errors\nare not thrown, so it's good to check for them:\n\n```ts\nimport { isErr } from \"earthstar\";\n\nconst result = replica.set(authorKeypair, {\n  path: \"/hey\",\n  text: \"Hello\",\n});\n\nif (isErr(result)) {\n  console.error(\n    \"Something went wrong when you tried to write some data!\",\n    result,\n  );\n}\n```\n\n## Changing the cryptographic driver\n\nThe Deno and browser versions of this module are configured by default to use\nthe fastest cryptographic libraries available to them. The Node version is not,\nhowever.\n\n```ts\nimport { setGlobalCryptoDriver } from \"earthstar\";\nimport { CryptoDriverChloride } from \"earthstar/node\";\n\nsetGlobalCryptoDriver(CryptoDriverChloride);\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearthstar-project%2Fearthstar","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fearthstar-project%2Fearthstar","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fearthstar-project%2Fearthstar/lists"}