{"id":13486473,"url":"https://github.com/holepunchto/hypercore","last_synced_at":"2025-05-12T15:34:09.061Z","repository":{"id":43896568,"uuid":"48335985","full_name":"holepunchto/hypercore","owner":"holepunchto","description":"Hypercore is a secure, distributed append-only log.","archived":false,"fork":false,"pushed_at":"2025-05-12T09:34:36.000Z","size":10931,"stargazers_count":2663,"open_issues_count":26,"forks_count":184,"subscribers_count":63,"default_branch":"main","last_synced_at":"2025-05-12T10:45:05.401Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://docs.holepunch.to","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/holepunchto.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2015-12-20T20:24:46.000Z","updated_at":"2025-05-12T09:34:38.000Z","dependencies_parsed_at":"2023-02-13T23:30:30.743Z","dependency_job_id":"6a57d9f2-a980-461a-bf71-a33fd0533067","html_url":"https://github.com/holepunchto/hypercore","commit_stats":{"total_commits":1216,"total_committers":55,"mean_commits":22.10909090909091,"dds":"0.17927631578947367","last_synced_commit":"666bc6c49ce6528c19288fae3597fb2ecea55469"},"previous_names":["mafintosh/hypercore","hypercore-protocol/hypercore"],"tags_count":565,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holepunchto%2Fhypercore","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holepunchto%2Fhypercore/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holepunchto%2Fhypercore/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/holepunchto%2Fhypercore/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/holepunchto","download_url":"https://codeload.github.com/holepunchto/hypercore/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253725915,"owners_count":21954089,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T18:00:46.828Z","updated_at":"2025-05-12T15:34:09.033Z","avatar_url":"https://github.com/holepunchto.png","language":"JavaScript","funding_links":[],"categories":["Applications","JavaScript","[🎓 research](https://github.com/stars/ketsapiwiq/lists/research)"],"sub_categories":[],"readme":"# Hypercore\n\n### [See the full API docs at docs.pears.com](https://docs.pears.com/building-blocks/hypercore)\n\nHypercore is a secure, distributed append-only log.\n\nBuilt for sharing large datasets and streams of real time data\n\n## Features\n\n* **Sparse replication.** Only download the data you are interested in.\n* **Realtime.** Get the latest updates to the log fast and securely.\n* **Performant.** Uses a simple flat file structure to maximize I/O performance.\n* **Secure.** Uses signed merkle trees to verify log integrity in real time.\n* **Modular.** Hypercore aims to do one thing and one thing well - distributing a stream of data.\n\nNote that the latest release is Hypercore 10, which adds support for truncate and many other things.\nVersion 10 is not compatible with earlier versions (9 and earlier), but is considered LTS, meaning the storage format and wire protocol is forward compatible with future versions.\n\n## Install\n\n```sh\nnpm install hypercore\n```\n\n\u003e [!NOTE]\n\u003e This readme reflects Hypercore 11, our latest major version that is backed by RocksDB for storage and atomicity.\n\u003e Whilst we are fully validating that, the npm dist-tag for latest is set to latest version of Hypercore 10, the previous major, to avoid too much disruption.\n\u003e It will be updated to 11 in a few weeks.\n\n## API\n\n#### `const core = new Hypercore(storage, [key], [options])`\n\nMake a new Hypercore instance.\n\n`storage` should be set to a directory where you want to store the data and core metadata.\n\n``` js\nconst core = new Hypercore('./directory') // store data in ./directory\n```\n\nAlternatively you can pass a [Hypercore Storage](https://github.com/holepunchto/hypercore-storage) or use a [Corestore](https://github.com/holepunchto/corestore) if you want to make many Hypercores efficiently. Note that `random-access-storage` is no longer supported.\n\n`key` can be set to a Hypercore key which is a hash of Hypercore's internal auth manifest, describing how to validate the Hypercore. If you do not set this, it will be loaded from storage. If nothing is previously stored, a new auth manifest will be generated giving you local write access to it.\n\n`options` include:\n\n``` js\n{\n  createIfMissing: true, // create a new Hypercore key pair if none was present in storage\n  overwrite: false, // overwrite any old Hypercore that might already exist\n  valueEncoding: 'json' | 'utf-8' | 'binary', // defaults to binary\n  encodeBatch: batch =\u003e { ... }, // optionally apply an encoding to complete batches\n  keyPair: kp, // optionally pass the public key and secret key as a key pair\n  encryption: { key: buffer }, // the block encryption key\n  onwait: () =\u003e {}, // hook that is called if gets are waiting for download\n  timeout: 0, // wait at max some milliseconds (0 means no timeout)\n  writable: true, // disable appends and truncates\n  inflightRange: null, // Advanced option. Set to [minInflight, maxInflight] to change the min and max inflight blocks per peer when downloading.\n  ongc: (session) =\u003e { ... }, // A callback called when the session is garbage collected\n  notDownloadingLinger: 20000, // How many milliseconds to wait after downloading finishes keeping the connection open. Defaults to a random number between 20-40s\n  allowFork: true, // Enables updating core when it forks\n}\n```\n\nYou can also set valueEncoding to any [compact-encoding](https://github.com/compact-encoding) instance.\n\nvalueEncodings will be applied to individual blocks, even if you append batches. If you want to control encoding at the batch-level, you can use the `encodeBatch` option, which is a function that takes a batch and returns a binary-encoded batch. If you provide a custom valueEncoding, it will not be applied prior to `encodeBatch`.\n\nThe user may provide a custom encryption module as `opts.encryption`, which should satisfy the [HypercoreEncryption](https://github.com/holepunchto/hypercore-encryption) interface.\n\n#### `const { length, byteLength } = await core.append(block)`\n\nAppend a block of data (or an array of blocks) to the core.\nReturns the new length and byte length of the core.\n\n``` js\n// simple call append with a new block of data\nawait core.append(Buffer.from('I am a block of data'))\n\n// pass an array to append multiple blocks as a batch\nawait core.append([Buffer.from('batch block 1'), Buffer.from('batch block 2')])\n```\n\n#### `const block = await core.get(index, [options])`\n\nGet a block of data.\nIf the data is not available locally this method will prioritize and wait for the data to be downloaded.\n\n``` js\n// get block #42\nconst block = await core.get(42)\n\n// get block #43, but only wait 5s\nconst blockIfFast = await core.get(43, { timeout: 5000 })\n\n// get block #44, but only if we have it locally\nconst blockLocal = await core.get(44, { wait: false })\n```\n\n`options` include:\n\n``` js\n{\n  wait: true, // wait for block to be downloaded\n  onwait: () =\u003e {}, // hook that is called if the get is waiting for download\n  timeout: 0, // wait at max some milliseconds (0 means no timeout)\n  valueEncoding: 'json' | 'utf-8' | 'binary', // defaults to the core's valueEncoding\n  decrypt: true // automatically decrypts the block if encrypted\n}\n```\n\n#### `const has = await core.has(start, [end])`\n\nCheck if the core has all blocks between `start` and `end`.\n\n#### `const updated = await core.update([options])`\n\nWaits for initial proof of the new core length until all `findingPeers` calls has finished.\n\n``` js\nconst updated = await core.update()\n\nconsole.log('core was updated?', updated, 'length is', core.length)\n```\n\n`options` include:\n\n``` js\n{\n  wait: false\n}\n```\n\nUse `core.findingPeers()` or `{ wait: true }` to make `await core.update()` blocking.\n\n#### `const [index, relativeOffset] = await core.seek(byteOffset, [options])`\n\nSeek to a byte offset.\n\nReturns `[index, relativeOffset]`, where `index` is the data block the `byteOffset` is contained in and `relativeOffset` is\nthe relative byte offset in the data block.\n\n``` js\nawait core.append([Buffer.from('abc'), Buffer.from('d'), Buffer.from('efg')])\n\nconst first = await core.seek(1) // returns [0, 1]\nconst second = await core.seek(3) // returns [1, 0]\nconst third = await core.seek(5) // returns [2, 1]\n```\n\n``` js\n{\n  wait: true, // wait for data to be downloaded\n  timeout: 0 // wait at max some milliseconds (0 means no timeout)\n}\n```\n\n#### `const stream = core.createReadStream([options])`\n\nMake a read stream to read a range of data out at once.\n\n``` js\n// read the full core\nconst fullStream = core.createReadStream()\n\n// read from block 10-15\nconst partialStream = core.createReadStream({ start: 10, end: 15 })\n\n// pipe the stream somewhere using the .pipe method on Node.js or consume it as\n// an async iterator\n\nfor await (const data of fullStream) {\n  console.log('data:', data)\n}\n```\n\n`options` include:\n\n``` js\n{\n  start: 0,\n  end: core.length,\n  live: false,\n  snapshot: true // auto set end to core.length on open or update it on every read\n}\n```\n\n#### `const bs = core.createByteStream([options])`\n\nMake a byte stream to read a range of bytes.\n\n``` js\n// Read the full core\nconst fullStream = core.createByteStream()\n\n// Read from byte 3, and from there read 50 bytes\nconst partialStream = core.createByteStream({ byteOffset: 3, byteLength: 50 })\n\n// Consume it as an async iterator\nfor await (const data of fullStream) {\n  console.log('data:', data)\n}\n\n// Or pipe it somewhere like any stream:\npartialStream.pipe(process.stdout)\n```\n\n`options` include:\n\n``` js\n{\n  byteOffset: 0,\n  byteLength: core.byteLength - options.byteOffset,\n  prefetch: 32\n}\n```\n\n#### `const cleared = await core.clear(start, [end], [options])`\n\nClear stored blocks between `start` and `end`, reclaiming storage when possible.\n\n``` js\nawait core.clear(4) // clear block 4 from your local cache\nawait core.clear(0, 10) // clear block 0-10 from your local cache\n```\n\nThe core will also gossip to peers it is connected to, that is no longer has these blocks.\n\n`options` include:\n```js\n{\n  diff: false // Returned `cleared` bytes object is null unless you enable this\n}\n```\n\n#### `await core.truncate(newLength, [forkId])`\n\nTruncate the core to a smaller length.\n\nPer default this will update the fork id of the core to `+ 1`, but you can set the fork id you prefer with the option.\nNote that the fork id should be monotonely incrementing.\n\n#### `const hash = await core.treeHash([length])`\n\nGet the Merkle Tree hash of the core at a given length, defaulting to the current length of the core.\n\n#### `const range = core.download([range])`\n\nDownload a range of data.\n\nYou can await when the range has been fully downloaded by doing:\n\n```js\nawait range.done()\n```\n\nA range can have the following properties:\n\n``` js\n{\n  start: startIndex,\n  end: nonInclusiveEndIndex,\n  blocks: [index1, index2, ...],\n  linear: false // download range linearly and not randomly\n}\n```\n\nTo download the full core continuously (often referred to as non sparse mode) do\n\n``` js\n// Note that this will never be considered downloaded as the range\n// will keep waiting for new blocks to be appended.\ncore.download({ start: 0, end: -1 })\n```\n\nTo download a discrete range of blocks pass a list of indices.\n\n```js\ncore.download({ blocks: [4, 9, 7] })\n```\n\nTo cancel downloading a range simply destroy the range instance.\n\n``` js\n// will stop downloading now\nrange.destroy()\n```\n\n#### `const session = core.session([options])`\n\nCreates a new Hypercore instance that shares the same underlying core.\n\nYou must close any session you make.\n\nOptions are inherited from the parent instance, unless they are re-set.\n\n`options` are the same as in the constructor with the follow additions:\n\n```\n{\n  weak: false // Creates the session as a \"weak ref\" which closes when all non-weak sessions are closed\n  exclusive: false, // Create a session with exclusive access to the core. Creating an exclusive session on a core with other exclusive sessions, will wait for the session with access to close before the next exclusive session is `ready`\n  checkout: undefined, // A index to checkout the core at. Checkout sessions must be an atom or a named session\n  atom: undefined, // A storage atom for making atomic batch changes across hypercores\n  name: null, // Name the session creating a persisted branch of the core. Still beta so may break in the future\n}\n```\n\nAtoms allow making atomic changes across multiple hypercores. Atoms can be created using a `core`'s `storage` (eg. `const atom = core.state.storage.createAtom()`). Changes made with an atom based session is not persisted until the atom is flushed via `await atom.flush()`, but can be read at any time. When atoms flush, all changes made outside of the atom will be clobbered as the core blocks will now match the atom's blocks. For example:\n\n```js\nconst core = new Hypercore('./atom-example')\nawait core.ready()\n\nawait core.append('block 1')\n\nconst atom = core.state.storage.createAtom()\nconst atomicSession = core.session({ atom })\n\nawait core.append('block 2') // Add blocks not using the atom\n\nawait atomicSession.append('atom block 2') // Add different block to atom\nawait atom.flush()\n\nconsole.log((await core.get(core.length - 1)).toString()) // prints 'atom block 2' not 'block 2'\n```\n\n#### `const { byteLength, length } = core.commit(session, opts = {})`\n\nAttempt to apply blocks from the session to the `core`. `core` must be a default core, aka a non-named session.\n\nReturns `null` if committing failed.\n\n#### `const snapshot = core.snapshot([options])`\n\nSame as above, but backed by a storage snapshot so will not truncate nor append.\n\n#### `const info = await core.info([options])`\n\nGet information about this core, such as its total size in bytes.\n\nThe object will look like this:\n\n```js\nInfo {\n  key: Buffer(...),\n  discoveryKey: Buffer(...),\n  length: 18,\n  contiguousLength: 16,\n  byteLength: 742,\n  fork: 0,\n  padding: 8,\n  storage: {\n    oplog: 8192, \n    tree: 4096, \n    blocks: 4096, \n    bitfield: 4096 \n  }\n}\n```\n\n`options` include:\n\n```js\n{\n  storage: false // get storage estimates in bytes, disabled by default\n}\n```\n\n#### `await core.close()`\n\nFully close this core.\n\n#### `core.on('close')`\n\nEmitted when the core has been fully closed.\n\n#### `await core.ready()`\n\nWait for the core to fully open.\n\nAfter this has called `core.length` and other properties have been set.\n\nIn general you do NOT need to wait for `ready`, unless checking a synchronous property,\nas all internals await this themself.\n\n#### `core.on('ready')`\n\nEmitted after the core has initially opened all its internal state.\n\n#### `core.writable`\n\nCan we append to this core?\n\nPopulated after `ready` has been emitted. Will be `false` before the event.\n\n#### `core.readable`\n\nCan we read from this core? After closing the core this will be false.\n\nPopulated after `ready` has been emitted. Will be `false` before the event.\n\n#### `core.id`\n\nString containing the id (z-base-32 of the public key) identifying this core.\n\nPopulated after `ready` has been emitted. Will be `null` before the event.\n\n#### `core.key`\n\nBuffer containing the public key identifying this core.\n\nPopulated after `ready` has been emitted. Will be `null` before the event.\n\n#### `core.keyPair`\n\nObject containing buffers of the core's public and secret key\n\nPopulated after `ready` has been emitted. Will be `null` before the event.\n\n#### `core.discoveryKey`\n\nBuffer containing a key derived from the core's public key.\nIn contrast to `core.key` this key does not allow you to verify the data but can be used to announce or look for peers that are sharing the same core, without leaking the core key.\n\nPopulated after `ready` has been emitted. Will be `null` before the event.\n\n#### `core.length`\n\nHow many blocks of data are available on this core.\n\nPopulated after `ready` has been emitted. Will be `0` before the event.\n\n#### `core.signedLength`\n\nHow many blocks of data are available on this core that have been signed by a quorum. This is equal to `core.length` for Hypercores's with a single signer.\n\nPopulated after `ready` has been emitted. Will be `0` before the event.\n\n#### `core.contiguousLength`\n\nHow many blocks are contiguously available starting from the first block of this core?\n\nPopulated after `ready` has been emitted. Will be `0` before the event.\n\n#### `core.fork`\n\nWhat is the current fork id of this core?\n\nPopulated after `ready` has been emitted. Will be `0` before the event.\n\n#### `core.padding`\n\nHow much padding is applied to each block of this core? Will be `0` unless block encryption is enabled.\n\n#### `const stream = core.replicate(isInitiatorOrReplicationStream)`\n\nCreate a replication stream. You should pipe this to another Hypercore instance.\n\nThe `isInitiator` argument is a boolean indicating whether you are the initiator of the connection (ie the client)\nor if you are the passive part (ie the server).\n\nIf you are using a P2P swarm like [Hyperswarm](https://github.com/hyperswarm/hyperswarm) you can know this by checking if the swarm connection is a client socket or server socket. In Hyperswarm you can check that using the [client property on the peer details object](https://github.com/hyperswarm/hyperswarm#swarmonconnection-socket-details--)\n\nIf you want to multiplex the replication over an existing Hypercore replication stream you can pass\nanother stream instance instead of the `isInitiator` boolean.\n\n``` js\n// assuming we have two cores, localCore + remoteCore, sharing the same key\n// on a server\nconst net = require('net')\nconst server = net.createServer(function (socket) {\n  socket.pipe(remoteCore.replicate(false)).pipe(socket)\n})\n\n// on a client\nconst socket = net.connect(...)\nsocket.pipe(localCore.replicate(true)).pipe(socket)\n```\n\n#### `const done = core.findingPeers()`\n\nCreate a hook that tells Hypercore you are finding peers for this core in the background. Call `done` when your current discovery iteration is done.\nIf you're using Hyperswarm, you'd normally call this after a `swarm.flush()` finishes.\n\nThis allows `core.update` to wait for either the `findingPeers` hook to finish or one peer to appear before deciding whether it should wait for a merkle tree update before returning.\n\n#### `core.on('append')`\n\nEmitted when the core has been appended to (i.e. has a new length / byteLength), either locally or remotely.\n\n#### `core.on('truncate', ancestors, forkId)`\n\nEmitted when the core has been truncated, either locally or remotely.\n\n#### `core.on('peer-add')`\n\nEmitted when a new connection has been established with a peer.\n\n#### `core.on('peer-remove')`\n\nEmitted when a peer's connection has been closed.\n\n#### `core.on('upload', index, byteLength, peer)`\n\nEmitted when a block is uploaded to a peer.\n\n#### `core.on('download', index, byteLength, peer)`\n\nEmitted when a block is downloaded from a peer.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fholepunchto%2Fhypercore","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fholepunchto%2Fhypercore","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fholepunchto%2Fhypercore/lists"}