{"id":18877556,"url":"https://github.com/nftstorage/carbites","last_synced_at":"2025-07-14T00:35:07.536Z","repository":{"id":43723712,"uuid":"372991456","full_name":"nftstorage/carbites","owner":"nftstorage","description":"***Notice: This repository is no longer maintained.*** 🚗 🚙 🚕 Chunking for CAR files. Split a single CAR into multiple CARs.","archived":false,"fork":false,"pushed_at":"2024-04-01T11:00:40.000Z","size":330,"stargazers_count":24,"open_issues_count":3,"forks_count":7,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-06-19T13:05:54.717Z","etag":null,"topics":["car","chunking","cid","ipld","multiformats","splitting"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nftstorage.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-06-02T00:07:45.000Z","updated_at":"2024-07-01T22:37:17.000Z","dependencies_parsed_at":"2024-06-18T15:26:02.097Z","dependency_job_id":"7959da57-8fd9-475e-89f8-a893899c3d73","html_url":"https://github.com/nftstorage/carbites","commit_stats":{"total_commits":79,"total_committers":4,"mean_commits":19.75,"dds":0.05063291139240511,"last_synced_commit":"ed697ccc93f2b04f679afd76aef127a270978edd"},"previous_names":["alanshaw/carbites"],"tags_count":27,"template":false,"template_full_name":null,"purl":"pkg:github/nftstorage/carbites","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nftstorage%2Fcarbites","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nftstorage%2Fcarbites/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nftstorage%2Fcarbites/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nftstorage%2Fcarbites/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nftstorage","download_url":"https://codeload.github.com/nftstorage/carbites/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nftstorage%2Fcarbites/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265227898,"owners_count":23731059,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["car","chunking","cid","ipld","multiformats","splitting"],"created_at":"2024-11-08T06:20:00.515Z","updated_at":"2025-07-14T00:35:07.398Z","avatar_url":"https://github.com/nftstorage.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# carbites\n\n[![Build](https://github.com/nftstorage/carbites/actions/workflows/main.yml/badge.svg)](https://github.com/nftstorage/carbites/actions/workflows/main.yml)\n[![dependencies Status](https://status.david-dm.org/gh/nftstorage/carbites.svg)](https://david-dm.org/nftstorage/carbites)\n[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)\n[![npm bundle size](https://img.shields.io/bundlephobia/minzip/carbites)](https://bundlephobia.com/package/carbites)\n\nChunking for [CAR files](https://ipld.io/specs/transport/car/). Split a single CAR into multiple CARs.\n\n## Install\n\n```sh\nnpm install carbites\n```\n\n## Usage\n\nCarbites supports 3 different strategies:\n\n1. [**Simple**](#simple) (default) - fast but naive, only the first CAR output has a root CID, subsequent CARs have a placeholder \"empty\" CID.\n2. [**Rooted**](#rooted) - like simple, but creates a custom root node to ensure all blocks in a CAR are referenced.\n3. [**Treewalk**](#treewalk) - walks the DAG to pack sub-graphs into each CAR file that is output. Every CAR has the same root CID, but contains a different portion of the DAG.\n\n### Simple\n\n```js\nimport { CarSplitter } from 'carbites'\nimport { CarReader } from '@ipld/car'\nimport fs from 'fs'\n\nconst bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))\nconst targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs\nconst splitter = new CarSplitter(bigCar, targetSize) // (simple strategy)\n\nfor await (const car of splitter.cars()) {\n  // Each `car` is an AsyncIterable\u003cUint8Array\u003e\n}\n```\n\n⚠️ Note: The first CAR output has roots in the header, subsequent CARs have an empty root CID [`bafkqaaa`](https://cid.ipfs.io/#bafkqaaa) as [recommended](https://ipld.io/specs/transport/car/carv1/#number-of-roots).\n\n### Rooted\n\nInstead of an empty CID, carbites can generate a special root node for each split CAR that references all the blocks _and_ the original roots (only in the first CAR). To do this, use the `RootedCarSplitter` constructor. When reading/extracting data from the CARs, the root node should be discarded.\n\n\u003cdetails\u003e\n  \u003csummary\u003eExample\u003c/summary\u003e\n\n  ```js\n  import { RootedCarSplitter } from 'carbites/rooted'\n  import { CarReader } from '@ipld/car/reader'\n  import * as dagCbor from '@ipld/dag-cbor'\n  import fs from 'fs'\n\n  const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))\n  const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs\n  const splitter = new RootedCarSplitter(bigCar, targetSize)\n\n  const cars = splitter.cars()\n\n  // Every CAR has a single root - a CBOR node that is an tuple of `/carbites/1`,\n  // an array of root CIDs and an array of block CIDs.\n  // e.g. ['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']]\n\n  const { done, value: car } = await cars.next()\n  const reader = await CarReader.fromIterable(car)\n  const rootCids = await reader.getRoots()\n  const rootNode = dagCbor.decode(await reader.get(rootCids[0]))\n\n  console.log(rootNode[0]) // /carbites/1\n  console.log(rootNode[1]) // Root CIDs (only in first CAR)\n  /*\n  [\n    CID(bafybeictvyf6polqzgop3jt32owubfmsg3kl226omqrfte4eyidubc4rpq)\n  ]\n  */\n  console.log(rootNode[2]) // Block CIDs (all blocks in this CAR)\n  /*\n  [\n    CID(bafybeictvyf6polqzgop3jt32owubfmsg3kl226omqrfte4eyidubc4rpq),\n    CID(bafyreihcsxqhd6agqpboc3wrlvpy5bwuxctv5upicdnt3u2wojv4exxl24),\n    CID(bafyreiasq7d2ihbqm5xvhjjzlmzsensuadrpmpt2tkjsuwq42xpa34qevu)\n  ]\n  */\n  ```\n\n\u003c/details\u003e\n\nThe root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.\n\n### Treewalk\n\nEvery CAR file has the _same_ root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.\n\n\u003cdetails\u003e\n  \u003csummary\u003eExample\u003c/summary\u003e\n\n  ```js\n  import { TreewalkCarSplitter } from 'carbites/treewalk'\n  import { CarReader } from '@ipld/car/reader'\n  import * as dagCbor from '@ipld/dag-cbor'\n  import fs from 'fs'\n\n  const bigCar = await CarReader.fromIterable(fs.createReadStream('/path/to/big.car'))\n  const [rootCid] = await bigCar.getRoots()\n  const targetSize = 1024 * 1024 * 100 // chunk to ~100MB CARs\n  const splitter = new TreewalkCarSplitter(bigCar, targetSize)\n\n  for await (const car of splitter.cars()) {\n    // Each `car` is an AsyncIterable\u003cUint8Array\u003e\n    const reader = await CarReader.fromIterable(car)\n    const [splitCarRootCid] = await reader.getRoots()\n    console.assert(rootCid.equals(splitCarRootCid)) // all cars will have the same root\n  }\n  ```\n\n\u003c/details\u003e\n\n### CLI\n\nInstall the CLI tool to use Carbites from the comfort of your terminal:\n\n```sh\nnpm i -g carbites-cli\n\n# Split a big CAR into many smaller CARs\ncarbites split big.car --size 100MB --strategy simple # (default size \u0026 strategy)\n\n# Join many split CARs back into a single CAR.\ncarbites join big-0.car big-1.car ...\n# Note: not a tool for joining arbitrary CARs together! The split CARs MUST\n# belong to the same CAR and big-0.car should be the first argument.\n```\n\n## API\n\n* [`class CarSplitter`](#class-carsplitter)\n    * [`constructor(car: AsyncIterable\u003cUint8Array\u003e, targetSize: number)`](#constructorcar-asynciterableuint8array-targetsize-number)\n    * [`cars(): AsyncGenerator\u003cAsyncIterable\u003cUint8Array\u003e \u0026 RootsReader\u003e`](#cars-asyncgeneratorasynciterableuint8array--rootsreader)\n    * [`static async fromBlob(blob: Blob, targetSize: number): CarSplitter`](#static-async-fromblobblob-blob-targetsize-number-carsplitter)\n    * [`static async fromIterable(iterable: AsyncIterable\u003cUint8Array\u003e, targetSize: number): CarSplitter`](#static-async-fromiterableiterable-asynciterableuint8array-targetsize-number-carsplitter)\n* [`class CarJoiner`](#class-carjoiner)\n    * [`constructor(cars: Iterable\u003cAsyncIterable\u003cUint8Array\u003e\u003e)`](#constructorcars-iterableasynciterableuint8array)\n    * [`car(): AsyncGenerator\u003cUint8Array\u003e`](#car-asyncgeneratoruint8array)\n* [`class RootedCarSplitter`](#class-rootedcarsplitter)\n* [`class RootedCarJoiner`](#class-rootedcarjoiner)\n* [`class TreewalkCarSplitter`](#class-treewalkcarsplitter)\n* [`class TreewalkCarJoiner`](#class-treewalkcarjoiner)\n\n### `class CarSplitter`\n\nSplit a CAR file into several smaller CAR files.\n\nImport in the browser:\n\n```js\nimport { CarSplitter } from 'https://cdn.skypack.dev/carbites'\n```\n\nImport in Node.js:\n\n```js\nimport { CarSplitter } from 'carbites'\n```\n\nNote: This is an alias of `SimpleCarSplitter` - the default strategy for splitting CARs.\n\n#### `constructor(car: CarReader, targetSize: number)`\n\nCreate a new `CarSplitter` for the passed CAR file, aiming to generate CARs of around `targetSize` bytes in size.\n\n#### `cars(): AsyncGenerator\u003cAsyncIterable\u003cUint8Array\u003e \u0026 RootsReader\u003e`\n\nSplit the CAR file and create multiple smaller CAR files. Returns an `AsyncGenerator` that yields the split CAR files (of type `AsyncIterable\u003cUint8Array\u003e`).\n\nThe CAR files output also implement the [`RootsReader`](https://github.com/ipld/js-car/blob/8c74dc3c7273213b83f4610e4f88cf1ad2830fa6/api.ts#L18-L21) interface from `@ipld/car` which means you can call `getRoots(): Promise\u003cCID[]\u003e` to obtain the root CIDs.\n\n#### `static async fromBlob(blob: Blob, targetSize: number): CarSplitter`\n\nConvenience function to create a new `CarSplitter` from a [blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) of CAR file data.\n\n#### `static async fromIterable(iterable: AsyncIterable\u003cUint8Array\u003e, targetSize: number): CarSplitter`\n\nConvenience function to create a new `CarSplitter` from an `AsyncIterable\u003cUint8Array\u003e` of CAR file data.\n\n### `class CarJoiner`\n\nJoin together split CAR files into a single big CAR.\n\nImport in the browser:\n\n```js\nimport { CarJoiner } from 'https://cdn.skypack.dev/carbites'\n```\n\nImport in Node.js:\n\n```js\nimport { CarJoiner } from 'carbites'\n```\n\nNote: This is an alias of `SimpleCarJoiner` - a joiner for the the default CAR splitting strategy.\n\n#### `constructor(cars: Iterable\u003cCarReader\u003e)`\n\nCreate a new `CarJoiner`  for joining the passed CAR files together.\n\n#### `car(): AsyncGenerator\u003cUint8Array\u003e`\n\nJoin the CAR files together and return the joined CAR.\n\n### `class RootedCarSplitter`\n\nSplit a CAR file into several smaller CAR files ensuring every CAR file contains a single root node that references all the blocks _and_ the original roots (only in the first CAR). When reading/extracting data from the CARs, the root node should be discarded.\n\nImport in the browser:\n\n```js\nimport { RootedCarSplitter } from 'https://cdn.skypack.dev/carbites/rooted'\n```\n\nImport in Node.js:\n\n```js\nimport { RootedCarSplitter } from 'carbites/rooted'\n```\n\nThe API is the same as for [`CarSplitter`](#class-carsplitter).\n\n#### Root Node Format\n\nThe root node is a `dag-cbor` node that is a tuple of the string `/carbites/1`, an array of root CIDs (only seen in first CAR) and an array of block CIDs (all the blocks in the CAR). e.g. `['/carbites/1', ['bafkroot'], ['bafy1', 'bafy2']]`.\n\nNote: The root node is limited to 4MB in size (the largest message IPFS will bitswap). Depending on the settings used to construct the DAG in the CAR, this may mean a split CAR size limit of around 30GiB.\n\n### `class RootedCarJoiner`\n\nJoin together CAR files that were split using [`RootedCarSplitter`](#class-rootedcarsplitter).\n\nThe API is the same as for [`CarJoiner`](#class-carjoiner).\n\n### `class TreewalkCarSplitter`\n\nSplit a CAR file into several smaller CAR files. Every CAR file has the _same_ root CID but a different portion of the DAG. The DAG is traversed from the root node and each block is decoded and links extracted in order to determine which sub-graph to include in each CAR.\n\nImport in the browser:\n\n```js\nimport { TreewalkCarSplitter } from 'https://cdn.skypack.dev/carbites/treewalk'\n```\n\nImport in Node.js:\n\n```js\nimport { TreewalkCarSplitter } from 'carbites/treewalk'\n```\n\nThe API is the same as for [`CarSplitter`](#class-carsplitter).\n\n### `class TreewalkCarJoiner`\n\nJoin together CAR files that were split using [`TreewalkCarSplitter`](#class-treewalkcarsplitter).\n\nThe API is the same as for [`CarJoiner`](#class-carjoiner).\n\n## Releasing\n\nYou can publish by either running `npm publish` in the `dist` directory or using `npx ipjs publish`.\n\n## Related\n  \n* [Carbites in Golang](https://github.com/alanshaw/go-carbites)\n\n## Contribute\n\nFeel free to dive in! [Open an issue](https://github.com/nftstorage/carbites/issues/new) or submit PRs.\n\n## License\n\nDual-licensed under [MIT](https://github.com/nftstorage/carbites/blob/main/LICENSE-MIT) + [Apache 2.0](https://github.com/nftstorage/carbites/blob/main/LICENSE-APACHE)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnftstorage%2Fcarbites","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnftstorage%2Fcarbites","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnftstorage%2Fcarbites/lists"}