Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.
Awesome Lists | Featured Topics | Projects
https://github.com/storacha/ipfs-car

🚘 Convert files to content-addressable archives and back
https://github.com/storacha/ipfs-car
car ipfs
Last synced: about 1 month ago
JSON representation
🚘 Convert files to content-addressable archives and back
Host: GitHub
URL: https://github.com/storacha/ipfs-car
Owner: storacha
License: other
Created: 2021-05-11T15:06:44.000Z (over 3 years ago)
Default Branch: main
Last Pushed: 2024-01-22T11:53:40.000Z (about 1 year ago)
Last Synced: 2024-09-10T13:09:13.092Z (4 months ago)
Topics: car, ipfs
Language: JavaScript
Homepage:
Size: 1.92 MB
Stars: 148
Watchers: 4
Forks: 43
Open Issues: 24
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project

README

        # ipfs-car 🚘✨⬢

> Convert files to content-addressable archives (.car) and back

[![Build](https://github.com/web3-storage/ipfs-car/actions/workflows/main.yml/badge.svg)](https://github.com/web3-storage/ipfs-car/actions/workflows/main.yml)

[![dependencies Status](https://status.david-dm.org/gh/web3-storage/ipfs-car.svg)](https://david-dm.org/web3-storage/ipfs-car)

[![JavaScript Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://standardjs.com)

[![Downloads](https://img.shields.io/npm/dm/ipfs-car.svg)](https://www.npmjs.com/package/ipfs-car)

[![Minzipped size](https://badgen.net/bundlephobia/minzip/ipfs-car)](https://bundlephobia.com/result?p=ipfs-car)

## Description

`ipfs-car` is a library and CLI tool to pack & unpack files from [Content Addressable aRchives (CAR)](https://ipld.io/specs/transport/car/) file. A thin wrapper over [@ipld/car](https://github.com/ipld/js-car) and [unix-fs](https://github.com/ipfs/js-ipfs-unixfs).

Content-addressable archives store data as blocks (a sequence of bytes) each prefixed with the [Content ID (CID)](https://docs.ipfs.tech/concepts/content-addressing/) derived from the hash of the data; typically in a file with a `.car` extension.

Use `ipfs-car` to pack your files into a .car; a portable, verifiable, IPFS compatible archive.

```sh

$ ipfs-car pack path/to/files --output my-files.car

```

or unpack files from a .car, and verify that every block matches it's CID

```sh

$ ipfs-car unpack my-files.car --output path/to/write/to

```

Fetch and locally verify files from a IPFS gateway over http

```sh

curl "https://ipfs.io/ipfs/bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu?format=car" | ipfs-car unpack -o images

```

## Install

```sh

# install it as a dependency

$ npm i ipfs-car

# OR use the cli without installing via `npx`

$ npx ipfs-car --help

```

## Usage

Pack files into a .car

```sh

# write a content addressed archive to stdout.

$ ipfs-car pack path/to/file/or/dir

# note: CAR data streamed to stdout will not have roots set in CAR header!

# specify the car file name.

$ ipfs-car pack path/to/files --output path/to/write/a.car

# by default, ipfs-car will wrap files in an IPFS directory.

# use --no-wrap to avoid this.

$ ipfs-car pack path/to/file --no-wrap --output path/to/write/a.car

```

Unpack files from a .car

```sh

# unpack files to a specific path.

$ ipfs-car unpack path/to/my.car --output /path/to/unpack/files/to

# unpack a specific root.

$ ipfs-car unpack path/to/my.car --root 

# unpack files from a .car on stdin.

$ cat path/to/my.car | ipfs-car unpack

```

Show the files and directories in a .car

```sh

# show the files and directories.

$ ipfs-car ls path/to/my.car

# show the files and directories, their CIDs and byte sizes.

$ ipfs-car ls path/to/my.car --verbose

```

Show the root CIDs in a .car

```sh

# show the CID roots found in the CAR header.

$ ipfs-car roots path/to/my.car

# show the CID roots found implicitly from the blocks in the file.

$ ipfs-car roots --implicit path/to/my.car

```

Show the block CIDs in a .car

```sh

# show the CIDs for all the blocks.

$ ipfs-car blocks path/to/my.car

```

Get other information about a CAR

```sh

# generate CID for a CAR.

$ ipfs-car hash path/to/my.car

```

## API

To pack files into content-addressable archives, you can use the following:

- `createFileEncoderStream` a factory function for creating a `ReadableStream` that encodes a single file into DAG `Block`s.

- `createDirectoryEncoderStream` a factory function for creating a `ReadableStream` for encoding a directory of files into DAG `Block`s.

- `CAREncoderStream` a `TransformStream` sub-class that you can write `Block`s to and read `Uint8Array` CAR file data from.

To unpack content-addressable archives to files, you should use `@ipld/car` and `ipfs-unixfs-exporter` modules.

### Examples

#### Basic single file pack

```js

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])

const carStream = createFileEncoderStream(file).pipeThrough(new CAREncoderStream())

// carStream.pipeTo(somewhereWritable)

```

#### Directory pack to file system in Node.js

```js

import { Writable } from 'stream'

import { createDirectoryEncoderStream, CAREncoderStream } from 'ipfs-car'

import { filesFromPaths } from 'files-from-path'

const files = await filesFromPaths(process.argv.slice(2))

await createDirectoryEncoderStream(files)

  .pipeThrough(new CAREncoderStream())

  .pipeTo(Writable.toWeb(process.stdout))

```

Usage: `node script.js file0 file1 dir0 > my.car`.

#### Obtaining the root CID

The root CID is the final block generated by the file/directory encoder stream. Use a transform stream to record the CID of the last block generated:

```js

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])

let rootCID

await createFileEncoderStream(file)

  .pipeThrough(new TransformStream({

    transform (block, controller) {

      rootCID = block.cid

      controller.enqueue(block)

    }

  }))

  .pipeThrough(new CAREncoderStream())

  .pipeTo(new WritableStream())

console.log(rootCID.toString())

```

#### Adding root CIDs to the CAR header

If you need root CIDs in the CAR header, there are two approaches you can use:

1. Buffer all the DAG blocks, then encode with known root:

```js

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])

const blocks = []

// buffer the output

await createFileEncoderStream(file)

  .pipeTo(new WritableStream({ write: b => blocks.push(b) }))

const rootCID = blocks.at(-1).cid

const blockStream = new ReadableStream({

  pull (controller) {

    if (blocks.length) {

      controller.enqueue(blocks.shift())

    } else {

      controller.close()

    }

  }

})

await blockStream

  .pipeThrough(new CAREncoderStream([rootCID])) // pass root to CAR encoder

  .pipeTo(new WritableStream())

```

2. Write to disk with placeholder CID, then update after DAG is completely generated (Note: Node.js only):

```js

import fs from 'fs'

import { Writable } from 'stream'

import { CarWriter } from '@ipld/car/writer'

import { CID } from 'multiformats/cid'

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

// Root CID written in CAR file header before it is updated with the real root CID.

const placeholderCID = CID.parse('bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi')

const file = new Blob(['Hello ipfs-car!'])

let rootCID

await createFileEncoderStream(file)

  .pipeThrough(new TransformStream({

    transform (block, controller) {

      rootCID = block.cid

      controller.enqueue(block)

    }

  }))

  .pipeThrough(new CAREncoderStream(placeholderCID))

  .pipeTo(Writable.toWeb(fs.createWriteStream('path/to/my.car')))

// update roots in CAR header

const fd = await fs.promises.open(opts.output, 'r+')

await CarWriter.updateRootsInFile(fd, [rootCID])

await fd.close()

```

#### Unpacking files from a CAR

This functionality is not provided by this library, but is easy to do with `@ipld/car` and `ipfs-unixfs-exporter` modules:

```js

import { CarIndexedReader } from '@ipld/car/indexed-reader'

import { recursive as exporter } from 'ipfs-unixfs-exporter'

const reader = await CarIndexedReader.fromFile('path/to/my.car')

const roots = await reader.getRoots()

const entries = exporter(roots[0], {

  async get (cid) {

    const block = await reader.get(cid)

    return block.bytes

  }

})

for await (const entry of entries) {

  if (entry.type === 'file' || entry.type === 'raw') {

    console.log('file', entry.path, entry.content)

  } else if (entry.type === 'directory') {

    console.log('directory', entry.path)

  }

}

```

## Contributing

Feel free to join in. All welcome. [Open an issue](https://github.com/web3-storage/ipfs-car/issues)!

## License

Dual-licensed under [MIT + Apache 2.0](https://github.com/web3-storage/ipfs-car/blob/main/LICENSE.md)