Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/infura/ipfs-pump
A tool to transfer data between IPFS nodes or storage
https://github.com/infura/ipfs-pump
ipfs
Last synced: about 1 month ago
JSON representation
A tool to transfer data between IPFS nodes or storage
- Host: GitHub
- URL: https://github.com/infura/ipfs-pump
- Owner: INFURA
- License: mit
- Created: 2019-06-21T16:03:14.000Z (over 5 years ago)
- Default Branch: master
- Last Pushed: 2023-04-25T21:13:33.000Z (over 1 year ago)
- Last Synced: 2024-11-15T22:07:55.553Z (about 2 months ago)
- Topics: ipfs
- Language: Go
- Homepage:
- Size: 77.1 KB
- Stars: 60
- Watchers: 38
- Forks: 16
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Codeowners: CODEOWNERS
Awesome Lists containing this project
README
# ipfs-pump
`ipfs-pump` is a command line tool to copy data between IPFS nodes, cluster or storage.
It support multiple interfaces:
- the IPFS API
- direct access to a FlatFS datastore
- direct access to a Badger datastore
- direct access to a S3 datastore
- a file with a list of CID
## Concepts
An `Enumerator` is a source that will enumerate the list of existing blocks.
A `Collector` is a source that where to retrieve block's data.
A `Drain` is a destination where to push blocks.
Note: even though you will likely want the `Enumerator` and the `Collector` to be the same source, they don't necessarily have to.
## Install
```
go get -u github.com/INFURA/ipfs-pump
```
## Usage and examples
You need to provide as arguments the three types of `Enumerator`, `Collector` and `Drain`, as well as the corresponding configuration flags.
Copy between two live nodes using the API:
```
ipfs-pump \
apipin --enum-api-pin-url=127.0.0.1:5001 --enum-api-pin-stream \
api --coll-api-url=127.0.0.1:5001 \
api --drain-api-url=127.0.0.1:5002 \
--worker=10
```
Copy from a FlatFS storage to a Badger storage when the nodes are stopped:
```
ipfs-pump \
flatfs --enum-flatfs-path=~/.ipfs/blocks \
flatfs --coll-flatfs-path=~/.ipfs/blocks \
badger --drain-badger-path=~/.ipfs/badgerds \
--worker=50
```
## Parallel processing
Using the `--worker` flag you can enable parallel processing and greatly increase the throughput.
## License
MIT