https://github.com/application-research/add-cids-to-bstore
Simple CID fetcher to Delta/Edge compatible blockstore
https://github.com/application-research/add-cids-to-bstore
Last synced: 27 days ago
JSON representation
Simple CID fetcher to Delta/Edge compatible blockstore
- Host: GitHub
- URL: https://github.com/application-research/add-cids-to-bstore
- Owner: application-research
- Created: 2023-07-23T13:15:28.000Z (almost 3 years ago)
- Default Branch: main
- Last Pushed: 2023-07-25T04:24:19.000Z (almost 3 years ago)
- Last Synced: 2025-03-19T08:49:21.353Z (about 1 year ago)
- Language: Go
- Size: 32.2 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Fetch CIDs to delta/edge compatible blockstore
Fetch a list of CIDs (Content Identifier) from a specific URL and download the corresponding files from the IPFS (InterPlanetary File System) network. The downloaded files will be stored in a delta/edge compatible blockstore, which is a data structure used to manage content-addressable data efficiently.
## Steps:
- Fetch the list of CIDs from a designated URL (e.g., https://example.com/cids.txt) and store them in a file named cids.txt.
- Sample list of CIDs: https://bafybeifcghbafml4yrk43m3pvplin4auibnwrdv5v3rnwnovjjpkt6tkju.ipfs.dweb.link/
- Loop through each CID in cids.txt, download the corresponding file from the IPFS network using the https://dweb.link/ipfs/ base URL, and show a progress bar for each download.
- If there is an error while downloading a file for a specific CID, record the problematic CID in a list of failed CIDs.
- After successfully downloading all the valid files, print the list of CIDs that encountered errors during the download process.
- Store the downloaded files in a delta/edge compatible blockstore for efficient management and future use.
## Install
```
go mod tidy
go build -o addc
```
## Running
```
./addc --repo= --cids-url-source="https://bafybeifcghbafml4yrk43m3pvplin4auibnwrdv5v3rnwnovjjpkt6tkju.ipfs.dweb.link/"
```
## Author
Protocol Labs Outercore Engineering.