Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/application-research/delta-edge
Edge Gateway + Storage deal making using Delta
https://github.com/application-research/delta-edge
delta ipfs uploader
Last synced: about 1 month ago
JSON representation
Edge Gateway + Storage deal making using Delta
- Host: GitHub
- URL: https://github.com/application-research/delta-edge
- Owner: application-research
- License: other
- Created: 2023-01-09T05:58:34.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-07-13T04:51:29.000Z (over 1 year ago)
- Last Synced: 2024-10-06T12:09:13.746Z (about 2 months ago)
- Topics: delta, ipfs, uploader
- Language: Go
- Homepage:
- Size: 12.9 MB
- Stars: 11
- Watchers: 1
- Forks: 6
- Open Issues: 28
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
# EdgeUR (Upload/Retrieve)
[](https://github.com/application-research/edge-ur/actions/workflows/codeql.yml)
*Edge is currently under heavy development.*
## Goal/Purpose
Dedicated light node to upload and retrieve their CIDs. To do this, we decoupled the upload and retrieval aspect from the Estuary API node so we can create a node that can live on the "edge" closer to the customer.
By decoupling this to a light node, we achieve the following:
- dedicated node assignment for each customer. The customer or user can now launch an edge node and use it for both uploading to Estuary and retrieval using the same API keys issued from Estuary.
- switches the upload protocol. The user still needs to upload via HTTP but the edge node will transfer the file over to a delta node to make deals.
## Features
- Only supports online/e2e verified deals for now.
- Accepts concurrent uploads (small to large)
- Stores the CID and content on the local blockstore using whypfs
- Save the data on local sqlite DB
- retries the storage deals if it fails. Uses delta `auto_retry` feature.
- periodically checks the status of the deals and update the database.
- For 32GB and above, the node will split the file into 32GB chunks and make deals for each chunk and car them. **[WIP]**
## Quick how to
### Upload a file with a given miner
We have a working live server that you can use. To run basic upload, please get API key first. You can get one here https://auth.estuary.tech/register-new-token
```
curl --location 'http://localhost:3000/api/v1/content/add' \
--header 'Authorization: Bearer [APIKEY]' \
--form 'data=@"/file.zip"' \
--form 'miners="f0137168,f0717969"' // list of miners (optional)
{
"status": "success",
"message": "File uploaded and pinned successfully to miners. Please take note of the ids.",
"contents": [
{
"ID": 53,
"name": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y.zip",
"size": 1157548,
"cid": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y",
"delta_content_id": 0,
"delta_node_url": "https://node.delta.store",
"status": "pinned",
"last_message": "",
"miner": "f0137168",
"replication": 0,
"created_at": "2023-04-22T14:40:43.10918+02:00",
"updated_at": "2023-04-22T14:40:43.10918+02:00"
},
{
"ID": 54,
"name": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y.zip",
"size": 1157548,
"cid": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y",
"delta_content_id": 0,
"delta_node_url": "https://node.delta.store",
"status": "pinned",
"last_message": "",
"miner": "f0717969",
"replication": 0,
"created_at": "2023-04-22T14:40:43.112161+02:00",
"updated_at": "2023-04-22T14:40:43.112161+02:00"
}
]
}
```
### Check status
Get the content id and use the following call
```
curl --location 'http://localhost:3000/api/v1/status/53' \
--header 'Authorization: Bearer [APIKEY]'
{
"content": {
"ID": 53,
"name": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y.zip",
"size": 1157548,
"cid": "bafybeicgdjdvwes3e5aaicqljrlv6hpdfsducknrjvsq66d4gsvepolk6y",
"delta_content_id": 67547,
"delta_node_url": "https://node.delta.store",
"status": "uploaded-to-delta",
"last_message": "",
"miner": "f0137168",
"replication": 0,
"created_at": "2023-04-22T14:40:43.10918+02:00",
"updated_at": "2023-04-22T14:40:46.564971+02:00"
}
}
```
### View the file using the gateway url
```
http://localhost:1313/gw/
http://localhost:1313/gw/content/
```
## Getting Started
To get started, follow the guide [here](docs/README.md).
# Author
Protocol Labs Outercore Engineering