Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/tablelandnetwork/basin-provider
Data ingestion for Tableland Basin
https://github.com/tablelandnetwork/basin-provider
basin depin filecoin
Last synced: 7 days ago
JSON representation
Data ingestion for Tableland Basin
- Host: GitHub
- URL: https://github.com/tablelandnetwork/basin-provider
- Owner: tablelandnetwork
- License: mit
- Created: 2023-08-18T15:50:14.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-22T19:45:08.000Z (9 months ago)
- Last Synced: 2024-03-22T20:37:08.803Z (9 months ago)
- Topics: basin, depin, filecoin
- Language: Rust
- Homepage: https://tableland.xyz
- Size: 754 KB
- Stars: 4
- Watchers: 3
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# basin-provider
[](https://github.com/tablelandnetwork/basin-provider/actions/workflows/test.yml)
[](./LICENSE)
[](https://github.com/RichardLitt/standard-readme)
> Data ingestion for Tableland Basin
# Table of Contents
- [basin-provider](#basin-provider)
- [Table of Contents](#table-of-contents)
- [Background](#background)
- [Usage](#usage)
- [Basin Worker](#basin-worker)
- [Development](#development)
- [Running](#running)
- [Run Tests](#run-tests)
- [Contributing](#contributing)
- [License](#license)
# Background
Tableland Basin is a secure and verifiable open data platform. The Basin Provider is a collection of daemons and libraries that work in conjuction with [`basin-cli`](https://github.com/tablelandnetwork/basin-cli.git) and [`basin-storage`](https://github.com/tablelandnetwork/basin-storage.git).
`basin-provider` is responsible for ingesting Basin Publications that are archived on Filecoin by [`basin-storage`](https://github.com/tablelandnetwork/basin-storage.git). Publication data can be ingested in two forms:
1. Continuously published data from a connected database (currently, only PostgreSQL is supported)
2. Raw parquet files
See [`basin-cli`](https://github.com/tablelandnetwork/basin-cli.git) for usage info.
This repo contains the following crates:
- [`basin_worker`](/lib/worker)`: Daemon that exposes a Cap’n Proto RPC API for creating publications and ingesting publication data.
[`basin-storage`](https://github.com/tablelandnetwork/basin-storage.git).
- [`basin_evm`](/lib/evm): Rust bindings for the [`basin-storage`](https://github.com/tablelandnetwork/basin-storage.git) EVM contract.
- [`basin_protocol`](/lib/protocol): Cap’n Proto protocol definitions and Rust bindings.
- [`basin_common`](/lib/common): Shared types and utilities.
🚧 Basin is currently not in a production-ready state. Any data that is pushed to the network may be subject to deletion. 🚧
# Usage
## Basin Worker
```bash
basin_worker --help
Ingest daemon for Tableland Basin
Usage: basin_worker [OPTIONS] --export-bucket --export-credentials --database-url
Options:
--evm-type
EVM type (other EVM flags are ignored when this is 'mem')
[env: EVM_TYPE=]
[default: remote]
Possible values:
- mem: Use an in-memory ephemeral EVM w/ random wallet (a BasinStorage contract will be deployed)
- remote: Requires wallet private key, contract address, provider URL, and chain ID
--evm-wallet-pk
Wallet private key (ECDSA, secp256k1) to use with the EVM (must have PUB_ADMIN_ROLE)
[env: EVM_WALLET_PK=]
--evm-contract-address
BasinStorage EVM contract address (ECDSA, secp256k1)
[env: EVM_CONTRACT_ADDRESS=]
--evm-provider-url
EVM provider URL
[env: EVM_PROVIDER_URL=]
[default: http://127.0.0.1:8545]
--evm-chain-id
EVM chain ID
[env: EVM_CHAIN_ID=]
[default: 31337]
--export-bucket
Parquet export GCS bucket
[env: EXPORT_BUCKET=]
--export-credentials
Parquet export sink credentials
[env: EXPORT_CREDENTIALS=]
--database-url
Postgres-style database URL
[env: DATABASE_URL=]
--bind-address
Host and port to bind the RPC API to
[env: BIND_ADDRESS=]
[default: 127.0.0.1:3000]
--bind-health-address
Host and port to bind the Health API to
[env: BIND_HEALTH_ADDRESS=]
[default: 127.0.0.1:3001]
-v, --verbosity...
Logging verbosity (repeat for more verbose logging)
[env: VERBOSITY=]
-q, --quiet
Silence logging
[env: QUIET=]
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
```
# Development
## Running
```bash
docker compose up
```
## Run Tests
```bash
docker compose up crdb
cargo test
```
# Contributing
PRs accepted.
Small note: If editing the README, please conform to the
[standard-readme](https://github.com/RichardLitt/standard-readme) specification.
# License
MIT AND Apache-2.0, © 2021-2023 Tableland Network Contributors