Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pinax-network/substreams-sink-clickhouse
Substreams ClickHouse Sink
https://github.com/pinax-network/substreams-sink-clickhouse
Last synced: about 2 months ago
JSON representation
Substreams ClickHouse Sink
- Host: GitHub
- URL: https://github.com/pinax-network/substreams-sink-clickhouse
- Owner: pinax-network
- License: mit
- Created: 2023-09-26T15:33:40.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-03-12T02:51:59.000Z (10 months ago)
- Last Synced: 2024-09-06T07:28:36.301Z (4 months ago)
- Language: TypeScript
- Homepage:
- Size: 1.48 MB
- Stars: 3
- Watchers: 4
- Forks: 0
- Open Issues: 3
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Substreams ClickHouse Sink
> `substreams-sink-clickhouse` is a tool to pipe in data from the blockchain into a [ClickHouse](https://clickhouse.com/) database.
## Table of contents
- [Executable binaries](#executable-binaries)
- [Features](#features)
- [Quickstart](#quickstart)
- [`.env`](#environment)
- [CLI](#cli)
## Executable [binaries](https://github.com/pinax-network/substreams-sink-clickhouse/releases)
See [releases](https://github.com/pinax-network/substreams-sink-clickhouse/releases)
```bash
$ ./substreams-sink-clickhouse
```
## [Features](/docs/features.md)
See detailed [list of features](/docs/features.md).
- Support for [entity changes](https://crates.io/crates/substreams-entity-change/)
- Support for [database changes](https://crates.io/crates/substreams-database-change)
- Table initialization
- SQL schemas
- GraphQL schemas
- Materialized View
- Automatic block metadata
- Serveless
- [Webhooks](https://github.com/pinax-network/substreams-sink-webhook) Ed25519 signatures
- No data loss
## Quickstart
Every request can also be executed via the [online UI](http://localhost:3000).
1. Associate Webhook Ed25519 public keys from [substreams-sink-webhook](https://github.com/pinax-network/substreams-sink-webhook).
```bash
$ echo "PUBLIC_KEY=,,..." >> .env
```
1. Initialize the database (_set database credentials in [environment](#environment)_)
```bash
$ ./substream-sink-clickhouse
$ curl --location 'localhost:3000/health' # --> OK
$ curl --location --request PUT "localhost:3000/init" # --> OK
```
1. Create TABLE for your EntityChanges Substreams (more details on [table initialization](/docs/features.md#table-initialization))
```bash
$ curl --location --request PUT 'localhost:3000/schema/sql?schema-url='
```
**OR**
```bash
$ curl --location --request PUT "localhost:3000/schema/sql" \
--header "Content-Type: text/plain" \
--data "CREATE TABLE foo () ENGINE=MergeTree ORDER BY();"
```
## Environment
Create a `.env` file in the root of the project.
```bash
$ cp .env.example .env
```
```env
# ClickHouse DB (optional)
HOST=http://127.0.0.1:8123
USERNAME=default
DATABASE=default
PASSWORD=
# Webhook Ed25519 public key (comma separated for multiple) (optional)
# https://github.com/pinax-network/substreams-sink-webhook
PUBLIC_KEY=
# Sink HTTP server (optional)
PORT=3000
HOSTNAME=0.0.0.0
VERBOSE=true
```
## CLI
Each field in [environment](#environment) can be overriden when starting the sink.
```bash
$ ./substreams-sink-clickhouse --help
```
```bash
Substreams Clickhouse Sink
Options:
-V, --version output the version number
-v, --verbose Enable verbose logging (choices: "true", "false", default: "true", env: VERBOSE)
-p, --port Sink HTTP server port (default: "3000", env: PORT)
--hostname Sink HTTP server hostname (default: "0.0.0.0", env: HOSTNAME)
--public-key Webhook Ed25519 public key (comma separated for multiple) (env: PUBLIC_KEY)
--host Clickhouse DB hostname (default: "http://localhost:8123", env: HOST)
--username Clickhouse DB username (default: "default", env: USERNAME)
--password Clickhouse DB password (default: "", env: PASSWORD)
--database Clickhouse DB database (default: "default", env: DATABASE)
-h, --help display help for command
```
## Database structure
See [detailed documentation](/docs/database.md)
## Development
```bash
$ bun install
$ bun dev
```