https://github.com/streamingfast/substreams-sink-mongodb

https://github.com/streamingfast/substreams-sink-mongodb

Last synced: about 2 months ago
JSON representation

• Host: GitHub
• URL: https://github.com/streamingfast/substreams-sink-mongodb
• Owner: streamingfast
• License: apache-2.0
• Created: 2022-06-20T16:54:29.000Z (about 2 years ago)
• Default Branch: develop
• Last Pushed: 2023-08-30T17:55:30.000Z (10 months ago)
• Last Synced: 2024-04-01T15:10:32.495Z (3 months ago)
• Language: Go
• Size: 255 KB
• Stars: 2
• Watchers: 3
• Forks: 2
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Lists

• awesome-substreams - MongoDB Sink

README

        
# Substreams MongoDB Sink

This is a command line tool to quickly sync a substream with a MongoDB database.

### Running It

1) Your substream needs to implement a `map` that has an output type of `proto:substreams.database.v1.DatabaseChanges`.

   By convention, we name the `map` module `db_out`. The [substreams-data-change](https://github.com/streamingfast/substreams-database-change) crate, contains the rust objects.

2) Create a schema for the database changes

   For the moment, the only supported types are:

   - INTEGER

   - DOUBLE

   - BOOLEAN

   - TIMESTAMP

   - NULL

   - DATE

   - STRING (default value for mongodb)

   The schema must be json file and its path will be passed to the sink `run` command.

Here is an example of a schema:

```json

{

  "pair": {

    "created_at" : "timestamp",

    "block_num": "integer"

  }

}

```

> Note: any field which is of type string does not need to be declared in the schema since it will be automatically considered as a string.

3) Run the sink

| Note: to connect to substreams you will need an authentication token, follow this [guide](https://substreams.streamingfast.io/reference-and-specs/authentication) |

|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|

Usage:

```shell

Runs  extractor code

Usage:

  substreams-sink-mongodb run       [:] [flags]

Flags:

  -h, --help        help for run

  -k, --insecure    Skip certificate validation on GRPC connection

  -p, --plaintext   Establish GRPC connection in plaintext

Global Flags:

      --delay-before-start duration   [OPERATOR] Amount of time to wait before starting any internal processes, can be used to perform to maintenance on the pod before actually letting it starts

      --metrics-listen-addr string    [OPERATOR] If non-empty, the process will listen on this address for Prometheus metrics request(s) (default "localhost:9102")

      --pprof-listen-addr string      [OPERATOR] If non-empty, the process will listen on this address for pprof analysis (see https://golang.org/pkg/net/http/pprof/) (default "localhost:6060")

```

Example:

```shell

go install ./cmd/substreams-sink-mongodb

substreams-sink-mongodb run \

mongodb://root:root@localhost:27017 \

tokens \

./schema.json \

mainnet.eth.streamingfast.io:443 \

./substreams-v0.0.1.spkg \

db_out

```