Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/pinax-network/substreams-sink-victoria-metrics
Substreams Sink VictoriaMetrics
https://github.com/pinax-network/substreams-sink-victoria-metrics
streamingfast substreams thegraph
Last synced: about 2 months ago
JSON representation
Substreams Sink VictoriaMetrics
- Host: GitHub
- URL: https://github.com/pinax-network/substreams-sink-victoria-metrics
- Owner: pinax-network
- License: apache-2.0
- Created: 2023-03-29T12:55:18.000Z (almost 2 years ago)
- Default Branch: main
- Last Pushed: 2023-11-27T18:09:51.000Z (about 1 year ago)
- Last Synced: 2024-10-05T14:47:55.901Z (3 months ago)
- Topics: streamingfast, substreams, thegraph
- Language: TypeScript
- Homepage: https://www.npmjs.com/package/substreams-sink-victoria-metrics
- Size: 245 KB
- Stars: 0
- Watchers: 5
- Forks: 0
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE-APACHE
Awesome Lists containing this project
README
# [`Substreams`](https://substreams.streamingfast.io/) [`VictoriaMetrics`](https://victoriametrics.com/) CLI `Node.js`
> `substreams-sink-victoria-metrics` is a tool that allows developers to pipe data extracted from a blockchain to a VictoriaMetrics TSDB (Time Series Database).
## 📖 Documentation
### https://www.npmjs.com/package/substreams-sink-victoria-metrics
### Further resources
- [**Substreams** documentation](https://substreams.streamingfast.io)
- [**`Victoria Metrics`**](https://victoriametrics.com/)
### Protobuf
## CLI
[**Use pre-built binaries**](https://github.com/pinax-network/substreams-sink-victoria-metrics/releases)
- [ ] MacOS
- [x] Linux
- [ ] Windows
**Install** globally via npm
```
$ npm install -g substreams-sink-victoria-metrics
```
**Run**
```
$ substreams-sink-victoria-metrics run [options]
```
## Features
- [x] Export substreams results to csv files
- [x] Import from csv to Victoriametrics
## Command specific options
### Run
```bash
Usage: substreams-sink-victoria-metrics run [options]
Substreams VictoriaMetrics sink module
Options:
-e --substreams-endpoint Substreams gRPC endpoint to stream data from (env: SUBSTREAMS_ENDPOINT)
--manifest URL of Substreams package (env: MANIFEST)
--module-name Name of the output module (declared in the manifest) (env: MODULE_NAME)
-s --start-block Start block to stream from (defaults to -1, which means the initialBlock of the first module you are streaming) (default: "-1", env: START_BLOCK)
-t --stop-block Stop block to end stream at, inclusively (env: STOP_BLOCK)
-p, --params Set a params for parameterizable modules. Can be specified multiple times. (ex: -p module1=valA -p module2=valX&valY) (default: [], env: PARAMS)
--substreams-api-token API token for the substream endpoint or API key if '--auth-issue-url' is specified (default: "", env: SUBSTREAMS_API_TOKEN)
--auth-issue-url URL used to issue a token (default: "https://auth.pinax.network/v1/auth/issue", env: AUTH_ISSUE_URL)
--delay-before-start Delay (ms) before starting Substreams (default: 0, env: DELAY_BEFORE_START)
--cursor-path File path or URL to cursor lock file (default: "cursor.lock", env: CURSOR_PATH)
--http-cursor-auth Basic auth credentials for http cursor (ex: username:password) (env: HTTP_CURSOR_AUTH)
--production-mode Enable production mode, allows cached substreams data if available (default: "false", env: PRODUCTION_MODE)
--inactivity-seconds If set, the sink will stop when inactive for over a certain amount of seconds (default: 300, env: INACTIVITY_SECONDS)
--hostname The process will listen on this hostname for any HTTP and Prometheus metrics requests (default: "localhost", env: HOSTNAME)
--port The process will listen on this port for any HTTP and Prometheus metrics requests (default: 9102, env: PORT)
--metrics-labels [string...] To apply generic labels to all default metrics (ex: --labels foo=bar) (default: {}, env: METRICS_LABELS)
--collect-default-metrics Collect default metrics (default: "false", env: COLLECT_DEFAULT_METRICS)
--headers [string...] Set headers that will be sent on every requests (ex: --headers X-HEADER=headerA) (default: {}, env: HEADERS)
--final-blocks-only Only process blocks that have pass finality, to prevent any reorg and undo signal by staying further away from the chain HEAD (default: "false", env: FINAL_BLOCKS_ONLY)
--verbose Enable verbose logging (default: "false", env: VERBOSE)
--host VictoriaMetrics address to connect. (default: "http://0.0.0.0:8428")
-i --scrape-interval Scrape Interval (default: "30")
-l --labels [...string] To apply generic labels to all default metrics (ex: --labels foo=bar) (default: {})
-h, --help display help for command
```
### CSV export
```bash
Usage: substreams-sink-victoria-metrics csv export [options]
Export CSV files from Substreams
Options:
<...>
-i --scrape-interval Scrape Interval (seconds) (default: "30")
--csv-root CSV root (default: "./csv")
--folder-granular folder granular (default: 1000) (default: "1000")
--file-granular file granular (default: 100) (default: "100")
-h, --help display help for command
```
## Folder structure
Breakdown of the csv folder structure is as follow:
- [csv-root]
- [module hash]
- [block folder]
- [files]*.csv
**Example usage**
```bash
substreams-sink-victoria-metrics csv export --manifest https://github.com/pinax-network/subtivity-substreams/releases/download/v0.2.1/subtivity-antelope-v0.2.1.spkg --module-name prom_out -e https://eos.firehose.eosnation.io:9001 --cursor-file antelope1.lock -s 10000000 -t +1000000 --csv-root=./export_root_folder --folder-granular=50000 --file-granular=1000
```
### Import CSV
```bash
Usage: substreams-sink-victoria-metrics csv import [options]
Import CSV files to VictoriaMetrics
Options:
--verbose Enable verbose logging (default: false)
--host VictoriaMetrics address to connect. (default: "http://0.0.0.0:8428")
--csv-root CSV root (default: "./csv")
-l --labels [...string] To apply generic labels to all default metrics (ex: --labels foo=bar) (default: {})
-h, --help display help for command
```
**Example usage**
```bash
substreams-sink-victoria-metrics csv import --labels 'job=substivity&network=127.0.0.1&block_version=antelope&hostname=localhost&app=app1' --csv-root=import_csv_folder
```
## Performance measured
This is a comparison of relative speed between v1, v2 and the cache given a scrape interval of 30s.
| Range | Time | blk/ s| ver|
|-----------|--------|-------|----|
| 190M-195M | 38m37s | 2157 | v2
| 200M-205M | 52m48s | 1578 | v1
| 200M-205M | 39m10s | 2127 | cached
| 200M-205M | 40m32s | 2055 | cached
| 200M-205M | 33m | 2525 | cached
| 205M-210M | 42m17s | 1970 | v1
| 210M-215M | 44m38s | 1867 | v1
| 215M-220M | 45m51s | 1817 | v1
| 215M-220M | 38m9s | 2184 | cached
| 215M-220M | 38m24s | 2170 | cached
| 220M-225M | 54m21s | 1533 | v1
| 230M-235M | 37m53s | 2199 | v2
| 240M-245M | 38m29s | 2165 | v2
| 300M-305M | 43m14s | 1927 | v1
| 300M-305M | 35m23s | 2355 | cached