https://github.com/duneanalytics/node-indexer
Ingest blockchain data into Dune
https://github.com/duneanalytics/node-indexer
Last synced: 12 months ago
JSON representation
Ingest blockchain data into Dune
- Host: GitHub
- URL: https://github.com/duneanalytics/node-indexer
- Owner: duneanalytics
- License: mit
- Created: 2024-05-30T13:40:37.000Z (about 2 years ago)
- Default Branch: main
- Last Pushed: 2025-02-20T12:53:38.000Z (over 1 year ago)
- Last Synced: 2025-06-10T06:43:29.326Z (12 months ago)
- Language: Go
- Homepage:
- Size: 4.58 MB
- Stars: 18
- Watchers: 2
- Forks: 4
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Blockchain node indexer
A program that indexes blockchain data into http://dune.com by connecting directly to an RPC node.
# Limitations
This program works with EVM compatible blockchains, doing direct, EVM-specific JSON-RPC calls to the Node RPC endpoint.
# How to use:
There are only 3 required arguments for running the indexer:
1. DUNE_API_KEY: Your Dune API Key, you can get this at: https://dune.com/settings/api
1. BLOCKCHAIN_NAME: The name of the blockchain as configured on Dune (for example: "ethereum" blockchain)
1. RPC_NODE_URL: The URL of the NODE RPC endpoint, for example: https://sepolia.optimism.io/
For more details see the configuration options section below.
## Docker container
You can run our [public container image on DockerHub](https://hub.docker.com/r/duneanalytics/node-indexer) as such:
```bash
docker run -e BLOCKCHAIN_NAME='foo' -e RPC_NODE_URL='http://localhost:8545' -e DUNE_API_KEY='your-key-here' duneanalytics/node-indexer
```
## Binary executable
You can also just build and run a binary executable after cloning this repository:
Build the binary for your OS:
```bash
$ make build
$ BLOCKCHAIN_NAME='foo' RPC_NODE_URL='http://localhost:8545' DUNE_API_KEY='your-key-here' LOG=debug ./indexer
```
Or run it directly with `go run`:
```bash
$ go run cmd/main.go --blockchain-name foo ...
```
## Configuration options
You can see all the configuration options by using the `--help` argument:
```bash
docker run duneanalytics/node-indexer --help
```
Also, we mention some of the options here:
### Log level
The `log` flag (environment variable `LOG`) controls the log level. Use `--log debug`/`LOG=debug` to emit more logs than the default `info` level. To emit less logs, use `warn`, or `error` (least).
### Tuning RPC concurrency
The flag `--rpc-concurrency` (environment variable `RPC_CONCURRENCY`) specifies the number of threads (goroutines) to run concurrently to perform RPC node requests. See `--help` for up to date default value.
### Tuning for throughput
Throughput depends on: latency & request rate between RPC <-> Node Indexer <--> DuneAPI and can be tuned via a combination of:
1. RPC_CONCURRENCY, higher values feed more blocks into the node indexer to process
1. MAX_BATCH_SIZE, higher values send more blocks per request to DuneAPI
1. BLOCK_SUBMIT_INTERVAL, the interval at which blocks to DuneAPI
See `--help` for up to date default values.
### RPC poll interval
The flag `--rpc-poll-interval` (environment variable `RPC_POLL_INTERVAL`) specifies the duration to wait before checking
if the RPC node has a new block. Default is `300ms`.
### Adding extra HTTP headers to RPC requests
If you wish to add HTTP headers to RPC requests you can do so by using the flag `--rpc-http-header` (once per header).
```
go run cmd/main.go ... --rpc-http-header header1:value1 --rpc-http-header header2:value2`
```
Or with the environment variable `RPC_HTTP_HEADERS='header1:value1|header2:value2|...'`, i.e. a `|` separated list of pairs,
where each pair is separated by `:` (make sure to quote the full string to avoid creating a pipe).
```
docker run --env RPC_HTTP_HEADERS='header1:value1|header2:value2' ... duneanalytics/node-indexer:
```
## Metrics
The process exposes prometheus metrics which you can scrape and explore: `http://localhost:2112/metrics`