https://github.com/onflow/flow-public-key-indexer
A observer service for indexing flow Accounts and by their associated Keys.
https://github.com/onflow/flow-public-key-indexer
Last synced: about 2 months ago
JSON representation
A observer service for indexing flow Accounts and by their associated Keys.
- Host: GitHub
- URL: https://github.com/onflow/flow-public-key-indexer
- Owner: onflow
- License: apache-2.0
- Created: 2022-05-05T20:00:55.000Z (about 3 years ago)
- Default Branch: main
- Last Pushed: 2024-05-28T21:49:08.000Z (about 1 year ago)
- Last Synced: 2024-05-29T07:09:42.423Z (about 1 year ago)
- Language: Go
- Size: 17 MB
- Stars: 3
- Watchers: 7
- Forks: 1
- Open Issues: 6
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Flow Public Key Indexer
A observer service for indexing flow Accounts public keys and REST service that serves up public key data.
## Run Parameters
`KEYIDX_LOGLEVEL` default: "info"
Log Level: Takes string log level value for zerolog, "debug", "info", ...
`KEYIDX_PORT` default: "8080"
Port: The port the REST service is hosted on
`KEYIDX_FLOWURL1` default: "access.mainnet.nodes.onflow.org:9000"
Flow Url: Access node endpoint blockchain data is pulled from, needs to match up with Chain Id. Up to 4 access nodes can be provided, only one is required. The access nodes are cycled through each request to get public key data.
`KEYIDX_FLOWURL2` default: none
Flow Url: Access node endpoint
`KEYIDX_FLOWURL3` default: none
Flow Url: Access node endpoint
`KEYIDX_FLOWURL4` default: none
Flow Url: Access node endpoint
`KEYIDX_CHAINID` default: "flow-mainnet"
Chain Id: target blockchain, valid values are "flow-testnet" and "flow-mainnet". Needs to match up with Flow Url
`KEYIDX_MAXACCTKEYS` default: 1000
Max Acct Keys: maximum number of keys to index per account, accounts that have more public keys that exceed max do not get indexed. Accounts are logged out that exceed the max
`KEYIDX_BATCHSIZE` default: 50000
Batch Size: max number of accounts in a batch sent to cadence script that access node executes. Cadence script can exceed execution if accounts have a lot of keys
`KEYIDX_IGNOREZEROWEIGHT` default: true
Ignore Zero Weight: tells the cadence script to ignore public keys with zero weight. These keys will not be indexed
`KEYIDX_IGNOREREVOKED` default: false
Ignore Revoked: tells the cadence script to ignore public keys that have been revoked. These keys will not be indexed
`KEYIDX_WAITNUMBLOCKS` default: 200
Wait Num Blocks: number of blocks to wait before running an incremental data load
`KEYIDX_BLOCKPOLINTERVALSEC` default: 180
Block Pol Interval Sec: number of seconds to wait before checking current block height to determine to run an incremental data load
`KEYIDX_SYNCDATAPOLINTERVALMIN` default: 1
Sync Data Polling Interval: number of minutes to wait between sync data operations
`KEYIDX_SYNCDATASTARTINDEX` default: 30000000
Sync Data Start Index: starting block height for sync operations
`KEYIDX_MAXBLOCKRANGE` default: 600
Max Block Range: number of blocks that will trigger a bulk load if services falls behind
`KEYIDX_FETCHSLOWDOWNMS` default: 500
Fetch Slowdown: milliseconds to wait between fetch operations
`KEYIDX_PURGEONSTART` default: false
Purge on Start: When changing the data structure or want to clear the database and start from scratch change this variable to true
`KEYIDX_ENABLESYNCDATA` default: true
Enable Sync Data: Run this service as a sync service. It's possible to run this service only as rest service
`KEYIDX_ENABLEINCREMENTAL` default: true
Enable Incremental: Enable incremental updates of the database
## PostgreSQL configurations
`KEYIDX_POSTGRESQLHOST` default: "localhost"
`KEYIDX_POSTGRESQLPORT` default: 5432
`KEYIDX_POSTGRESQLUSERNAME` default: "postgres"
`KEYIDX_POSTGRESQLPASSWORD` not required, no default
`KEYIDX_POSTGRESQLDATABASE` default: "keyindexer"
`KEYIDX_POSTGRESQLSSL` default: true
`KEYIDX_POSTGRESQLLOGQUERIES` default: false
`KEYIDX_POSTGRESQLSETLOGGER` default: false
`KEYIDX_POSTGRESQLRETRYNUMTIMES` default: 30
`KEYIDX_POSTGRESQLRETRYSLEEPTIME` default: "1s"
`KEYIDX_POSTGRESQLPOOLSIZE` default: 1
`KEYIDX_POSTGRESLOGGERPREFIX` default: "keyindexer"
`KEYIDX_POSTGRESPROMETHEUSSUBSYSTEM` default: "keyindexer"
## Re-indexing Accounts
The service supports re-indexing of specific accounts by adding them to the `addressprocessing` table. This feature is useful when you need to:
- Update account information that might have changed
- Re-process accounts that may have had errors during initial indexing
- Force a refresh of specific account data
### How to Re-index Accounts
1. Add addresses to the `addressprocessing` table:
```sql
INSERT INTO addressprocessing (account)
VALUES ('0x1234...'), ('0x5678...')
ON CONFLICT (account) DO NOTHING;
```
2. The service will automatically:
- Pick up these addresses during the next bulk processing cycle
- Re-fetch their public key information
- Update the database with any changes
- Remove the addresses from the `addressprocessing` table once processed
### Processing Behavior
- Addresses in the `addressprocessing` table are processed in batches (defined by `KEYIDX_BATCHSIZE`)
- Duplicate addresses are automatically ignored (ON CONFLICT DO NOTHING)
- Processing occurs during the bulk load cycle (controlled by `KEYIDX_SYNCDATAPOLINTERVALMIN`)
- After successful processing, addresses are automatically removed from the `addressprocessing` table
### Related Configuration Parameters
- `KEYIDX_BATCHSIZE` default: 50000
Controls how many addresses are processed in each batch
- `KEYIDX_SYNCDATAPOLINTERVALMIN` default: 1
Determines how frequently the service checks for new addresses to process
## How to Run
Since this is a golang service there are many ways to run it. Below are two ways to run this service
### Command line
```go run .```
### Docker
Configuration: Run docker in default 10 gig memory size. Reducing the running memory size reduces performance, the lowest is 6 gig, bulk sync and public key query responses are reasonable compared to running with more memory.
Create a docker container
```docker build -t key-indexer .```
``` need to configure to use postgresql ```
This service stores public key data and needs persistent storage in postgresql
Run the docker and map the rest service port
Notice that environmental variables can be passed in. See variables above
```docker run -p 8888:8080 --env KEYIDX_POSTGRESQLHOST=localhost```
To see the logs of the container, get the container id
```docker container ls```
View the containers logs
```docker logs ```
## REST service
`Endpoints`
* `GET /key/{public key}`
note: public key is in base64 format
serves up json object
```json
{
"publicKey": string, // public key string in base64
"accounts": [
{
"address": string, // Flow account address
"keyId": int, // Key index in the account
"weight": int, // Key weight for signing
"sigAlgo": int, // Signing algorithm identifier
"hashAlgo": int, // Hashing algorithm identifier
"signing": string, // Human-readable signing algorithm name
"hashing": string // Human-readable hashing algorithm name
}
]
}
```
sigAlgo - signing: 1 - ECDSA_P256, 2 - ECDSA_secp256k1, 3 - ECDSA_P384, 4 - ECDSA_P521, 5 - ECDSA_secp256r1
hashAlgo - hashing: 1 - SHA2_256, 2 - SHA3_256, 3 - SHA2_384, 4 - SHA3_384, 5 - SHA2_512, 6 - SHA3_512
* `GET /status`
note: this endpoint gives ability to see if the server is active and updating
```json
{
"Count": int, // Number of unique public keys indexed
"LoadedToBlock": int, // Last processed block height
"CurrentBlock": int // Current block height on the Flow network
}
```