{"id":21531981,"url":"https://github.com/chainbound/valtrack","last_synced_at":"2025-03-17T19:28:20.807Z","repository":{"id":240018682,"uuid":"792752640","full_name":"chainbound/valtrack","owner":"chainbound","description":"An Ethereum validator crawler","archived":false,"fork":false,"pushed_at":"2024-05-22T19:37:14.000Z","size":7982,"stargazers_count":1,"open_issues_count":13,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-05-22T20:44:31.784Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/chainbound.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-04-27T13:31:22.000Z","updated_at":"2024-05-27T22:57:48.814Z","dependencies_parsed_at":"2024-05-22T20:40:56.521Z","dependency_job_id":null,"html_url":"https://github.com/chainbound/valtrack","commit_stats":null,"previous_names":["chainbound/valtrack"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chainbound%2Fvaltrack","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chainbound%2Fvaltrack/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chainbound%2Fvaltrack/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/chainbound%2Fvaltrack/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/chainbound","download_url":"https://codeload.github.com/chainbound/valtrack/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":244096087,"owners_count":20397335,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-24T02:18:24.108Z","updated_at":"2025-03-17T19:28:20.784Z","avatar_url":"https://github.com/chainbound.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Valtrack\n\n\u003e Valtrack is a suite of tools aimed at geo-locating and tracking Ethereum validators.\n\nSample output in CSV: [validator_metadata_events_sample.csv](validator_metadata_events_sample.csv)\n\n## Getting Started\n\nTo get started with Valtrack, you need to clone the repository and install the dependencies.\n\n### 1. Clone the Repository\n\n```shell\ngit clone git@github.com:chainbound/valtrack.git\n```\n\n### 2. Install Dependencies and Build\n\nValtrack:\n\n```shell\ngo mod download\ngo build\n```\n\nNATS:\n\n-   https://docs.nats.io/running-a-nats-service/introduction/installation\n\n### 3. Run the Application\n\nValtrack consists of 3 main components: the **sentry**, the **consumer** and a NATS JetStream server. The sentry is responsible for discovering and tracking Ethereum validators, while the consumer consumes the data published by the sentry, processed it and stores it in a database (Parquet files for now). Read more under [Architecture](#architecture).\n\n#### Sentry\n\nTo run the sentry:\n\n```shell\n./valtrack --nats-url nats://localhost:4222 sentry\n```\n\n#### Consumer\n\n```shell\n./valtrack --nats-url nats://localhost:4222 consumer\n```\n\n#### NATS JetStream\n\nWe provide an example configuration file for the NATS server in [server/nats-server.conf](server/nats-server.conf). To run the NATS server with JetStream enabled, you can run the following command:\n\n```shell\nnats-server --jetstream --config server/nats-server.conf\n```\n\nThis will create a `data` directory in the current working directory with all the JetStream data.\n\n\u003cdetails\u003e\n\u003csummary\u003eThis should print this help text\u003c/summary\u003e\n\n```text\nNAME:\n   valtrack - Ethereum consensus validator tracking tool\n\nUSAGE:\n   valtrack [global options] command [command options] [arguments...]\n\nCOMMANDS:\n   sentry    run the sentry node\n   consumer  run the consumer\n   help, h   Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --log-level value, -l value  log level (default: \"info\")\n   --nats-url value, -n value   NATS server URL (needs JetStream) (default: \"nats://localhost:4222\")\n   --help, -h                   show help\n```\n\n\u003c/details\u003e\n\n## Architecture\n\n### Beacon Sentry\n\nThe Beacon Sentry is a service that crawls the Ethereum discv5 DHT and discovers nodes and listens for incoming connections.\nIt tries to connect to any discovered peers and performs a handshake for their [`Status`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#status) and [`MetaData`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/p2p-interface.md#metadata).\n\nIn parallel, it will try to gauge the peer's GossipSub subscriptions, specifically to the attestation subnet topics. Once it has all of that data,\nit will publish it to the NATS Jetstream server.\n\nPeers are stored in a peerstore that will periodically (every epoch) run through this process again. This allows us to get multiple data points over time which will provide more accuracy in determining the number of validators attached to a beacon node.\n\nTODO: The beacon sentry should later store estimations on the number of validators attached to a beacon node. It should then expose it over an API.\n\n### Consumer\n\nConsumer is a service which consumes the sentry data from the NATS Jetstream server and stores it in parquet file (database soon). Maintains 3 tables:\n\n-   `discovery_events`: contains the discovery events of the sentry\n-   `metadata_events`: contains the metadata events of the sentry\n-   `validator_metadata_events`: a derived table from the metadata events, which contains data points of validators\n\n### NATS Server\n\n[NATS](https://docs.nats.io/nats-concepts/what-is-nats) is a message oriented middleware. Valtrack uses NATS Jetstream which enables message persistence funcionalities.\n\n#### Installation\n\nCheck the official NATS documentation for NATS installation instructions [DOCS](https://docs.nats.io/using-nats/nats-tools/nats_cli)\n\nTo run the NATS Jetstream server:\n\n```shell\nnats-server --jetstream\n```\n\n#### Stream Configuration\n\nIt is important to create stream with the right configuration according to the requirements. [DOCS](https://docs.nats.io/nats-concepts/jetstream/streams#configuration)\n\nValtrack uses the following configuration:\n\n```golang\njetstreamCfg := jetstream.StreamConfig{\n\t\tName:      \"EVENTS\",\n\t\tRetention: jetstream.InterestPolicy,\n\t\tSubjects:  []string{\"events.metadata_received\", \"events.peer_discovered\"},\n\t}\n```\n\n-   RetentionPolicy is set to InterestPolicy, which means that the messages are retained based on the consumer interest in the messages. The messages are retained until they're acknowledged by all the consumers. If there are no consumers, the messages are not retained. [DOCS](https://docs.nats.io/nats-concepts/jetstream/streams#retentionpolicy)\n-   The subjects are the NATS subjects where the sentry data is published.\n\n#### Consumer Configuration\n\nConsumer configuration [DOCS](https://docs.nats.io/nats-concepts/jetstream/consumers#configuration)\nValtrack uses the following configuration:\n\n```golang\nconsumerCfg := jetstream.ConsumerConfig{\n\t\tName:        fmt.Sprintf(\"consumer-%s\", uniqueID),\n\t\tDurable:     fmt.Sprintf(\"consumer-%s\", uniqueID),\n\t\tDescription: \"Consumes valtrack events\",\n\t\tAckPolicy:   jetstream.AckExplicitPolicy,\n\t}\n```\n\n-   Durable is set, which means that the consumer will be binded until explicitly deleted.\n-   AckPolicy is set to AckExplicitPolicy, which means that the consumer has to explicitly acknowledge the message. [DOCS](https://docs.nats.io/nats-concepts/jetstream/consumers#ackpolicy)\n\n### API Usage\n\nThe consumer will expose an API to query the data stored in the database. The API will be a REST API.\n\nPublic access (regular user, no key required):\n\n```shell\ncurl http://localhost:8080/validators\n```\n\nAdmin access (requires an API key):\n\n```shell\ncurl -H \"X-API-KEY: \u003cAPI-KEY\u003e\" http://localhost:8080/validators\n```\n\nAPI keys can be added in the `api_keys.txt` file.\n\n## Credits\n\nShoutout to the following projects for inspiration and reference:\n\n-   [Hermes](https://github.com/probe-lab/hermes)\n-   [Armiarma](https://github.com/migalabs/armiarma/)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchainbound%2Fvaltrack","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fchainbound%2Fvaltrack","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fchainbound%2Fvaltrack/lists"}