{"id":20394520,"url":"https://github.com/migalabs/goteth","last_synced_at":"2026-02-25T12:30:48.866Z","repository":{"id":37863512,"uuid":"485828923","full_name":"migalabs/goteth","owner":"migalabs","description":"Ethereum Chain metrics indexer (CL and EL): duties, rewards, balances, and many more","archived":false,"fork":false,"pushed_at":"2025-04-08T18:23:19.000Z","size":14897,"stargazers_count":20,"open_issues_count":11,"forks_count":9,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T12:12:46.508Z","etag":null,"topics":["cosensus-layer","ethereum","execution-layer","indexer","metrics"],"latest_commit_sha":null,"homepage":"https://ethseer.io/?network=mainnet","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/migalabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":"AUTHORS.md","dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-04-26T14:47:56.000Z","updated_at":"2025-04-07T17:36:26.000Z","dependencies_parsed_at":"2023-12-22T16:38:43.819Z","dependency_job_id":"f687aa8c-115a-4835-9bb0-f96a11ebbcf0","html_url":"https://github.com/migalabs/goteth","commit_stats":null,"previous_names":["cortze/eth-cl-state-analyzer","cortze/eth2-state-analyzer","migalabs/goteth"],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/migalabs%2Fgoteth","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/migalabs%2Fgoteth/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/migalabs%2Fgoteth/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/migalabs%2Fgoteth/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/migalabs","download_url":"https://codeload.github.com/migalabs/goteth/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248565082,"owners_count":21125418,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cosensus-layer","ethereum","execution-layer","indexer","metrics"],"created_at":"2024-11-15T03:53:21.171Z","updated_at":"2026-02-25T12:30:48.854Z","avatar_url":"https://github.com/migalabs.png","language":"Go","readme":"# GotEth\n\nGotEth is a go-written client that indexes all validator-related duties, parameters and transactions from Ethereum's consensus and execution layers.\n\nThe client indexes all the validator/epoch related metrics into a set of clickhouse tables which later on can be used to monitor the performance of validators in the beaconchain. See the [docs/tables.md](https://github.com/migalabs/goteth/blob/master/docs/tables.md) for more information on the tables indexed by Goteth.\n\nThis tool has been used to power the [migalabs.io](https://migalabs.io/) public dashboard.\n\n## Prerequisites\n\nTo use the tool, the following requirements need to be installed in the machine:\n\n- [go](https://go.dev/doc/install) preferably on its 1.21 version or above. Go also needs to be executable from the terminal.\n- Clickhouse DB\n- Access to an Ethereum consensus archival node (we have only tested using lighthouse in archival mode, other clients/configs might not work). IMPORTANT: Goteth requires the `/eth/v2/debug/beacon/states` endpoint enabled. To be able to fetch blob sidecars, the `--supernode` flag must be enabled in Lighthouse after Fulu hardfork.\n- Access to an Ethereum execution node (optional)\n- Access to a Clickhouse server database (use native port, usually 9000)\n\n## Cloning\n\nGoteth uses a fork of [github.com/attestantio/go-relay-client](https://github.com/attestantio/go-relay-client) as a git submodule. In order to be able to run goteth, you will need to clone the submodule as well with: `--recurse-submodules` flag.\n\n```\ngit clone --recurse-submodules https://github.com/migalabs/goteth.git\n```\n\n## Installation\n\nThe repository provides a Makefile that will take care of all your problems.\n\nTo compile locally the client, just type the following command at the root of the directory:\n\n```\nmake build\n```\n\nOr if you prefer to install the client locally type:\n\n```\nmake install\n```\n\n## Metrics: database tables\n\n- block: downloads withdrawals, blocks and block rewards\n- epoch: download epoch metrics, proposer duties, validator last status,\n- rewards: persists validator rewards metrics to database (activates epoch metrics)\n- api_rewards: block rewards (consensus layer) are hard to calculate, but they can be downloaded from the Beacon API. However, keep in mind this takes a few seconds per block when not at the head (not recommended for backfilling). Without this, reward cannot be compared to max_reward when a validator is a proposer (32/1000k validators in an epoch). It depends on the Lighthouse API and we have registered some cases where the block reward was not returned.\n- transactions: requests transaction receipts from the execution layer, eth1 deposits and blob sidecars from consensus layer (activates block metrics)\n\nGo to [docs/tables.md](https://github.com/migalabs/goteth/blob/master/docs/tables.md) for more information on the tables indexed by Goteth.\n\n### Table Sizes\n\nData from mainnet, may 2025.\n\n- `t_validator_rewards_summary`: 1 month of data: `68GB`\n- `t_validator_rewards_aggregations`: The size of `t_validator_rewards_summary` divided by `GOTETH_REWARDS_AGGREGATION_EPOCHS`. For example, if you set `GOTETH_REWARDS_AGGREGATION_EPOCHS=10`, the size of this table will be `6.8GB`.\n- `t_transactions`: Since merge: `405GB`\n- Rest of tables: `10GB`\n\nMost tables in Goteth use the ClickHouse `ReplacingMergeTree` engine. For optimal operation, ClickHouse requires free disk space equal to the full size of each table to perform background optimizations and deletions. For example, if the rewards table occupies 68GB, you must have at least 68GB of free disk space available to safely delete rows (such as when using the validator window script). Insufficient free space may prevent these operations from completing successfully, further complicating the situation.\n\n### Validator rewards aggregation\n\nThe tool can aggregate the rewards of the validators in the `t_validator_rewards_summary` table. This is done by aggregating the rewards of the last `GOTETH_REWARDS_AGGREGATION_EPOCHS` epochs. The aggregation is done by summing up the columns of each validator in the last `GOTETH_REWARDS_AGGREGATION_EPOCHS` epochs and storing the result in the `t_validator_rewards_aggregations` table.\n\nIt can be very useful when monitoring rewards over a long period of time, without having to worry about the size of the `t_validator_rewards_summary` table, if combined with the [`val-window` command](#validator-rewards-window). Please note that `GOTETH_REWARDS_AGGREGATION_EPOCHS` must be set to a value greater than 1 to be enabled and also be lower than `GOTETH_VAL_WINDOW_NUM_EPOCHS` to avoid data loss.\n\n## Download mode\n\n- Historical: this mode loops over slots between `initSlot` and `finalSlot`, which are configurable. Once all slots have been analyzed, the tool finishes the execution.\n- Finalized: `initSlot` and `finalSlot` are ignored. The tool starts the historical mode from the database last slot to the current head (beacon node) and then follows the chain head. To do this, the tool subscribes to `head` events. See [here](https://ethereum.github.io/beacon-APIs/#/Events/eventstream) for more information.\n\n## Running the tool\n\nTo execute the tool, you can simply modify the `.env` file with your own configuration.\n\n_Running the tool (configurable in the `.env` file)_:\n\n```\ndocker compose up goteth\n```\n\n_Available Commands_:\n\n```\nCOMMANDS:\n   blocks   analyze the Beacon Block of a given slot range\n   val-window Removes old rows from the validator rewards table according to given parameters\n   help, h  Shows a list of commands or help for one command\n```\n\n_Available Options (configurable in the `.env` file)_\n\n```\n\nBlocks\nOPTIONS:\n   --bn-endpoint value     beacon node endpoint (to request the Beacon Blocks)\n   --el-endpoint value \t   execution node endpoint (to request the Transaction Receipts, optional)\n   --init-slot value       init slot from where to start (default: 0)\n   --final-slot value      init slot from where to finish (default: 0)\n   --rewards-aggregation-epochs value  Number of epochs to aggregate rewards (default: 1 (no aggregation))\n   --log-level value       log level: debug, warn, info, error\n   --db-url value          example: clickhouse://beaconchain:beaconchain@localhost:9000/beacon_states?x-multi-statement=true\n   --workers-num value     example: 3 (default: 4)\n   --db-workers-num value  example: 3 (default: 4)\n   --download-mode value   example: historical,finalized. Default: finalized\n   --metrics value         example: epoch,block,rewards,transactions,api_rewards,blob_sidecars. Empty for all (default: epoch,block)\n   --prometheus-port value Port on which to expose prometheus metrics (default: 9081)\n   --max-request-retries value         Number of retries to make when a request fails. For head mode it shouldn't be higher than 3-4, for historical its recommended to be higher (default: 3)\n   --beacon-contract-address value     Beacon contract address. Can be 'mainnet', 'holesky', 'sepolia' or directly the contract address in format '0x...' (default: mainnet)\n   --help, -h              show help (default: false)\n```\n\n### Validator Rewards Window\n\nThe validator rewards table can get large in the database (see [Table Sizes](#table-sizes)), storing rewards for epochs which might not be relevant anymore to the user. We have developed a subcommand of the tool which maintains the last n epochs of rewards data in the database, prunning from the defined threshold backwards. So, one can configure the tool to maintain the last 100 epochs of data in the database, while prunning the rest.\nThe window only affects the `t_validator_rewards_summary` table.\n\nSimply configure `GOTETH_VAL_WINDOW_NUM_EPOCHS` variable and run\n\n```\ndocker compose up val-window\n```\n\n## Database migrations\n\nIn case you encounter any issue with the database, you can force the database version using the golang-migrate command line. Please refer [here](https://github.com/golang-migrate/migrate) for more information.\nMore specifically, one could clean the migrations by forcing the version with \u003cbr\u003e\n`migrate -path / -database \"clickhouse://host:port?username=user\u0026password=password\u0026database=clicks\u0026x-multi-statement=true\" force \u003ccurrent_version\u003e` \u003cbr\u003e\nIf specific upgrades or downgrades need to be done manually, one could do this with \u003cbr\u003e\n`migrate -path database/migration/ -database \"clickhouse://host:port?username=user\u0026password=password\u0026database=clicks\u0026x-multi-statement=true\" -verbose up`\n\n# Migrating from `v2` to `v3` (Postgres to Clickhouse)\n\nDuring `v3.0.0` we migrated our database system from PostgreSQL to Clickhouse.\nIf you wish to migrate your existing database, please follow [this](https://migalabs.notion.site/PostgreSQL-to-Clickhouse-migration-611a52a457824cd494d701773365f62f) guide.\n\n# Maintainers\n\n@santi1234567\n\n# Contributing\n\nThe project is open for everyone to contribute!\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmigalabs%2Fgoteth","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmigalabs%2Fgoteth","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmigalabs%2Fgoteth/lists"}