{"id":19336572,"url":"https://github.com/subsquid/eth-archive","last_synced_at":"2025-04-23T01:30:49.957Z","repository":{"id":78408013,"uuid":"488055773","full_name":"subsquid/eth-archive","owner":"subsquid","description":"Stage 1 implementation of Squid Archives for EVM networks","archived":true,"fork":false,"pushed_at":"2024-05-23T08:37:40.000Z","size":1216,"stargazers_count":19,"open_issues_count":7,"forks_count":5,"subscribers_count":6,"default_branch":"master","last_synced_at":"2025-03-13T01:33:52.315Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/subsquid.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-03T02:33:28.000Z","updated_at":"2024-05-23T23:10:01.000Z","dependencies_parsed_at":"2024-05-23T09:41:00.012Z","dependency_job_id":"3d06372e-2a02-4e28-bd7c-4eb86bc7c301","html_url":"https://github.com/subsquid/eth-archive","commit_stats":null,"previous_names":[],"tags_count":55,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Feth-archive","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Feth-archive/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Feth-archive/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/subsquid%2Feth-archive/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/subsquid","download_url":"https://codeload.github.com/subsquid/eth-archive/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250352177,"owners_count":21416454,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-10T03:11:40.645Z","updated_at":"2025-04-23T01:30:49.618Z","avatar_url":"https://github.com/subsquid.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# eth-archive\n\n\u003e [!CAUTION]\n\u003e V1 Subsquid Archives are deprecated. Use [Subsquid Network](https://docs.subsquid.io/subsquid-network/) instead.\n\n## Usage\n\n### Requirements\n\n- Latest stable version of Rust. Code is only tested and built using the latest stable version of rustc.\n\n#### Install cargo make\n\n- `cargo install --force cargo-make`\n\n#### Build and run ingester\n\n- `makers ingester`\n\n#### Build and run worker\n\n- `makers worker`\n\n#### Run with s3\n- configure `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`\n- `makers ingester-s3`\n- `makers worker-s3`\n\n#### Helm charts\n\nReference Helm charts for the worker and the ingester services can be found in the `charts` folder.\n\n\n\n#### CLI\n\u003cdetails\u003e\n\n\u003csummary\u003eIngester\u003c/summary\u003e\n  \n```\nUsage: eth-archive-ingester.exe [OPTIONS] --data-path \u003cDATA_PATH\u003e --request-timeout-secs \u003cREQUEST_TIMEOUT_SECS\u003e --connect-timeout-ms \u003cCONNECT_TIMEOUT_MS\u003e --block-batch-size \u003cBLOCK_BATCH_SIZE\u003e --http-req-concurrency \u003cHTTP_REQ_CONCURRENCY\u003e --best-block-offset \u003cBEST_BLOCK_OFFSET\u003e --max-blocks-per-file \u003cMAX_BLOCKS_PER_FILE\u003e --max-txs-per-file \u003cMAX_TXS_PER_FILE\u003e --max-logs-per-file \u003cMAX_LOGS_PER_FILE\u003e --max-row-groups-per-file \u003cMAX_ROW_GROUPS_PER_FILE\u003e\n\nOptions:\n      --data-path \u003cDATA_PATH\u003e\n          Path to store parquet files\n      --request-timeout-secs \u003cREQUEST_TIMEOUT_SECS\u003e\n          Http request timeout in seconds\n      --connect-timeout-ms \u003cCONNECT_TIMEOUT_MS\u003e\n          Http connect timeout in milliseconds\n      --block-batch-size \u003cBLOCK_BATCH_SIZE\u003e\n          Number of blocks a single get block batch request will cover\n      --http-req-concurrency \u003cHTTP_REQ_CONCURRENCY\u003e\n          Number of concurrent requests to make\n      --best-block-offset \u003cBEST_BLOCK_OFFSET\u003e\n          Offset from tip of the chain (to avoid rollbacks)\n      --rpc-urls \u003cRPC_URLS\u003e\n\n      --get-receipts\n          Get transaction receipts. This fills the transaction.status field. Requires eth_getBlockReceipts to be available on the RPC API\n      --wait-between-rounds \u003cWAIT_BETWEEN_ROUNDS\u003e\n          Wait this amount of seconds between rounds\n      --max-rpc-endpoint-best-block-diff \u003cMAX_RPC_ENDPOINT_BEST_BLOCK_DIFF\u003e\n          An rpc endpoint is considered behind and excluded if max_best_block - rpc_endpoint.best_block \u003e max_rpc_endpoint_best_block_diff [default: 5]\n      --target-rpc-endpoint \u003cTARGET_RPC_ENDPOINT\u003e\n          The real target rpc endpoint. This is useful when using the rpc_proxy\n      --num-tries \u003cNUM_TRIES\u003e\n\n      --secs-between-tries \u003cSECS_BETWEEN_TRIES\u003e\n          [default: 3]\n      --max-blocks-per-file \u003cMAX_BLOCKS_PER_FILE\u003e\n          Maximum number of blocks per parquet file\n      --max-txs-per-file \u003cMAX_TXS_PER_FILE\u003e\n          Maximum number of transactions per file\n      --max-logs-per-file \u003cMAX_LOGS_PER_FILE\u003e\n          Maximum number of logs per parquet file\n      --max-row-groups-per-file \u003cMAX_ROW_GROUPS_PER_FILE\u003e\n          Maximum number of row groups per parquet file\n      --parquet-page-size \u003cPARQUET_PAGE_SIZE\u003e\n          Page size for parquet files in bytes. Defaults to 1MB\n      --max-pending-folder-writes \u003cMAX_PENDING_FOLDER_WRITES\u003e\n          Maximum number of pending folder writes. This effects maximum memory consumption [default: 8]\n      --folder-write-concurrency \u003cFOLDER_WRITE_CONCURRENCY\u003e\n          [default: 8]\n      --metrics-addr \u003cMETRICS_ADDR\u003e\n          Address to serve prometheus metrics from [default: 127.0.0.1:8181]\n      --s3-src-bucket \u003cS3_SRC_BUCKET\u003e\n          S3 bucket name to initial sync from\n      --s3-src-format-ver \u003cS3_SRC_FORMAT_VER\u003e\n          Source data format version\n      --local-src-path \u003cLOCAL_SRC_PATH\u003e\n          Local file system path to sync from\n      --local-src-format-ver \u003cLOCAL_SRC_FORMAT_VER\u003e\n          Local source data format version\n      --s3-endpoint \u003cS3_ENDPOINT\u003e\n\n      --s3-bucket-name \u003cS3_BUCKET_NAME\u003e\n\n      --s3-bucket-region \u003cS3_BUCKET_REGION\u003e\n\n      --s3-sync-interval-secs \u003cS3_SYNC_INTERVAL_SECS\u003e\n\n      --s3-concurrency \u003cS3_CONCURRENCY\u003e\n```\n  \n\u003c/details\u003e\n \n\u003cdetails\u003e\n\n\u003csummary\u003eWorker\u003c/summary\u003e\n  \n```\nUsage: eth-archive-worker.exe [OPTIONS] --db-path \u003cDB_PATH\u003e --request-timeout-secs \u003cREQUEST_TIMEOUT_SECS\u003e --connect-timeout-ms \u003cCONNECT_TIMEOUT_MS\u003e --block-batch-size \u003cBLOCK_BATCH_SIZE\u003e --http-req-concurrency \u003cHTTP_REQ_CONCURRENCY\u003e --best-block-offset \u003cBEST_BLOCK_OFFSET\u003e --max-resp-body-size \u003cMAX_RESP_BODY_SIZE\u003e --resp-time-limit \u003cRESP_TIME_LIMIT\u003e\n\nOptions:\n      --db-path \u003cDB_PATH\u003e\n          Database path\n      --data-path \u003cDATA_PATH\u003e\n          Path to read parquet files from\n      --request-timeout-secs \u003cREQUEST_TIMEOUT_SECS\u003e\n          Http request timeout in seconds\n      --connect-timeout-ms \u003cCONNECT_TIMEOUT_MS\u003e\n          Http connect timeout in milliseconds\n      --block-batch-size \u003cBLOCK_BATCH_SIZE\u003e\n          Number of blocks a single get block batch request will cover\n      --http-req-concurrency \u003cHTTP_REQ_CONCURRENCY\u003e\n          Number of concurrent requests to make\n      --best-block-offset \u003cBEST_BLOCK_OFFSET\u003e\n          Offset from tip of the chain (to avoid rollbacks)\n      --rpc-urls \u003cRPC_URLS\u003e\n\n      --get-receipts\n          Get transaction receipts. This fills the transaction.status field. Requires eth_getBlockReceipts to be available on the RPC API\n      --wait-between-rounds \u003cWAIT_BETWEEN_ROUNDS\u003e\n          Wait this amount of seconds between rounds\n      --max-rpc-endpoint-best-block-diff \u003cMAX_RPC_ENDPOINT_BEST_BLOCK_DIFF\u003e\n          An rpc endpoint is considered behind and excluded if max_best_block - rpc_endpoint.best_block \u003e max_rpc_endpoint_best_block_diff [default: 5]\n      --target-rpc-endpoint \u003cTARGET_RPC_ENDPOINT\u003e\n          The real target rpc endpoint. This is useful when using the rpc_proxy\n      --num-tries \u003cNUM_TRIES\u003e\n\n      --secs-between-tries \u003cSECS_BETWEEN_TRIES\u003e\n          [default: 3]\n      --server-addr \u003cSERVER_ADDR\u003e\n          Address to be used for running the server [default: 127.0.0.1:8080]\n      --initial-hot-block-range \u003cINITIAL_HOT_BLOCK_RANGE\u003e\n          Initial hot block range. If None, hot blocks will start from 0\n      --max-resp-body-size \u003cMAX_RESP_BODY_SIZE\u003e\n          Query stops as soon as the response body size in megabytes reaches this number. Response body might be bigger than this amount of MBs\n      --max-concurrent-queries \u003cMAX_CONCURRENT_QUERIES\u003e\n          Maximum number of concurrent queries [default: 32]\n      --max-parquet-query-concurrency \u003cMAX_PARQUET_QUERY_CONCURRENCY\u003e\n          Maximum number of threads per query to use to query parquet folders [default: 8]\n      --resp-time-limit \u003cRESP_TIME_LIMIT\u003e\n          Response time limit in milliseconds. The query will stop and found data will be returned if the request takes more than this amount of time to handle\n      --db-query-batch-size \u003cDB_QUERY_BATCH_SIZE\u003e\n          Size of each database query. Database queries are batched because we don't want to query the entire db at once [default: 200]\n      --s3-endpoint \u003cS3_ENDPOINT\u003e\n\n      --s3-bucket-name \u003cS3_BUCKET_NAME\u003e\n\n      --s3-bucket-region \u003cS3_BUCKET_REGION\u003e\n\n      --s3-sync-interval-secs \u003cS3_SYNC_INTERVAL_SECS\u003e\n\n      --s3-concurrency \u003cS3_CONCURRENCY\u003e\n```\n  \n\u003c/details\u003e\n\n## Architecture\n\n\u003cimg src=\"https://user-images.githubusercontent.com/8627422/225257301-a1ff18bb-57ee-4e7a-960e-62ac83afda7d.png\" width=\"75%\"\u003e\n\n## API Docs\n\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\u003ccode\u003eGET\u003c/code\u003e \u003ccode\u003e\u003cb\u003e/height\u003c/b\u003e\u003c/code\u003e \u003ccode\u003e(get height of the archive)\u003c/code\u003e\u003c/summary\u003e\n\n##### Example Response\n\n```json\n{\n  \"height\": 16576911\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\u003ccode\u003ePOST\u003c/code\u003e \u003ccode\u003e\u003cb\u003e/query\u003c/b\u003e\u003c/code\u003e \u003ccode\u003e(query logs and transactions)\u003c/code\u003e\u003c/summary\u003e\n\n##### Query Fields\n\n- **fromBlock**: Block number to start from (inclusive).\n- **toBlock**: Block number to end on (inclusive) (optional). If this is not given, the query will go on for a fixed amount of time or until it reaches the height of the archive.\n- **logs.address**: Array of addresses to query for. A log will be included in the response if the log's address matches any of the addresses given in the query. (null or empty array means any address).\n- **log.topics**: Array of arrays of topics. Outer array has an element for each topic an EVM log can have. Each inner array represents possible matching values for a topic. For example topics[2] is an array of possible values that should match the log's third topic or the log won't be included in the response. Empty arrays match everything.\n- **transactions.from** and **transactions.to**: Array of addresses that should match the transaction's `to` field or the transaction's `from`. If none of these match, the transaction won't be included in the response. If both are null or empty array, any address will pass.\n- **transactions.sighash**: Array of values that should match first four bytes of the transaction input. null or empty array means any value will pass.\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\n\n##### Example Request\n\u003c/summary\u003e\n\n```json\n{\n  \"fromBlock\": 14495889,\n  \"toBlock\": 14495889,\n  \"logs\": [\n    {\n      \"address\": [\n        \"0x3883f5e181fccaF8410FA61e12b59BAd963fb645\"\n      ],\n      \"topics\": [\n        [\n          \"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef\"\n        ]\n      ],\n      \"fieldSelection\": {\n        \"block\": {\n          \"parentHash\": true,\n          \"sha3Uncles\": true,\n          \"miner\": true,\n          \"stateRoot\": true,\n          \"transactionsRoot\": true,\n          \"receiptsRoot\": true,\n          \"logsBloom\": true,\n          \"difficulty\": true,\n          \"number\": true,\n          \"gasLimit\": true,\n          \"gasUsed\": true,\n          \"timestamp\": true,\n          \"extraData\": true,\n          \"mixHash\": true,\n          \"nonce\": true,\n          \"totalDifficulty\": true,\n          \"baseFeePerGas\": true,\n          \"size\": true,\n          \"hash\": true\n        },\n        \"transaction\": {\n          \"type\": true,\n          \"nonce\": true,\n          \"to\": true,\n          \"gas\": true,\n          \"value\": true,\n          \"input\": true,\n          \"maxPriorityFeePerGas\": true,\n          \"maxFeePerGas\": true,\n          \"yParity\": true,\n          \"chainId\": true,\n          \"v\": true,\n          \"r\": true,\n          \"s\": true,\n          \"from\": true,\n          \"blockHash\": true,\n          \"blockNumber\": true,\n          \"index\": true,\n          \"gasPrice\": true,\n          \"hash\": true,\n          \"status\": true\n        },\n        \"log\": {\n          \"address\": true,\n          \"blockHash\": true,\n          \"blockNumber\": true,\n          \"data\": true,\n          \"index\": true,\n          \"removed\": true,\n          \"topics\": true,\n          \"transactionHash\": true,\n          \"transactionIndex\": true\n        }\n      }\n    }\n  ],\n  \"transactions\": [\n    {\n      \"address\": [\n        \"0x3883f5e181fccaf8410fa61e12b59bad963fb645\"\n      ],\n      \"sighash\": [\n        \"0xa9059cbb\"\n      ],\n      \"fieldSelection\": {\n        \"block\": {\n          \"parentHash\": true,\n          \"sha3Uncles\": true,\n          \"miner\": true,\n          \"stateRoot\": true,\n          \"transactionsRoot\": true,\n          \"receiptsRoot\": true,\n          \"logsBloom\": true,\n          \"difficulty\": true,\n          \"number\": true,\n          \"gasLimit\": true,\n          \"gasUsed\": true,\n          \"timestamp\": true,\n          \"extraData\": true,\n          \"mixHash\": true,\n          \"nonce\": true,\n          \"totalDifficulty\": true,\n          \"baseFeePerGas\": true,\n          \"size\": true,\n          \"hash\": true\n        },\n        \"transaction\": {\n          \"type\": true,\n          \"nonce\": true,\n          \"to\": true,\n          \"gas\": true,\n          \"value\": true,\n          \"input\": true,\n          \"maxPriorityFeePerGas\": true,\n          \"maxFeePerGas\": true,\n          \"yParity\": true,\n          \"chainId\": true,\n          \"v\": true,\n          \"r\": true,\n          \"s\": true,\n          \"from\": true,\n          \"blockHash\": true,\n          \"blockNumber\": true,\n          \"index\": true,\n          \"gasPrice\": true,\n          \"hash\": true,\n          \"status\": true\n        },\n        \"log\": {\n          \"address\": true,\n          \"blockHash\": true,\n          \"blockNumber\": true,\n          \"data\": true,\n          \"index\": true,\n          \"removed\": true,\n          \"topics\": true,\n          \"transactionHash\": true,\n          \"transactionIndex\": true\n        }\n      }\n    }\n  ]\n}\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003e\n\n##### Example Response\n\u003c/summary\u003e\n\n```json\n{\n  \"data\": [\n    [\n      {\n        \"block\": {\n          \"parentHash\": \"0x455864413159d92478ad496a627533ce6fdd83d6ed47528b8790b96135325d64\",\n          \"sha3Uncles\": \"0x9098605a7fc9c4c70622f6458f168b9acb302e6f84d02235670b49e263a3bd13\",\n          \"miner\": \"0xea674fdde714fd979de3edf0f56aa9716b898ec8\",\n          \"stateRoot\": \"0xfb8b174c4e118d95ff5097014ed1d60c8b49094b541b1e9e4e16198963ddbdf6\",\n          \"transactionsRoot\": \"0xb8eebad6f727be7afa3d6812ff2c05adb1e3053ac15bfb6acdd0265c8ffd2cce\",\n          \"receiptsRoot\": \"0x4fa02eab7297cfbadd39d3e47cdf52f85159f4b9719621c2ec6a9b608638a482\",\n          \"logsBloom\": \"0xb43fd1cee1dfdf4b7cbef44ced7f9ebe70c8e7ddfe764cb84ba96beb352ff49d751f196fd0b87fcf3d3e5a2241f56f714a1280c30be3ed27ef7f9f7bf0b7aa7d94de4fe8ea476fea6f7364ed917bd3e3af3f7a1ec3d4f910fee067a6cabffb7f7eaadbe70e5e8a6ec4ecf6f8959b7adbec62fdbc8513855c66ea15fbacca1b64517c8f94e766be97b4fefdd8a5952ef45ae56ffbc9cad78948b32ddf6ed4b83333fdee65b97d6897fe9fe8f9ed583c4c4a90e2cbe9d5bb9f2375932c77bddd77bf08ffa3df263f777de5228103bff47b8d6df97fa79fce9b0c7a3fea0f43e73f143c388f5d732b9cb9e5fe889bd04ff31b4b84e1ffe407d222ffce85f8567aeb\",\n          \"difficulty\": \"0x2e878564548bfa\",\n          \"number\": 14495889,\n          \"gasLimit\": \"0x01ca35ef\",\n          \"gasUsed\": \"0x01c9e9fe\",\n          \"timestamp\": \"0x6246042d\",\n          \"extraData\": \"0x617369612d65617374322d3134\",\n          \"mixHash\": \"0xbfcdb3683cbefbfe178e0334acf34005cd20dc06440f311cd9270236b6dea952\",\n          \"nonce\": \"17265704492997010732\",\n          \"totalDifficulty\": \"0x0991caa08ff39c6e95a4\",\n          \"baseFeePerGas\": \"0x0c97b03dcf\",\n          \"size\": \"0x023860\",\n          \"hash\": \"0x344fc5e67555bfb42f759be7ee372fad30bcbebf780cbfb901f546683ed22517\"\n        },\n        \"transactions\": [\n          {\n            \"type\": 2,\n            \"nonce\": \"6\",\n            \"to\": \"0x3883f5e181fccaf8410fa61e12b59bad963fb645\",\n            \"gas\": \"0xd3bb\",\n            \"value\": \"0x00\",\n            \"input\": \"0xa9059cbb000000000000000000000000254d3f04543145f3a991b61675ca0bb353f669c90000000000000000000000000000000000000000000000008ac7230489e80000\",\n            \"maxPriorityFeePerGas\": \"0x4a817c80\",\n            \"maxFeePerGas\": \"0x0cff79e223\",\n            \"chainId\": 1,\n            \"v\": \"0\",\n            \"r\": \"0xbd768420f1c173f6942d201b83bc7aedef90d4b9947598a2e931525560b722ed\",\n            \"s\": \"0x78387f73fd4b6cd006eaf8deeda0857234e29740c06e01221460c7118daef348\",\n            \"from\": \"0x5bf2f6612dfc3d0d1e0c6799534228b41369d39e\",\n            \"blockHash\": \"0x344fc5e67555bfb42f759be7ee372fad30bcbebf780cbfb901f546683ed22517\",\n            \"blockNumber\": 14495889,\n            \"index\": 299,\n            \"gasPrice\": \"0x0ce231ba4f\",\n            \"hash\": \"0x8f45965dd61dc189b94306b0f13cc3338374f687d527c64c1f19c994b39ae3b2\"\n          }\n        ],\n        \"logs\": [\n          {\n            \"address\": \"0x3883f5e181fccaf8410fa61e12b59bad963fb645\",\n            \"blockHash\": \"0x344fc5e67555bfb42f759be7ee372fad30bcbebf780cbfb901f546683ed22517\",\n            \"blockNumber\": 14495889,\n            \"data\": \"0x0000000000000000000000000000000000000000000000008ac7230489e80000\",\n            \"index\": 379,\n            \"removed\": false,\n            \"topics\": [\n              \"0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef\",\n              \"0x0000000000000000000000005bf2f6612dfc3d0d1e0c6799534228b41369d39e\",\n              \"0x000000000000000000000000254d3f04543145f3a991b61675ca0bb353f669c9\"\n            ],\n            \"transactionHash\": \"0x8f45965dd61dc189b94306b0f13cc3338374f687d527c64c1f19c994b39ae3b2\",\n            \"transactionIndex\": 299\n          }\n        ]\n      }\n    ]\n  ],\n  \"archiveHeight\": 16577057,\n  \"nextBlock\": 14495890,\n  \"totalTime\": 220\n}\n```\n\u003c/details\u003e\n\n\u003c/details\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubsquid%2Feth-archive","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsubsquid%2Feth-archive","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsubsquid%2Feth-archive/lists"}