{"id":13751220,"url":"https://github.com/Keep-Reth-Strange/reth-crawler","last_synced_at":"2025-05-09T17:33:22.692Z","repository":{"id":197244527,"uuid":"698247040","full_name":"Keep-Reth-Strange/reth-crawler","owner":"Keep-Reth-Strange","description":"Reth crawler","archived":false,"fork":false,"pushed_at":"2024-02-03T16:17:03.000Z","size":240,"stargazers_count":61,"open_issues_count":6,"forks_count":8,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-03-13T07:43:49.613Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://etherclients.com/","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Keep-Reth-Strange.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null}},"created_at":"2023-09-29T13:40:41.000Z","updated_at":"2025-01-17T08:50:40.000Z","dependencies_parsed_at":null,"dependency_job_id":"8c205528-952f-413a-b512-f839dbf02fc6","html_url":"https://github.com/Keep-Reth-Strange/reth-crawler","commit_stats":null,"previous_names":["keep-reth-strange/reth-crawler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keep-Reth-Strange%2Freth-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keep-Reth-Strange%2Freth-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keep-Reth-Strange%2Freth-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Keep-Reth-Strange%2Freth-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Keep-Reth-Strange","download_url":"https://codeload.github.com/Keep-Reth-Strange/reth-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":253295849,"owners_count":21885709,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T09:00:39.180Z","updated_at":"2025-05-09T17:33:22.333Z","avatar_url":"https://github.com/Keep-Reth-Strange.png","language":"Rust","funding_links":[],"categories":["Crawlers ##"],"sub_categories":[],"readme":"# Reth-crawler\n\n![CI](https://github.com/Keep-Reth-Strange/reth-crawler/actions/workflows/ci.yml/badge.svg)\n\nReth crawler MVP!\nRelated to [this issue](https://github.com/paradigmxyz/reth/issues/4778).\n\nA sample of the peer data:\n\n```json\n{\n\n    {\n        \"enode_url\":\"enode://0bb46e5532da93328bf6964309081f9fef7a19f29d91f93004f82ecd897766a15348f6863c4f36541f632d50c7d592c6b9181e439e9ede16c7e0cd18612cd048@158.220.96.114:34310\",\"id\":\"0x0bb46e5532da93328bf6964309081f9fef7a19f29d91f93004f82ecd897766a15348f6863c4f36541f632d50c7d592c6b9181e439e9ede16c7e0cd18612cd048\",\"address\":\"158.220.96.114\",\"tcp_port\":34310,\"client_version\":\"Geth/v1.13.4-stable-3f907d6a/linux-amd64/go1.21.3\",\"eth_version\":68,\"capabilities\":[\"eth/67\",\"eth/68\",\"snap/1\"],\"chain\":\"mainnet\",\"total_difficulty\":\"17179869184\",\"best_block\":\"0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3\",\"genesis_block_hash\":\"0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3\",\"last_seen\":\"2023-11-08 15:41:04.469473084 UTC\",\"country\":\"Germany\",\"city\":\"Düsseldorf\",\"synced\":null,\"isp\":\"Contabo GmbH\"\n    }\n}\n```\n\n## How to build the crawler\n\n`reth-crawler` is stable now and it's in active development. In order to try it, follow these steps:\n\n### Clone the repository\n\n```bash\ngit clone https://github.com/Keep-Reth-Strange/reth-crawler\n```\n\n### Run it\n\nOpen the directory where it's been installed previously and build it:\n\n```bash\ncargo build -p reth-crawler\n```\n\nGo to `target/debug` folder:\n\n```bash\ncd target/debug\n```\n\nRun it:\n\n```bash\n./reth-crawler crawl\n```\n\n### Run it locally without a centralized db\n\nFor local testing there is a flag to save peers in a `peers_data.db` file (sqlite db):\n\n```bash\n./reth-crawler crawl --local-db\n```\n\n### Node endpoint\n\nNote that the crawl command assumes that a local node is running at `wss://localhost:8546`. If that is not the case, you will need to pass a node websocket url as argument, for eg.\n```\n./reth-crawler crawl --local-db --eth-rpc-url wss://ethereum.publicnode.com\n```\n\n## The API server\n\nThe API server is not necessary for crawling the network but it's very useful to serve the crawled data.\n\nRight now it's configured by default to connect to the dynamo db database used by the crawler,\nso if you have not set up an Amazon db before, it will not run.\n\nWe are now working on migrate our infra from Amazon to a new service provider and we'll also\nlet the API server be more flexible for local testing.\n\n### How to run the API server\n\nOpen the directory where it's been installed previously and build it:\n\n```bash\ncargo build -p reth-crawler-api-server\n```\n\nGo to `target/debug` folder:\n\n```bash\ncd target/debug\n```\n\nRun it:\n\n```bash\n./reth-crawler-api-server start-api-server\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKeep-Reth-Strange%2Freth-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FKeep-Reth-Strange%2Freth-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FKeep-Reth-Strange%2Freth-crawler/lists"}