{"id":14956218,"url":"https://github.com/dennis-tra/nebula","last_synced_at":"2026-06-09T09:00:51.431Z","repository":{"id":39603326,"uuid":"379835217","full_name":"dennis-tra/nebula","owner":"dennis-tra","description":"🌌  An agnostic network crawler exposing comprehensive peer information and network topology information.","archived":false,"fork":false,"pushed_at":"2026-03-25T13:27:38.000Z","size":15982,"stargazers_count":372,"open_issues_count":3,"forks_count":50,"subscribers_count":9,"default_branch":"main","last_synced_at":"2026-04-14T11:31:20.129Z","etag":null,"topics":["cid","crawler","filecoin","golang","hacktoberfest","ipfs","libp2p"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/dennis-tra.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null},"funding":{"github":"dennis-tra"}},"created_at":"2021-06-24T07:07:06.000Z","updated_at":"2026-04-08T16:50:26.000Z","dependencies_parsed_at":"2023-10-31T14:31:04.363Z","dependency_job_id":"1aa8dbb3-0421-4bff-b04c-028dd4cd1b3a","html_url":"https://github.com/dennis-tra/nebula","commit_stats":{"total_commits":517,"total_committers":10,"mean_commits":51.7,"dds":0.0541586073500967,"last_synced_commit":"c41cfec78b8b70e61ce21e3405e80f40faa29cd2"},"previous_names":["dennis-tra/nebula-crawler"],"tags_count":11,"template":false,"template_full_name":null,"purl":"pkg:github/dennis-tra/nebula","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennis-tra%2Fnebula","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennis-tra%2Fnebula/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennis-tra%2Fnebula/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennis-tra%2Fnebula/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/dennis-tra","download_url":"https://codeload.github.com/dennis-tra/nebula/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/dennis-tra%2Fnebula/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34098952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cid","crawler","filecoin","golang","hacktoberfest","ipfs","libp2p"],"created_at":"2024-09-24T13:12:33.061Z","updated_at":"2026-06-09T09:00:51.425Z","avatar_url":"https://github.com/dennis-tra.png","language":"Go","funding_links":["https://github.com/sponsors/dennis-tra","https://www.buymeacoffee.com/dennistra"],"categories":[],"sub_categories":[],"readme":"![Nebula Logo](./docs/nebula-logo.svg)\n\n# Nebula\n\n[![standard-readme compliant](https://img.shields.io/badge/readme%20style-standard-brightgreen.svg)](https://github.com/RichardLitt/standard-readme)\n[![go test](https://github.com/dennis-tra/nebula/actions/workflows/pull_request_main.yml/badge.svg)](https://github.com/dennis-tra/nebula/actions/workflows/pull_request_main.yml)\n[![readme nebula](https://img.shields.io/badge/readme-Nebula-blueviolet)](README.md)\n[![GitHub license](https://img.shields.io/github/license/dennis-tra/nebula)](https://github.com/dennis-tra/nebula/blob/main/LICENSE)\n\nNebula is an agnostic network crawler. It starts with a set of bootstrap peers,\nasks them for other peers in the network and recursively repeats the process\nuntil all peers in the network have been contacted.\n\nCurrently, Nebula supports the following networks:\n\n- [IPFS](https://ipfs.network) - [_Amino DHT_](https://blog.ipfs.tech/2023-09-amino-refactoring/)\n- [Bitcoin](https://bitcoin.org/) | [Litecoin](https://litecoin.org/) | [Dogecoin](https://dogecoin.com/) (alpha)\n- [Ethereum](https://ethereum.org/en/) - [_Consensus Layer (discv5)_](https://ethereum.org/uz/developers/docs/networking-layer/#consensus-discovery) | [_Execution Layer (discv4)_](https://ethereum.org/uz/developers/docs/networking-layer/#discovery)\n- [Optimism](https://www.optimism.io/) compatible chains\n- [Portal](https://www.portal.network/) - (_alpha - [wire protocol](https://github.com/ethereum/portal-network-specs/blob/master/portal-wire-protocol.md) not implemented_)\n- [Filecoin](https://filecoin.io)\n- [Polkadot](https://polkadot.network/) - [_Kusama_](https://kusama.network/) | [_Rococo_](https://substrate.io/developers/rococo-network/) | [_Westend_](https://wiki.polkadot.network/docs/maintain-networks#westend-test-network)\n- [Avail](https://www.availproject.org/) - [_Mainnet_](https://docs.availproject.org/docs/networks#mainnet) | [_Turing_](https://docs.availproject.org/docs/networks#turing-testnet) | _\u003csmall\u003eLight Client + Full Node versions\u003c/small\u003e_\n- [Celestia](https://celestia.org/) - [_Mainnet_](https://blog.celestia.org/celestia-mainnet-is-live/) | [_Mocha_](https://docs.celestia.org/nodes/mocha-testnet) | [_Arabica_](https://github.com/celestiaorg/celestia-node/blob/9c0a5fb0626ada6e6cdb8bcd816d01a3aa5043ad/nodebuilder/p2p/bootstrap.go#L40)\n- [Pactus](https://pactus.org)\n- [Dria](https://dria.co/)\n- [Gnosis](https://www.gnosis.io/)\n- [Monero](https://www.getmonero.org/)\n- ... your network? Get in touch [team@probelab.io](mailto:team@probelab.io).\n\n\u003e You can run `nebula networks` to get a list of all supported networks.\n\nNebula supports the JSON, Postgres and ClickHouse storage backends. It powers\nmany graphs on [probelab.io](https://probelab.io/networks) for most of the\nsupported networks above. Check out the \"Topology\" sections for any of the\nnetworks to get an idea. Gnosis uses Nebula to source its Network metrics [here](https://metrics.gnosischain.com/).\n\n### Commercial Offering\n\nIn addition to the open-source version, [ProbeLab](https://probelab.io/) maintains a commercial offering\nof Nebula which includes:\n\n- **GossipSub Topic Tracking** — detailed tracking of GossipSub topic subscriptions across the network.\n- **Ethereum Handshake Analysis** — in-depth Ethereum protocol handshake data collection.\n- **Monero Network Crawling** — specialized peer discovery for the [Monero](https://www.getmonero.org/) network.\n- **Algorand Network Crawling** — specialized peer discovery for the [Algorand](https://algorand.co/) network.\n\nGoing forward, new feature development will continue exclusively as part of the\ncommercial offering. The open-source version will remain available and\nmaintained for existing functionality, but all new capabilities will be part of\nthe commercial offering.\n\nFor pricing and access, please reach out to [team@probelab.io](mailto:team@probelab.io).\n\n### Awards\n\nThe crawler was:\n\n- 🏆 _awarded a prize in the [DI2F Workshop hackathon](https://research.protocol.ai/blog/2021/decentralising-the-internet-with-ipfs-and-filecoin-di2f-a-report-from-the-trenches/)._ 🏆\n- 🎓 _used for the ACM SigCOMM'22 paper [Design and Evaluation of IPFS: A Storage Layer for the Decentralized Web](https://research.protocol.ai/publications/design-and-evaluation-of-ipfs-a-storage-layer-for-the-decentralized-web/trautwein2022.pdf)_ 🎓\n\n\n\nYou can find a demo on YouTube: [Nebula: A Network Agnostic DHT Crawler](https://www.youtube.com/watch?v=QDgvCBDqNMc) 📺\n\n![Screenshot from a Grafana dashboard](./docs/grafana-screenshot.png)\n\n\u003csmall\u003e_Grafana Dashboard is not part of this repository_\u003c/small\u003e\n\n## Table of Contents\n\n- [Table of Contents](#table-of-contents)\n- [Project Status](#project-status)\n- [Usage](#usage)\n- [Install](#install)\n  - [From source](#from-source)\n- [How does it work?](#how-does-it-work)\n  - [`crawl`](#crawl)\n  - [`monitor`](#monitor)\n  - [`resolve`](#resolve)\n- [Development](#development)\n  - [Database](#database)\n  - [Tests](#tests)\n- [Report](#report)\n- [Related Efforts](#related-efforts)\n- [Demo](#demo)\n- [Maintainers](#maintainers)\n- [Contributing](#contributing)\n- [Support](#support)\n- [Other Projects](#other-projects)\n- [License](#license)\n\n## Project Status\n\nThe crawler is powering critical [IPFS](https://ipfs.tech) [Amino DHT](https://blog.ipfs.tech/2023-09-amino-refactoring/) [KPIs](https://de.wikipedia.org/wiki/Key-Performance-Indicator), used for [Weekly IPFS Reports](https://github.com/probe-lab/network-measurements/tree/main/reports) as well as for many metrics on [`probelab.io`](https://probelab.io).\nThe `main` branch will contain the latest changes and should not be considered stable. The latest stable release that is production ready is version [2.4.0](https://github.com/dennis-tra/nebula/releases/tag/2.4.0).\n\n## Install\n\n### Precompile Binaries\n\nHead over to the release section and download binaries from the [latest stable release](https://github.com/dennis-tra/nebula/releases).\n\n### From source\n\n```shell\ngit clone https://github.com/dennis-tra/nebula\ncd nebula\njust build\n```\n\nNow you should find the `nebula` executable in the `dist` subfolder.\n\n## Usage\n\nNebula is a command line tool and provides the `crawl` sub-command.\n\n### Dry-Run\n\nTo simply crawl the IPFS Amino DHT network run:\n\n```shell\nnebula --dry-run crawl\n```\n\n\u003e [!NOTE]\n\u003e For backwards compatibility reasons IPFS is the default if no network is specified\n\nThe crawler can store its results as JSON documents, in a [Postgres](https://www.postgresql.org/), or in a [Clickhouse](https://clickhouse.com/) database -\nthe `--dry-run` flag prevents it from doing any of it. Nebula will just print a\nsummary of the crawl at the end instead. For the IPFS network, a crawl takes ~5-10 min depending on\nyour internet connection. You can also specify the network you want to crawl by\nappending, e.g., `--network FILECOIN` and limit the number of peers to crawl by\nproviding the `--limit` flag with the value of, e.g., `1000`. Example:\n\n```shell\nnebula --dry-run crawl --network FILECOIN --limit 1000\n```\n\nTo find out which other network values are supported, you can run:\n\n```shell\nnebula networks\n```\n\n### JSON Output\n\nTo store crawl results as JSON files provide the `--json-out` command line flag like so:\n\n```shell\nnebula --json-out ./results/ crawl\n```\n\nAfter the crawl has finished, you will find the JSON files in the `./results/` subdirectory.\n\nWhen providing only the `--json-out` command line flag you will see that the\n`*_neighbors.json` document is empty. This document would contain the full\nrouting table information of each peer in the network which is quite a bit of\ndata (~250MB for the Amino DHT as of April '23) and is therefore disabled by\ndefault\n\n### Track Routing Table Information\n\nTo populate the document, you'll need to pass the `--neighbors` flag to\nthe `crawl` subcommand.\n\n```shell\nnebula --json-out ./results/ crawl --neighbors\n```\n\nThe routing table information forms a graph and graph visualization tools often\noperate with [adjacency lists](https://en.wikipedia.org/wiki/Adjacency_list). To convert the `*_neighbors.json` document\nto an adjacency list, you can use [`jq`](https://stedolan.github.io/jq/) and the following command:\n\n```shell\njq -r '.NeighborIDs[] as $neighbor | [.PeerID, $neighbor] | @csv' ./results/2025-02-16T14:32_neighbors.json \u003e ./results/2025-02-16T14:32_neighbors.csv\n```\n\n### Postgres\n\nIf you want to store the information in a proper database, you could run `just start-postgres` to start a local postgres instance via docker in the background and run Nebula like:\n\n```shell\nnebula --db-user nebula_local --db-name nebula_local crawl --neighbors\n```\n\nAt this point, you can also start Nebula's monitoring process, which would periodically probe the discovered peers to track their uptime. Run in another terminal:\n\n```shell\nnebula --db-user nebula_local --db-name nebula_local monitor\n```\n\nWhen Nebula is configured to store its results in a postgres database, then it also tracks session information of remote peers. A session is one continuous streak of uptime (see below).\n\nHowever, this is not implemented for all supported networks. The [ProbeLab](https://probelab.network) team is using the monitoring feature for the IPFS, Celestia, Filecoin, and Avail networks. Most notably, the Ethereum discv4/discv5 and Bitcoin monitoring implementations still need work.\n\n---\n\nThere are a few more command line flags that are documented when you run`nebula --help` and `nebula crawl --help`:\n\n## How does it work?\n\n### `crawl`\n\nThe `crawl` sub-command starts by connecting to a set of bootstrap nodes and then\nrequesting the information of other peers in the network using the network-native\ndiscovery protocol. For most supported networks these are several Kademlia\n`FIND_NODE` RPCs. For Bitcoin-related networks it's a `getaddr` RPC.\n\nFor Kademlia-based networks Nebula constructs the routing tables (kademlia _k_-buckets)\nof the remote peer based on its [`PeerID`](https://docs.libp2p.io/concepts/peer-id/). Then `nebula` builds\nrandom `PeerIDs` with common prefix lengths (CPL) that fall in each of the peers' buckets, and asks if it knows any peers that are\ncloser (XOR distance) to the ones `nebula` just generated. This will effectively yield a list of all `PeerIDs` that a peer has\nin its routing table. The process repeats for all found peers until `nebula` does not find any new `PeerIDs`.\n\n\u003e [!TIP]\n\u003e You can use the `crawl` sub-command with the global `--dry-run` option that skips any database operations.\n\nCommand line help page:\n\n```text\nNAME:\n   nebula crawl - Crawls the entire network starting with a set of bootstrap nodes.\n\nUSAGE:\n   nebula crawl [command options]\n\nOPTIONS:\n   --addr-dial-type value                               Which type of addresses should Nebula try to dial (private, public, any) (default: \"public\") [$NEBULA_CRAWL_ADDR_DIAL_TYPE]\n   --addr-track-type value                              Which type addresses should be stored to the database (private, public, any) (default: \"public\") [$NEBULA_CRAWL_ADDR_TRACK_TYPE]\n   --bootstrap-peers value [ --bootstrap-peers value ]  Comma separated list of multi addresses of bootstrap peers (default: default IPFS) [$NEBULA_CRAWL_BOOTSTRAP_PEERS, $NEBULA_BOOTSTRAP_PEERS]\n   --limit value                                        Only crawl the specified amount of peers (0 for unlimited) (default: 0) [$NEBULA_CRAWL_PEER_LIMIT]\n   --neighbors                                          Whether to persist all k-bucket entries of a particular peer at the end of a crawl. (default: false) [$NEBULA_CRAWL_NEIGHBORS]\n   --network nebula networks                            Which network should be crawled. Presets default bootstrap peers and protocol. Run: nebula networks for more information. (default: \"IPFS\") [$NEBULA_CRAWL_NETWORK]\n   --protocols value [ --protocols value ]              Comma separated list of protocols that this crawler should look for [$NEBULA_CRAWL_PROTOCOLS, $NEBULA_PROTOCOLS]\n   --workers value                                      How many concurrent workers should dial and crawl peers. (default: 1000) [$NEBULA_CRAWL_WORKER_COUNT]\n\n   Network Specific Configuration:\n\n   --check-exposed               IPFS/AMINO: Whether to check if the Kubo API is exposed. Checking also includes crawling the API. (default: false) [$NEBULA_CRAWL_CHECK_EXPOSED]\n   --keep-enr                    ETHEREUM_CONSENSUS: Whether to keep the full ENR. (default: false) [$NEBULA_CRAWL_KEEP_ENR]\n   --udp-response-timeout value  ETHEREUM_EXECUTION: The response timeout for UDP requests in the disv4 DHT (default: 3s) [$NEBULA_CRAWL_UDP_RESPONSE_TIMEOUT]\n\n```\n\n### `monitor`\n\nThe `monitor` sub-command is only implemented for libp2p based networks and with the postgres database backend.\nIt polls every 10 seconds all sessions from the database (see above) that are due to be dialed\nin the next 10 seconds (based on the `next_visit_due_at` timestamp). It attempts to dial all peers using previously\nsaved multi-addresses and updates their `session` instances accordingly if they're dialable or not.\n\nThe `next_visit_due_at` timestamp is calculated based on the uptime that `nebula` has observed for that given peer.\nIf the peer is up for a long time `nebula` assumes that it stays up and thus decreases the dial frequency aka. sets\nthe `next_visit_due_at` timestamp to a time further in the future.\n\nCommand line help page:\n\n```text\nNAME:\n   nebula monitor - Monitors the network by periodically dialing previously crawled peers.\n\nUSAGE:\n   nebula monitor [command options]\n\nOPTIONS:\n   --workers value  How many concurrent workers should dial peers. (default: 1000) [$NEBULA_MONITOR_WORKER_COUNT]\n   --network value  Which network belong the database sessions to. Relevant for parsing peer IDs and muti addresses. (default: \"IPFS\") [$NEBULA_MONITOR_NETWORK]\n   --help, -h       show help\n\n```\n\n### `resolve`\n\nThe resolve sub-command is only available when using the postgres datbaase backend. It goes through all multi addresses that are present in the database and resolves them to their respective IP-addresses. Behind one multi address can be multiple IP addresses due to, e.g., the [`dnsaddr` protocol](https://github.com/multiformats/multiaddr/blob/master/protocols/DNSADDR.md).\nFurther, it queries the GeoLite2 database from [Maxmind](https://www.maxmind.com/en/home) to extract country information about the IP addresses and [UdgerDB](https://udger.com/) to detect datacenters. The command saves all information alongside the resolved addresses.\n\nCommand line help page:\n\n```text\nNAME:\n   nebula resolve - Resolves all multi addresses to their IP addresses and geo location information\n\nUSAGE:\n   nebula resolve [command options] [arguments...]\n\nOPTIONS:\n   --udger-db value    Location of the Udger database v3 [$NEBULA_RESOLVE_UDGER_DB]\n   --batch-size value  How many database entries should be fetched at each iteration (default: 100) [$NEBULA_RESOLVE_BATCH_SIZE]\n   --help, -h          show help (default: false)\n```\n\n## Development\n\nTo develop this project, you need Go `1.23` and the following tools:\n```\ngithub.com/golang-migrate/migrate/v4/cmd/migrate@v4.18.2\ngithub.com/volatiletech/sqlboiler/v4@v4.18.0\ngithub.com/volatiletech/sqlboiler/v4/drivers/sqlboiler-psql@v4.18.0\ngo.uber.org/mock/mockgen@v0.5.0\nmvdan.cc/gofumpt@v0.7.0\n```\n\nTo install the necessary tools you can run `just tools`. This will use the `go install` command to download and install the tools into your `$GOPATH/bin` directory. So make sure you have it in your `$PATH` environment variable.\n\n### Database\n\nYou need a running Postgres or ClickHouse instance to persist and/or read the crawl results.\nRun `just start-postgres` or `just start-clickhouse` respectively\nor use one of the following commands:\n\n```shell\n# for postgres\ndocker run --rm -d --name nebula-postgres-local -p 5432:5432 -e POSTGRES_DB=nebula_local -e POSTGRES_USER=nebula_local -e POSTGRES_PASSWORD=password_local postgres:14\n\n# for clickhouse\ndocker run --rm -d --name nebula-clickhouse-local -p 8123:8123 -p 9000:9000 -e CLICKHOUSE_DB=nebula_local -e CLICKHOUSE_USER=nebula_local -e CLICKHOUSE_PASSWORD=password_local clickhouse/clickhouse-server:24.12\n```\n\nThen you can connect to the database with:\n\n```shell\njust repl-postgres\n# or\njust repl-clickhouse\n```\n\nTo stop the containers:\n\n```shell\njust stop-postgres\n# or\njust stop-clickhouse\n```\n\nfor convenience there are also the `just restart-postgres` and `just restart-clickhouse` recipes.\n\n\u003e [!TIP]\n\u003e You can use the `crawl` sub-command with the global `--dry-run` option that skips any database operations or store the results as JSON files with the `--json-out` flag.\n\nThe default database settings for local development are:\n\n```toml\nName     = \"nebula_local\"\nPassword = \"password_local\"\nUser     = \"nebula_local\"\nHost     = \"localhost\"\n# postgres\nPort     = 5432\n# clickhouse\nPort     = 9000\n```\n\nMigrations are applied automatically when `nebula` starts and successfully establishes a database connection.\n\nTo run them manually you can run:\n\n```shell\n# Up migrations\njust migrate-postgres up\njust migrate-clickhouse up\n\n# Down migrations\njust migrate-postgres down\njust migrate-clickhouse down\n\n# Generate the ORMs with SQLBoiler (only postgres)\njust models # runs: sqlboiler\n```\n\n```shell\n# Create new migration\n# postgres\nmigrate create -ext sql -dir db/migrations/pg -seq some_migration_name\n\n# clickhouse\nmigrate create -ext sql -dir db/migrations/chlocal -seq some_migration_name\n```\n\n\u003e [!NOTE]\n\u003e Make sure to adjust the `chlocal` migration and copy it over to the `chcluster` folder. In a clustered clickhouse deployment the table engines need to be prefixed with `Replicated`, like `ReplicatedMergeTree` as opposed to just `MergeTree`.\n\n### Tests\n\nTo run the tests you need a running test database instance. The following command\nstarts test postgres and clickhouse containers, runs the tests and tears them\ndown again:\n\n```shell\njust test\n```\n\nThe test database containers won't interfere with other local containers as\nall names etc. are suffixed with `_test` as opposed to `_local`.\n\nTo speed up running database tests you can do the following:\n\n```shell\njust start-postgres test\njust start-clickhouse test\n```\n\nThen run the plain tests (without starting database containers):\n```shell\njust test-plain\n```\n\nEventually, stop the containers again:\n\n```shell\njust stop-postgres test\njust stop-clickhouse test\n```\n\n## Release Checklist\n\n- [ ] Merge everything into `main`\n- [ ] Create a new tag with the new version\n- [ ] Push tag to GitHub\n\nThis will trigger the [`goreleaser.yml`](./.github/workflows/goreleaser.yml) workflow which pushes creates a new _draft_ release in GitHub.\n\n## Related Efforts\n\n- [`wiberlin/ipfs-crawler`](https://github.com/wiberlin/ipfs-crawler) - A crawler for the IPFS network, code for their paper ([arXiv](https://arxiv.org/abs/2002.07747)).\n- [`adlrocha/go-libp2p-crawler`](https://github.com/adlrocha/go-libp2p-crawler) - Simple tool to crawl libp2p networks resources\n- [`libp2p/go-libp2p-kad-dht`](https://github.com/libp2p/go-libp2p-kad-dht/tree/master/crawler) - Basic crawler for the Kademlia DHT implementation on go-libp2p.\n- [`migalabs/armiarma`](https://github.com/migalabs/armiarma) - Armiarma is a Libp2p open-network crawler with a current focus on Ethereum's CL network\n- [`migalabs/eth-light-crawler`](https://github.com/migalabs/eth-light-crawler) - Ethereum light crawler by [@cortze](https://github.com/cortze).\n\n## Demo\n\nThe following presentation shows a ways to use Nebula by showcasing crawls of the Amino, Celestia, and Ethereum DHT's:\n\n[![Nebula: A Network Agnostic DHT Crawler - Dennis Trautwein](https://img.youtube.com/vi/QDgvCBDqNMc/0.jpg)](https://www.youtube.com/watch?v=QDgvCBDqNMc)\n\n## Networks\n\n\u003e [!NOTE]\n\u003e This section is work-in-progress and doesn't include information about all networks yet.\n\nThe following sections document our experience with crawling the different networks.\n\n### Ethereum Execution (disv4)\n\nUnder the hood Nebula uses packages from [`go-ethereum`](https://github.com/ethereum/go-ethereum) to facilitate peer\ncommunication. Mostly, Nebula relies on the [discover package](https://github.com/ethereum/go-ethereum/tree/master/p2p/discover).\nHowever, we made quite a few changes to the implementation that can be found in\nour fork of `go-ethereum` [here](https://github.com/probe-lab/go-ethereum/tree/nebula) in the `nebula` branch.\n\nMost notably, the custom changes include:\n\n- export of internal constants, functions, methods and types to customize their behaviour or call them directly\n- changes to the response matcher logic. UDP packets won't be forwarded to all matchers. This was required so that\n  concurrent requests to the same peer don't lead to unhandled packets\n\nDeployment recommendations:\n\n- CPUs: 4 (better 8)\n- Memory \u003e 4 GB\n- UDP Read Buffer size \u003e1 MiB (better 4 MiB) via the `--udp-buffer-size=4194304` command line flag or corresponding environment variable `NEBULA_UDP_BUFFER_SIZE`.\n  You might need to adjust the maximum buffer size on Linux, so that the flag takes effect:\n  ```shell\n  sysctl -w net.core.rmem_max=8388608 # 8MiB\n  ```\n- UDP Response timeout of `3s` (default)\n- Workers: 3000\n\n## Maintainers\n\n[@dennis-tra](https://github.com/dennis-tra).\n\n## Contributing\n\nFeel free to dive in! [Open an issue](https://github.com/dennis-tra/nebula/issues/new) or submit PRs.\n\n## Support\n\nIt would really make my day if you supported this project through [Buy Me A Coffee](https://www.buymeacoffee.com/dennistra).\n\n## Other Projects\n\nYou may be interested in one of my other projects:\n\n- [`pcp`](https://github.com/dennis-tra/pcp) - Command line peer-to-peer data transfer tool based on [libp2p](https://github.com/libp2p/go-libp2p).\n- [`image-stego`](https://github.com/dennis-tra/image-stego) - A novel way to image manipulation detection. Steganography-based image integrity - Merkle tree nodes embedded into image chunks so that each chunk's integrity can be verified on its own.\n\n## License\n\n[Apache License Version 2.0](LICENSE) © Dennis Trautwein\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdennis-tra%2Fnebula","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fdennis-tra%2Fnebula","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fdennis-tra%2Fnebula/lists"}