{"id":22153689,"url":"https://github.com/trudi-group/mc-crawler","last_synced_at":"2025-06-25T03:33:44.387Z","repository":{"id":38312500,"uuid":"399835522","full_name":"trudi-group/mc-crawler","owner":"trudi-group","description":"A MobileCoin network crawler. Corresponding preprint available on arXiv (https://arxiv.org/pdf/2111.12364.pdf).","archived":false,"fork":false,"pushed_at":"2024-05-01T15:28:45.000Z","size":11319,"stargazers_count":4,"open_issues_count":5,"forks_count":2,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-13T13:24:27.544Z","etag":null,"topics":["crawler","mobilecoin","rust"],"latest_commit_sha":null,"homepage":"","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/trudi-group.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2021-08-25T13:48:25.000Z","updated_at":"2024-02-14T14:17:08.000Z","dependencies_parsed_at":"2024-02-09T17:51:26.862Z","dependency_job_id":"b76207e0-c8cf-4ec1-849d-f0dacf130afd","html_url":"https://github.com/trudi-group/mc-crawler","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/trudi-group/mc-crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trudi-group%2Fmc-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trudi-group%2Fmc-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trudi-group%2Fmc-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trudi-group%2Fmc-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/trudi-group","download_url":"https://codeload.github.com/trudi-group/mc-crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/trudi-group%2Fmc-crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261798581,"owners_count":23211214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","mobilecoin","rust"],"created_at":"2024-12-02T01:36:34.049Z","updated_at":"2025-06-25T03:33:44.335Z","avatar_url":"https://github.com/trudi-group.png","language":"Rust","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mc-crawler - A [MobileCoin](https://github.com/mobilecoinfoundation/mobilecoin) Network Crawler\n\n[![Rust](https://camo.githubusercontent.com/5782bcc58a7786e9a7d00e2cf45937db8a2598232d9524ec9dcd149c7218671b/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f527573742d50726f6772616d6d696e672532304c616e67756167652d626c61636b3f7374796c653d666c6174266c6f676f3d72757374)](www.rust-lang.org)\n![MIT](https://img.shields.io/badge/license-MIT-blue.svg)\n[![CI](https://github.com/wiberlin/mc-crawler/actions/workflows/test.yml/badge.svg)](https://github.com/wiberlin/mc-crawler/actions/workflows/test.yml)\n[![codecov](https://codecov.io/gh/wiberlin/mc-crawler/branch/main/graph/badge.svg?token=GxUhfuKEoA)](https://codecov.io/gh/wiberlin/mc-crawler)\n[![dependency status](https://deps.rs/repo/github/wiberlin/mc-crawler/status.svg)](https://deps.rs/repo/github/wiberlin/mc-crawler)\n\nThis binary crawls the MobileCoin network and optionally provides 2 JSONs in [stellarbeat.io](https://stellarbeat.io) format.\n\nThe crawler communicates with the validator nodes using RPCs provided by the [mc-consensus-api](https://github.com/mobilecoinfoundation/mobilecoin/tree/master/consensus/api), and asks each new node for the last consensus message it broadcast to the other validators.\nThe response of the gRPC contains, among other information, the queried node's quorum set which in turn contains other validators that the crawler may have not yet seen.\n\nThe crawler, therefore, only finds validators (no watcher nodes), and will not find nodes that are not included in any validator's quorum set.\n\nThe Nodes-JSON contains the following data about every found node:\n\n    - Hostname\n    - Port\n    - Quorum Set\n    - Public Key\n    - Connectivity status\n    - (When available) IP-based Geolocation data, i.e. country and ISP\n\nThe Crawl Report contains the same data as the Nodes-JSON in addition to metadata about the crawl such as the duration and a timestamp.\n\nWe maintain an HTTP API at [https://api.crawler.mc.trudi.group/v1](https://api.crawler.mc.trudi.group/v1) with crawl data dating back to 23\u003csup\u003erd\u003c/sup\u003e August 2021.\n\nThe API returns a single JSON file of the crawl recorded at the requested timestamp which must be provided as an RFC 3339 and ISO 8601 date and time string.\n\nThe most recent data can be retrieved by not passing a timestamp; the oldest with [https://api.crawler.mc.trudi.group/v1/2021-08-23T20:00:00.007Z](https://api.crawler.mc.trudi.group/v1/2021-08-23T20:00:00.007Z).\n\n## 1. Required tools\n\n   - [Rust](https://www.rust-lang.org)\n        - Install: https://www.rust-lang.org/tools/install\n   - Rust's `nightly-2022-04-29` compiler which will be downloaded and installed automatically\n   - Protobuf compiler `protoc` which can be built from source or installed using a package manager or , e.g.\n\n        ``` apt install -y protobuf-compiler ``` on Ubuntu\n\n## 2. Environment Variables\nSome of the crates used in this library need the Intel SGX environment variables\n`SGX_MODE` and `IAS_MODE`.\n\nYou can set them in your terminal like below or pass them when [building the binary](#build).\n\n```\nexport SGX_MODE=SW IAS_MODE=DEV\n```\n\nHaving set the environment variables, the SGX variables do not need to be passed whenever\na call to a cargo subcommand is made.\n\nContinue to the [section on running the crawler](#run).\n\n## 3. Crawling the Network\n\n### Build\n`SGX_MODE=SW IAS_MODE=DEV cargo build --release`\n\n    - The environment variables are only necessary if you skipped step 2.\n    - The initial compilation will take several minutes due to some of the dependencies used in this project.\n\n### Run\n\n`SGX_MODE=SW IAS_MODE=DEV cargo run --release [-- path_to_bootstrap_file --output output_directory --debug --fbas --complete]`\n\n    - The environment variables are only necessary if you skipped step 2.\n    - Nodes that should be used to bootstrap the crawler are read from text file in which each node's URI is stored on a separate line. The default is \"bootstrap.txt\".\n    - The default output directory is set to \"crawl_data\".\n    - The crawler optionally writes a JSON with the FBAS discovered during the crawl when \"fbas\" is passed.\n    - The crawler optionally writes a JSON with additional data about the crawl when \"complete\" is passed.\n    - Debug level messages are suppressed by default.\n      Passing --debug results in more verbose terminal output during the crawl.\n\n## 4. Analysing the crawl data using the fbas_analyzer\nThe results presented in the paper can all be reproduced using the data obtained from the crawler and the [fbas_analyzer](https://github.com/wiberlin/fbas_analyzer).\n\nRefer to its documentation for installation instructions before proceeding.\n\nBelow are some example commands: (see `target/release/fbas_analyzer -h` for more analysis options)\n\n### Find all minimal quorums, minimal blocking sets and minimal splitting sets and output metrics about the sizes of the node sets.\n`target/release/fbas_analyzer -adp mobilecoin_nodes_completed_manually_2021-08-02.json`\n\n### Find the same sets as above, but merge by organisations\n`target/release/fbas_analyzer -adp mobilecoin_nodes_completed_manually_2021-08-02.json --merge-by-org mobilecoin_organisations_2021-08-02_created_manually.json`\n\n### Find the same sets as above, but merge by ISPs\n`target/release/fbas_analyzer -adp mobilecoin_nodes_completed_manually_2021-08-02.json --merge-by-isp`\n\n### Find the same sets as above, but merge by countries and output lists of node lists (instead of metrics)\n`target/release/fbas_analyzer -ap mobilecoin_nodes_completed_manually_2021-08-02.json --merge-by-country`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrudi-group%2Fmc-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftrudi-group%2Fmc-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftrudi-group%2Fmc-crawler/lists"}