{"id":13728521,"url":"https://github.com/satoshipay/stellar-core-parallel-catchup","last_synced_at":"2025-05-08T00:31:56.798Z","repository":{"id":54850931,"uuid":"163327688","full_name":"satoshipay/stellar-core-parallel-catchup","owner":"satoshipay","description":"Fast sync a Stellar Core validator node including full history with parallelization","archived":false,"fork":false,"pushed_at":"2021-01-25T13:27:34.000Z","size":21,"stargazers_count":23,"open_issues_count":2,"forks_count":10,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-08-04T02:08:08.961Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/satoshipay.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-12-27T19:06:03.000Z","updated_at":"2022-10-25T19:55:07.000Z","dependencies_parsed_at":"2022-08-14T04:40:57.437Z","dependency_job_id":null,"html_url":"https://github.com/satoshipay/stellar-core-parallel-catchup","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/satoshipay%2Fstellar-core-parallel-catchup","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/satoshipay%2Fstellar-core-parallel-catchup/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/satoshipay%2Fstellar-core-parallel-catchup/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/satoshipay%2Fstellar-core-parallel-catchup/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/satoshipay","download_url":"https://codeload.github.com/satoshipay/stellar-core-parallel-catchup/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224679947,"owners_count":17351896,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-03T02:00:43.902Z","updated_at":"2024-11-14T19:31:18.229Z","avatar_url":"https://github.com/satoshipay.png","language":"Shell","funding_links":[],"categories":["Developer Resources","开发者资源"],"sub_categories":[],"readme":"# Parallel Stellar Core Catchup ⚡\n\n## Background\n\n### Goal\n\nSync a full Stellar validator node (including full history) as fast as possible.\n\n### Problem\n\nA full catchup takes weeks/months – even without publishing to an archive.\n\n### Idea\n\n * Split the big ledger into small chunks of size `CHUNK_SIZE`.\n * Run a catchup for the chunks in parallel with `WORKERS` worker processes.\n * Stitch together the resulting database and history archive.\n\n## Usage\n\n```\n./catchup.sh DOCKER_COMPOSE_FILE LEDGER_MIN LEDGER_MAX CHUNK_SIZE WORKERS\n```\n\nArguments:\n\n* `DOCKER_COMPOSE_FILE`: use `docker-compose.pubnet.yaml` for the public network (`docker-compose.testnet.yaml` for testnet).\n* `LEDGER_MIN`: smallest ledger number you want. Use `1` for doing a full sync.\n* `LEDGER_MAX`: largest ledger number you want, usually you'll want the latest one which is exposed as `core_latest_ledger` in any synced Horizon server, e.g. https://stellar-horizon.satoshipay.io/.\n* `CHUNK_SIZE`: number of ledgers to work on in one worker.\n* `WORKERS`: number of workers that should be spawned. For best performance this should not exceed the number of CPUs.\n\n## Hardware sizing and timing examples\n\n* 2019-05-19: 23 hours with a `CHUNK_SIZE` of `32768` and 50 workers on a `n1-standard-64` machine on Google Cloud (64 CPUs, 240GB RAM, 2TB SSD)\n\n    ```\n    ./catchup.sh docker-compose.pubnet.yaml 1 23920640 32768 50 2\u003e\u00261 | tee catchup.log\n    ```\n\n* 2018-12-20: 24 hours with a `CHUNK_SIZE` of `32768` and 32 workers on a `n1-standard-32` machine on Google Cloud (32 CPUs, 120GB RAM, 1TB SSD) \n\n    ```\n    ./catchup.sh docker-compose.pubnet.yaml 1 20971520 32768 32 2\u003e\u00261 | tee catchup.log\n    ```\n\n* ... add your achieved result here by submitting a PR.\n\n## Example run on dedicated Google Cloud machine\n\n```\nsudo apt-get update\nsudo apt-get install -y \\\n  apt-transport-https \\\n  ca-certificates \\\n  curl \\\n  gnupg2 \\\n  software-properties-common \\\n  python-pip\ncurl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -\nsudo add-apt-repository \\\n  \"deb [arch=amd64] https://download.docker.com/linux/debian \\\n  $(lsb_release -cs) \\\n  stable\"\nsudo apt-get update\nsudo apt-get install -y docker-ce\nsudo pip install docker-compose\necho '{\"default-address-pools\":[{\"base\":\"172.80.0.0/16\",\"size\":29}]}' | sudo tee /etc/docker/daemon.json\nsudo usermod -G docker andre\nsudo reboot\n# log in again and check whether docker works\ndocker ps\n```\n\n```\ngit clone git@github.com:satoshipay/stellar-core-parallel-catchup.git\ncd stellar-core-parallel-catchup\n./catchup.sh docker-compose.pubnet.yaml 1 20971520 32768 32 2\u003e\u00261 | tee catchup.log\n```\n\nYou will get 3 important pieces of data for Stellar Core:\n\n* SQL database: if you need to move the data to another container/machine you can dump the database by running the following:\n\n    ```\n    docker exec catchup-result_stellar-core-postgres_1 pg_dump -F d -f catchup-sqldump -j 10 -U postgres -d stellar-core\n    ```\n\n    Then copy the `catchup-sqldump` directory to the target container/machine and restore with `pg_restore`.\n\n* `data-result` directory: contains the `buckets` directory that Stellar Core needs for continuing with the current state in the SQL database.\n* `history-result` directory: contains the full history that can be published to help other validator nodes to catch up (e.g., S3, GCS, IPFS, or any other file storage).\n\nNote: make sure you have a consistent state of the three pieces of data before starting Stellar Core in SCP mode (e.g., when moving data to another machine).\n\n## Reset\n\nIf you need to start from scratch again you can delete all docker-compose projects:\n\n```\nfor PROJECT in $(docker ps --filter \"label=com.docker.compose.project\" -q | xargs docker inspect --format='{{index .Config.Labels \"com.docker.compose.project\"}}'| uniq | grep catchup-); do docker-compose -f docker-compose.pubnet.yaml -p $PROJECT down -v; done\ndocker volume prune\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsatoshipay%2Fstellar-core-parallel-catchup","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsatoshipay%2Fstellar-core-parallel-catchup","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsatoshipay%2Fstellar-core-parallel-catchup/lists"}