https://github.com/wwwehr/hl-evm-block-sync
Fastest way to pull down hyperliquid evm block files from s3
https://github.com/wwwehr/hl-evm-block-sync
hyperliquid-dex hyperliquid-evm
Last synced: 4 months ago
JSON representation
Fastest way to pull down hyperliquid evm block files from s3
- Host: GitHub
- URL: https://github.com/wwwehr/hl-evm-block-sync
- Owner: wwwehr
- License: mit
- Created: 2025-08-20T19:47:07.000Z (9 months ago)
- Default Branch: master
- Last Pushed: 2025-08-22T04:06:30.000Z (9 months ago)
- Last Synced: 2025-08-22T06:22:19.312Z (9 months ago)
- Topics: hyperliquid-dex, hyperliquid-evm
- Language: Shell
- Homepage:
- Size: 9.77 KB
- Stars: 0
- Watchers: 0
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# 🚀 Hyperliquid DEX – S3Sync Runner
Fastest way to pull down evm block files from s3
This script automates syncing **massive S3 object stores** in a **safe, resumable, and time-tracked way**. The traditional `s3 sync` is just wayy to slow.
## Features
- ✅ Auto-installs [nidor1998/s3sync](https://github.com/nidor1998/s3sync) (latest release) into `~/.local/bin`
- ✅ Sequential per-prefix syncs (e.g., `21000000/`, `22000000/`, …)
- ✅ Per-prefix timing: `22000000 took 12 minutes!`
- ✅ Total runtime summary at the end
- ✅ Designed for **tiny files at scale** (EVM block archives)
- ✅ Zero-config bootstrap — just run the script
## Quick Start
Instead of cloning a full repo, you can just download the runner script directly:
```bash
curl -L -o s3sync-runner.sh https://raw.githubusercontent.com/wwwehr/hl-evm-block-sync/refs/heads/master/sync.sh
chmod +x s3sync-runner.sh
./s3sync-runner.sh
```
> Skipping to relevant block section
```bash
./s3sync-runner.sh --start-at 30000000
```
The script will:
* Install or update s3sync into ~/.local/bin
* Discover top-level prefixes in your S3 bucket
* Sync them one at a time, printing elapsed minutes
## Configuration
Edit the top of s3sync-runner.sh if needed:
```bash
BUCKET="hl-testnet-evm-blocks" # could be hl-mainnet-evm-blocks
REGION="ap-northeast-1" # hardcoded bucket region
DEST="$HOME/evm-blocks-testnet" # local target directory (this is what nanoreth will look at)
WORKERS=512 # worker threads per sync (lotsa workers needs lotsa RAM)
```
## Example Output
```bash
[2025-08-20 20:01:02] START 21000000
[2025-08-20 20:13:15] 21000000 took 12 minutes!
[2025-08-20 20:13:15] START 22000000
[2025-08-20 20:26:40] 22000000 took 13 minutes!
[2025-08-20 20:26:40] ALL DONE in 25 minutes.
```
## Hackathon Context
This runner was built as part of the Hyperliquid DEX Hackathon to accelerate:
* ⛓️ Blockchain archive node ingestion
* 📂 EVM block dataset replication
* 🧩 DEX ecosystem data pipelines