{"id":49416759,"url":"https://github.com/andyzengmath/mini-hadoop","last_synced_at":"2026-04-29T03:11:07.035Z","repository":{"id":350316129,"uuid":"1205333880","full_name":"andyzengmath/mini-Hadoop","owner":"andyzengmath","description":null,"archived":false,"fork":false,"pushed_at":"2026-04-23T05:30:52.000Z","size":366,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-04-23T06:07:51.756Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/andyzengmath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-08T21:38:23.000Z","updated_at":"2026-04-23T05:30:41.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/andyzengmath/mini-Hadoop","commit_stats":null,"previous_names":["andyzengmath/mini-hadoop"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/andyzengmath/mini-Hadoop","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyzengmath%2Fmini-Hadoop","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyzengmath%2Fmini-Hadoop/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyzengmath%2Fmini-Hadoop/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyzengmath%2Fmini-Hadoop/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/andyzengmath","download_url":"https://codeload.github.com/andyzengmath/mini-Hadoop/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/andyzengmath%2Fmini-Hadoop/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32408559,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T02:37:21.628Z","status":"ssl_error","status_checked_at":"2026-04-29T02:36:50.947Z","response_time":110,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-29T03:11:05.883Z","updated_at":"2026-04-29T03:11:07.029Z","avatar_url":"https://github.com/andyzengmath.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# mini-Hadoop\n\nA high-performance distributed computing framework in Go that reimplements the core architectural primitives of Apache Hadoop. Built via AI-native codebase rebuild — extracting essential features from Hadoop's 201K-node codegraph and reimplementing them in ~24K lines of Go.\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────────────┐\n│                          mini-Hadoop Cluster                            │\n│                                                                         │\n│  ┌──────────────┐  ┌──────────────────┐  ┌────────────────────────┐   │\n│  │  NameNode    │  │ ResourceManager   │  │  Web Dashboards        │   │\n│  │  (metadata)  │  │ (capacity sched.) │  │  :9100 (NN) :9110 (RM)│   │\n│  │  + HA standby│  │ + queue mgmt      │  │  /metrics  /health     │   │\n│  └──────┬───────┘  └────────┬─────────┘  └────────────────────────┘   │\n│         │ gRPC              │ gRPC                                      │\n│  ┌──────┴───────────────────┴──────────────────────────────────────┐   │\n│  │                     Worker Nodes (N nodes)                       │   │\n│  │  ┌────────────┐  ┌─────────────┐  ┌──────────────────────┐     │   │\n│  │  │ DataNode   │  │ NodeManager │  │ ApplicationMaster    │     │   │\n│  │  │ (blocks)   │  │ (containers)│  │ (MR or DAG engine)   │     │   │\n│  │  └────────────┘  └─────────────┘  └──────────────────────┘     │   │\n│  └─────────────────────────────────────────────────────────────────┘   │\n└─────────────────────────────────────────────────────────────────────────┘\n```\n\n## Features\n\n### Core (Phase 1)\n- **HDFS-lite**: Distributed block storage with 3x pipelined replication, streaming writes, heartbeat-based fault detection, automatic re-replication\n- **YARN-lite**: Resource management with capacity scheduler (queue-based multi-tenancy), locality-aware container allocation\n- **MapReduce**: Parallel batch processing with sort/spill, shuffle, combiner support, WordCount/SumByKey built-in jobs\n- **6 gRPC services**: NameNode, DataNode, ResourceManager, NodeManager, Shuffle, common types\n- **7 binaries**: namenode, datanode, resourcemanager, nodemanager, hdfs CLI, mapreduce CLI, mapworker\n\n### Advanced (Phase 2)\n- **HA NameNode**: Write-ahead edit log (WAL), leader election with pluggable backend (ZooKeeper/local), fencing tokens\n- **Capacity Scheduler**: Multi-queue resource management with min/max capacity percentages, elastic bursting\n- **Erasure Coding**: Reed-Solomon RS-6-3 (1.5x storage overhead vs 3x for replication, same fault tolerance)\n- **Block Compression**: Gzip codec with CompressBlock/DecompressBlock (99.2% compression on repetitive text)\n- **DAG Engine**: Spark-like RDD abstraction with Map, FlatMap, Filter, ReduceByKey, GroupByKey, DAG scheduler with stage splitting at shuffle boundaries\n- **Raw TCP Transport**: Custom wire protocol behind BlockTransport interface for maximum block transfer throughput\n- **Short-Circuit Reads**: Bypass DataNode gRPC for co-located data (direct local disk read)\n- **Ack-Queue Pipeline**: Dual-queue pattern for pipeline write reliability with drain-and-retry on failure\n- **Web Dashboard**: Auto-refreshing HTML UI for NameNode and ResourceManager metrics\n- **Metrics Endpoints**: JSON `/metrics` and `/health` HTTP endpoints\n\n### Security\n- Command allowlist for NodeManager process execution\n- Path traversal protection on block storage and container directories\n- Non-root Docker containers\n- Localhost-only port publishing\n\n## Quick Start\n\n### Prerequisites\n- Go 1.23+\n- Docker + Docker Compose (for cluster testing)\n\n### Build\n\n```bash\n# Build all 7 binaries\nmake build\n\n# Run all 136 unit + acceptance tests\nmake test\n\n# Run with race detector\nmake test-race\n```\n\n### Run with Docker Compose\n\n```bash\n# Build and start 3-node cluster\nmake docker-build\nmake docker-up\n\n# Test HDFS operations\ndocker compose -f docker/docker-compose.yml exec client sh -c \"\n  hdfs mkdir /data\n  echo 'hello world' \u003e /tmp/test.txt\n  hdfs put /tmp/test.txt /data/hello.txt\n  hdfs ls /data\n  hdfs get /data/hello.txt /tmp/downloaded.txt\n  cat /tmp/downloaded.txt\n\"\n\n# Run MapReduce WordCount\ndocker compose -f docker/docker-compose.yml exec client sh -c \"\n  mapreduce --job wordcount --input /tmp/test.txt --output /tmp/wc --local\n\"\n\n# View dashboards (from inside network)\ndocker compose -f docker/docker-compose.yml exec client sh -c \"\n  curl -s http://namenode:9100/metrics\n  curl -s http://resourcemanager:9110/metrics\n\"\n\n# Stop cluster\nmake docker-down\n```\n\n### Run Locally\n\n```bash\n# Terminal 1: NameNode (dashboard at localhost:9100)\n./bin/namenode --port 9000\n\n# Terminal 2: ResourceManager (dashboard at localhost:9110)\n./bin/resourcemanager --port 9010\n\n# Terminal 3: Worker\n./bin/datanode --id worker-1 --port 9001 \u0026\n./bin/nodemanager --id worker-1 --port 9011\n\n# Terminal 4: Use HDFS\n./bin/hdfs mkdir /data\n./bin/hdfs put ./local-file.txt /data/file.txt\n./bin/hdfs ls /data\n./bin/hdfs get /data/file.txt ./downloaded.txt\n```\n\n### DAG Engine (Spark-like)\n\n```go\nimport \"github.com/mini-hadoop/mini-hadoop/pkg/dagengine\"\n\nlines := dagengine.NewRDDFromLines(textLines, 4)\nwords := lines.FlatMap(splitWords)\ncounts := words.ReduceByKey(sumInts)\nresult := counts.Collect()\n```\n\n## Project Structure\n\n```\nmini-hadoop/\n├── cmd/                         # 7 binary entry points\n│   ├── namenode/                # NameNode server\n│   ├── datanode/                # DataNode server\n│   ├── resourcemanager/         # ResourceManager server\n│   ├── nodemanager/             # NodeManager server\n│   ├── hdfs/                    # HDFS client CLI\n│   ├── mapreduce/               # MapReduce job submission CLI\n│   └── mapworker/               # Map/reduce task binary\n├── proto/                       # 6 gRPC service definitions\n├── pkg/\n│   ├── namenode/                # Namespace, block mgr, persistence, edit log, election\n│   ├── datanode/                # Block storage, pipeline replication\n│   ├── hdfs/                    # Client library, ack queue\n│   ├── resourcemanager/         # Capacity scheduler, queues\n│   ├── nodemanager/             # Container lifecycle, security\n│   ├── mapreduce/               # MR engine, sort/spill, shuffle, AppMaster, combiner\n│   ├── dagengine/               # Spark-like RDD + DAG scheduler\n│   ├── block/                   # Block abstraction, checksums, compression, erasure coding\n│   ├── config/                  # Configuration (JSON + env vars)\n│   └── rpc/                     # gRPC helpers, TCP transport, metrics, dashboard\n├── test/acceptance/             # 7 acceptance tests\n├── docker/                      # Dockerfile + docker-compose.yml\n└── doc/                         # Design docs, reports, plans\n```\n\n## Test Results\n\n**136 tests, all passing with race detector:**\n\n| Package | Tests | Coverage |\n|---------|-------|----------|\n| pkg/block | 24 | Block IDs, checksums, compression, erasure coding |\n| pkg/config | 7 | Config load/save, env var overrides |\n| pkg/dagengine | 12 | RDD ops, WordCount pipeline, DAG scheduler |\n| pkg/datanode | 10 | Block storage, path traversal protection |\n| pkg/hdfs | 6 | Ack queue lifecycle |\n| pkg/mapreduce | 7 | WordCount, sort buffer, partitioner |\n| pkg/namenode | 41 | Namespace, block mgr, persistence, edit log |\n| pkg/nodemanager | 8 | Command allowlist, arg validation |\n| pkg/resourcemanager | 16 | FIFO scheduler, queues, locality |\n| test/acceptance | 7 | File I/O, replication, WordCount, shuffle |\n\n## Manual Testing (10/10 PASS)\n\nVerified on Windows 11 + Docker Desktop 29.3.1 with a fresh 3-node cluster. See [full sample run output](doc/manual-testing-guide_sample_run.md).\n\n| # | Test | What it proves | Result |\n|---|------|---------------|--------|\n| 1 | Cluster Health | 6 services running, metrics dashboards | PASS |\n| 2 | Create Directories | Namespace operations | PASS |\n| 3 | Small File Upload/Download | Basic write/read pipeline | PASS |\n| 4 | Large File + SHA-256 | 27MB data integrity across chunks | PASS |\n| 5 | Kill Worker Node | Fault tolerance — file readable after death | PASS |\n| 6 | WordCount MapReduce | Map → sort → reduce pipeline (23 words) | PASS |\n| 7 | Distributed Mapworker | 2-reducer partitioned shuffle (6000 words) | PASS |\n| 8 | File Operations | Delete, list, info commands | PASS |\n| 9 | Metrics Dashboard | JSON /metrics + /health endpoints | PASS |\n| 10 | SumByKey Job | Generic MapReduce (electronics=650) | PASS |\n\nTo run the tests yourself, follow [doc/manual-testing-guide.md](doc/manual-testing-guide.md).\n\n## Run on Multiple Machines\n\nmini-Hadoop can run across real separate machines (not just Docker on one host).\n\n### Prerequisites\n- All machines can reach each other over the network (TCP ports 9000-9011)\n- Go 1.23+ installed on all machines, OR copy pre-built binaries\n- Shared config: all workers know the NameNode and ResourceManager addresses\n\n### Step 1: Build binaries\n\n```bash\n# On your build machine\nmake build\n# Produces: bin/namenode, bin/datanode, bin/resourcemanager, bin/nodemanager, bin/hdfs, bin/mapreduce, bin/mapworker\n```\n\nCopy `bin/*` to all machines, or build on each machine.\n\n### Step 2: Start master services\n\nOn **Machine A** (master node):\n```bash\n# Set hostname so workers can reach this machine\nexport MINIHADOOP_HOSTNAME=machine-a.local\n\n# Start NameNode (port 9000) + dashboard (port 9100)\n./bin/namenode --port 9000 \u0026\n\n# Start ResourceManager (port 9010) + dashboard (port 9110)\n./bin/resourcemanager --port 9010 \u0026\n\necho \"Master services running on machine-a.local\"\n```\n\n### Step 3: Start workers\n\nOn **Machine B** (worker 1):\n```bash\nexport MINIHADOOP_HOSTNAME=machine-b.local\nexport MINIHADOOP_NAMENODE_HOST=machine-a.local\nexport MINIHADOOP_NAMENODE_PORT=9000\nexport MINIHADOOP_RM_HOST=machine-a.local\nexport MINIHADOOP_RM_PORT=9010\n\n# Start DataNode + NodeManager\n./bin/datanode --id worker-1 --port 9001 \u0026\n./bin/nodemanager --id worker-1 --port 9011 \u0026\n```\n\nOn **Machine C** (worker 2):\n```bash\nexport MINIHADOOP_HOSTNAME=machine-c.local\nexport MINIHADOOP_NAMENODE_HOST=machine-a.local\nexport MINIHADOOP_NAMENODE_PORT=9000\nexport MINIHADOOP_RM_HOST=machine-a.local\nexport MINIHADOOP_RM_PORT=9010\n\n./bin/datanode --id worker-2 --port 9001 \u0026\n./bin/nodemanager --id worker-2 --port 9011 \u0026\n```\n\nOn **Machine D** (worker 3):\n```bash\nexport MINIHADOOP_HOSTNAME=machine-d.local\nexport MINIHADOOP_NAMENODE_HOST=machine-a.local\nexport MINIHADOOP_NAMENODE_PORT=9000\nexport MINIHADOOP_RM_HOST=machine-a.local\nexport MINIHADOOP_RM_PORT=9010\n\n./bin/datanode --id worker-3 --port 9001 \u0026\n./bin/nodemanager --id worker-3 --port 9011 \u0026\n```\n\n### Step 4: Use the cluster\n\nFrom **any machine** with network access to the master:\n```bash\nexport MINIHADOOP_NAMENODE_HOST=machine-a.local\nexport MINIHADOOP_RM_HOST=machine-a.local\n\n# HDFS operations\n./bin/hdfs mkdir /data\n./bin/hdfs put ./local-file.txt /data/file.txt\n./bin/hdfs get /data/file.txt ./downloaded.txt\n\n# MapReduce\n./bin/mapreduce --job wordcount --input ./input.txt --output ./output --local\n\n# Check dashboards\ncurl http://machine-a.local:9100/metrics   # NameNode\ncurl http://machine-a.local:9110/metrics   # ResourceManager\n```\n\n### Network Topology\n\n```\nMachine A (master)          Machine B (worker-1)\n┌──────────────────┐       ┌──────────────────┐\n│ NameNode :9000   │◄─────►│ DataNode :9001   │\n│ RM       :9010   │◄─────►│ NodeManager:9011 │\n│ Dashboard:9100   │       └──────────────────┘\n│ Dashboard:9110   │\n└──────────────────┘       Machine C (worker-2)\n        ▲                  ┌──────────────────┐\n        ├─────────────────►│ DataNode :9001   │\n        │                  │ NodeManager:9011 │\n        │                  └──────────────────┘\n        │\n        │                  Machine D (worker-3)\n        │                  ┌──────────────────┐\n        └─────────────────►│ DataNode :9001   │\n                           │ NodeManager:9011 │\n                           └──────────────────┘\n```\n\n### Tips for Multi-Machine Deployment\n- **Firewall:** Ensure ports 9000-9011 and 9100-9110 are open between all machines\n- **DNS/Hosts:** Each machine must be resolvable by hostname. Use `/etc/hosts` if no DNS\n- **Data directories:** Each worker stores blocks at `MINIHADOOP_DATA_DIR` (default `/tmp/minihadoop/datanode`). Use a dedicated disk for production\n- **Scaling:** Add more workers by repeating Step 3 on additional machines with unique `--id` values\n\n## Configuration\n\n| Environment Variable | Default | Description |\n|---------------------|---------|-------------|\n| `MINIHADOOP_NAMENODE_HOST` | localhost | NameNode hostname |\n| `MINIHADOOP_NAMENODE_PORT` | 9000 | NameNode gRPC port |\n| `MINIHADOOP_RM_HOST` | localhost | ResourceManager hostname |\n| `MINIHADOOP_RM_PORT` | 9010 | ResourceManager gRPC port |\n| `MINIHADOOP_HOSTNAME` | os.Hostname() | Service registration hostname |\n| `MINIHADOOP_DATA_DIR` | /tmp/minihadoop/datanode | Block storage directory |\n| `MINIHADOOP_METADATA_DIR` | /tmp/minihadoop/namenode | NameNode state directory |\n| `MINIHADOOP_TEMP_DIR` | /tmp/minihadoop/temp | Temporary files |\n\n## Benchmarking (mini-Hadoop vs Apache Hadoop)\n\n18-benchmark comparison suite running both systems in Docker with identical configurations.\n\n```bash\n# Run all 18 benchmarks (~20-30 min)\n./scripts/benchmark.sh\n\n# Quick run (10MB, fewer iterations, ~10 min)\n./scripts/benchmark.sh quick\n\n# mini-Hadoop only (~5 min)\n./scripts/benchmark.sh mini-only\n```\n\n| Category | Benchmarks | Key Metrics |\n|----------|-----------|-------------|\n| Storage (S1-S4) | Write/read throughput, small files, concurrent integrity | MB/s, files/s, SHA-256 |\n| Fault Tolerance (F1-F4) | Detection time, durability, recovery, multi-failure | Seconds, pass/fail |\n| MapReduce (M1-M4) | WordCount, SumByKey, Sort, scaling | Wall-clock ms |\n| Resources (R1-R5) | Image size, memory, peak memory, startup, LOC | MB, seconds |\n| Correctness (C1-C3) | Output equivalence, edge cases, replication | Match/differ |\n\nResults saved to `benchmark_results_*/` with per-test files and summary. See [benchmark plan](doc/benchmark-plan.md) for full methodology.\n\n## Documentation\n\n- **[Manual Testing Guide](doc/manual-testing-guide.md)** — 10 step-by-step tests for verifying all features\n- **[Sample Run Output](doc/manual-testing-guide_sample_run.md)** — Verified test output (10/10 PASS)\n- **[Comprehensive Technical Report](doc/mini-hadoop-comprehensive-report.md)** — Full architecture analysis, data flow diagrams, algorithms\n- **[Benchmark Plan](doc/benchmark-plan.md)** — 18 benchmarks: mini-Hadoop vs Apache Hadoop\n- **[Scaling Analysis](doc/scaling-analysis.md)** — Bottleneck analysis + roadmap (3 → 1000+ nodes)\n- **[Integration Plan](doc/integration-plan.md)** — How to wire library-only features into servers\n- **[Phase 2 Implementation Plan](doc/phase2-plan.md)** — Detailed specs for advanced features\n\n## Design Decisions\n\n| Decision | Choice | Rationale |\n|----------|--------|-----------|\n| Language | Go | Near-Java performance + AI-friendly + goroutines |\n| Communication | gRPC + protobuf | Strongly typed, efficient, natural for Go |\n| Block transport | gRPC streaming + raw TCP option | BlockTransport interface allows swapping |\n| Replication | 3x pipelined + RS-6-3 erasure coding | Standard + storage-efficient option |\n| Scheduler | Capacity (queue-based) | Multi-tenant with elastic bursting |\n| Processing | MapReduce + DAG engine | Batch + Spark-like in-memory |\n| HA | Edit log WAL + leader election | Pluggable backend (ZooKeeper/local) |\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyzengmath%2Fmini-hadoop","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fandyzengmath%2Fmini-hadoop","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fandyzengmath%2Fmini-hadoop/lists"}