{"id":30559360,"url":"https://github.com/pramithamj/hadoop-cluster-manager","last_synced_at":"2025-08-28T09:34:37.745Z","repository":{"id":296921306,"uuid":"994990632","full_name":"PramithaMJ/hadoop-cluster-manager","owner":"PramithaMJ","description":"Complete Apache Hadoop 3.4.1 cluster installation and management toolkit with automated scripts, comprehensive documentation, and production-ready configuration templates for single-node and multi-node deployments.","archived":false,"fork":false,"pushed_at":"2025-06-02T20:36:20.000Z","size":34,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-03T11:34:02.114Z","etag":null,"topics":["big-data","cluster-management","hadoop","hdfs","mapreduce"],"latest_commit_sha":null,"homepage":"https://pramithamj.github.io/hadoop-cluster-manager/","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/PramithaMJ.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-02T19:51:14.000Z","updated_at":"2025-06-02T20:31:38.000Z","dependencies_parsed_at":"2025-06-08T01:17:29.751Z","dependency_job_id":null,"html_url":"https://github.com/PramithaMJ/hadoop-cluster-manager","commit_stats":null,"previous_names":["pramithamj/hadoop-cluster-manager"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/PramithaMJ/hadoop-cluster-manager","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fhadoop-cluster-manager","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fhadoop-cluster-manager/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fhadoop-cluster-manager/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fhadoop-cluster-manager/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/PramithaMJ","download_url":"https://codeload.github.com/PramithaMJ/hadoop-cluster-manager/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/PramithaMJ%2Fhadoop-cluster-manager/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":272476947,"owners_count":24940952,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-28T02:00:10.768Z","response_time":74,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["big-data","cluster-management","hadoop","hdfs","mapreduce"],"created_at":"2025-08-28T09:34:34.419Z","updated_at":"2025-08-28T09:34:37.718Z","avatar_url":"https://github.com/PramithaMJ.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Apache Hadoop Cluster Installation Guide\n\nThis comprehensive guide will walk you through installing and configuring an Apache Hadoop cluster, covering both single-node (pseudo-distributed) and multi-node (fully distributed) setups.\n\n## Table of Contents\n\n1. [Prerequisites](#prerequisites)\n2. [Download and Installation](#download-and-installation)\n3. [Environment Configuration](#environment-configuration)\n4. [Hadoop Configuration](#hadoop-configuration)\n5. [Single-Node Setup (Pseudo-Distributed)](#single-node-setup)\n6. [Multi-Node Setup (Fully Distributed)](#multi-node-setup)\n7. [Starting the Cluster](#starting-the-cluster)\n8. [Verification and Testing](#verification-and-testing)\n9. [Common Issues and Troubleshooting](#troubleshooting)\n10. [Useful Commands](#useful-commands)\n\n## Prerequisites\n\n### System Requirements\n- **Operating System**: Linux, macOS, or Windows (with WSL)\n- **Java**: OpenJDK 8, 11, or 17 (recommended: OpenJDK 11)\n- **Memory**: Minimum 4GB RAM (8GB+ recommended for multi-node)\n- **Disk Space**: Minimum 20GB available space\n- **Network**: SSH access between nodes (for multi-node setup)\n\n### Required Software\n- Java Development Kit (JDK)\n- SSH server and client\n- rsync (for file synchronization)\n\n## Download and Installation\n\n### Step 1: Download Hadoop\n\n| Version | Release Date | Binary Download |\n|---------|--------------|-----------------|\n| 3.4.1   | 2024 Oct 18  | [hadoop-3.4.1.tar.gz](https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz) |\n\n```bash\n# Download Hadoop 3.4.1\nwget https://archive.apache.org/dist/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz\n\n# Verify the download (optional)\nwget https://downloads.apache.org/hadoop/common/hadoop-3.4.1/hadoop-3.4.1.tar.gz.sha512\nshasum -a 512 -c hadoop-3.4.1.tar.gz.sha512\n```\n\n### Step 2: Extract and Install\n\n```bash\n# Extract the archive\ntar -xzf hadoop-3.4.1.tar.gz\n\n# Move to installation directory (optional)\nsudo mv hadoop-3.4.1 /opt/hadoop\n# OR keep it in your preferred location\nmv hadoop-3.4.1 ~/hadoop\n```\n\n## Environment Configuration\n\n### Step 1: Install Java (if not already installed)\n\n#### On Ubuntu/Debian:\n```bash\nsudo apt update\nsudo apt install openjdk-11-jdk\n```\n\n#### On CentOS/RHEL:\n```bash\nsudo yum install java-11-openjdk-devel\n```\n\n#### On macOS:\n```bash\nbrew install openjdk@11\n```\n\n### Step 2: Configure Environment Variables\n\nAdd the following to your `~/.bashrc`, `~/.zshrc`, or `~/.profile`:\n\n```bash\n# Java Environment\nexport JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64  # Linux\n# export JAVA_HOME=/opt/homebrew/Cellar/openjdk@11/11.0.XX/libexec/openjdk.jdk/Contents/Home  # macOS\n\n# Hadoop Environment\nexport HADOOP_HOME=/opt/hadoop  # or your installation path\nexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop\nexport HADOOP_MAPRED_HOME=$HADOOP_HOME\nexport HADOOP_COMMON_HOME=$HADOOP_HOME\nexport HADOOP_HDFS_HOME=$HADOOP_HOME\nexport YARN_HOME=$HADOOP_HOME\n\n# Add Hadoop binaries to PATH\nexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin\n```\n\nApply the changes:\n```bash\nsource ~/.bashrc  # or ~/.zshrc\n```\n\n### Step 3: Configure SSH (Required for cluster operations)\n\n```bash\n# Generate SSH key pair (if not exists)\nssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa\n\n# Add public key to authorized_keys\ncat ~/.ssh/id_rsa.pub \u003e\u003e ~/.ssh/authorized_keys\n\n# Set appropriate permissions\nchmod 0600 ~/.ssh/authorized_keys\n\n# Test SSH to localhost\nssh localhost\n```\n\n## Hadoop Configuration\n\nNavigate to the Hadoop configuration directory:\n```bash\ncd $HADOOP_HOME/etc/hadoop\n```\n\n### Step 1: Configure `hadoop-env.sh`\n\n```bash\n# Edit hadoop-env.sh\nvim hadoop-env.sh\n\n# Add or update the JAVA_HOME\nexport JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64\n```\n\n### Step 2: Configure Core Components\n\n#### `core-site.xml`\n```xml\n\u003cconfiguration\u003e\n    \u003cproperty\u003e\n        \u003cname\u003efs.defaultFS\u003c/name\u003e\n        \u003cvalue\u003ehdfs://localhost:9000\u003c/value\u003e\n        \u003cdescription\u003eThe default file system URI\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003ehadoop.tmp.dir\u003c/name\u003e\n        \u003cvalue\u003e/opt/hadoop/tmp\u003c/value\u003e\n        \u003cdescription\u003eTemporary directory for Hadoop\u003c/description\u003e\n    \u003c/property\u003e\n\u003c/configuration\u003e\n```\n\n#### `hdfs-site.xml`\n```xml\n\u003cconfiguration\u003e\n    \u003cproperty\u003e\n        \u003cname\u003edfs.namenode.name.dir\u003c/name\u003e\n        \u003cvalue\u003e/opt/hadoop/data/namenode\u003c/value\u003e\n        \u003cdescription\u003eDirectory for namenode metadata\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003edfs.datanode.data.dir\u003c/name\u003e\n        \u003cvalue\u003e/opt/hadoop/data/datanode\u003c/value\u003e\n        \u003cdescription\u003eDirectory for datanode data\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003edfs.replication\u003c/name\u003e\n        \u003cvalue\u003e1\u003c/value\u003e\n        \u003cdescription\u003eDefault block replication\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003edfs.namenode.checkpoint.dir\u003c/name\u003e\n        \u003cvalue\u003e/opt/hadoop/data/secondary\u003c/value\u003e\n        \u003cdescription\u003eSecondary namenode checkpoint directory\u003c/description\u003e\n    \u003c/property\u003e\n\u003c/configuration\u003e\n```\n\n#### `mapred-site.xml`\n```xml\n\u003cconfiguration\u003e\n    \u003cproperty\u003e\n        \u003cname\u003emapreduce.framework.name\u003c/name\u003e\n        \u003cvalue\u003eyarn\u003c/value\u003e\n        \u003cdescription\u003eMapReduce framework name\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003emapreduce.application.classpath\u003c/name\u003e\n        \u003cvalue\u003e$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*\u003c/value\u003e\n    \u003c/property\u003e\n\u003c/configuration\u003e\n```\n\n#### `yarn-site.xml`\n```xml\n\u003cconfiguration\u003e\n    \u003cproperty\u003e\n        \u003cname\u003eyarn.nodemanager.aux-services\u003c/name\u003e\n        \u003cvalue\u003emapreduce_shuffle\u003c/value\u003e\n        \u003cdescription\u003eAuxiliary services for NodeManager\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003eyarn.resourcemanager.hostname\u003c/name\u003e\n        \u003cvalue\u003elocalhost\u003c/value\u003e\n        \u003cdescription\u003eResourceManager hostname\u003c/description\u003e\n    \u003c/property\u003e\n    \u003cproperty\u003e\n        \u003cname\u003eyarn.nodemanager.env-whitelist\u003c/name\u003e\n        \u003cvalue\u003eJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME\u003c/value\u003e\n    \u003c/property\u003e\n\u003c/configuration\u003e\n```\n\n### Step 3: Create Required Directories\n\n```bash\n# Create data directories\nsudo mkdir -p /opt/hadoop/data/namenode\nsudo mkdir -p /opt/hadoop/data/datanode\nsudo mkdir -p /opt/hadoop/data/secondary\nsudo mkdir -p /opt/hadoop/tmp\n\n# Set appropriate ownership\nsudo chown -R $USER:$USER /opt/hadoop/data\nsudo chown -R $USER:$USER /opt/hadoop/tmp\n```\n\n## Single-Node Setup\n\n### Format the NameNode (First-time setup only)\n\n```bash\nhdfs namenode -format -force\n```\n\n### Start Hadoop Services\n\n```bash\n# Start HDFS\nstart-dfs.sh\n\n# Start YARN\nstart-yarn.sh\n\n# Or start all services at once\nstart-all.sh\n```\n\n### Verify the Installation\n\n```bash\n# Check running processes\njps\n\n# Expected output should include:\n# - NameNode\n# - DataNode\n# - ResourceManager\n# - NodeManager\n# - SecondaryNameNode\n```\n\n## Multi-Node Setup\n\n### Prerequisites for Multi-Node\n\n1. Multiple machines with Hadoop installed\n2. Network connectivity between all nodes\n3. SSH access from master to all slave nodes\n4. Same username on all nodes\n5. Synchronized time across all nodes\n\n### Step 1: Configure Master Node\n\n#### Update `core-site.xml` on master:\n```xml\n\u003cproperty\u003e\n    \u003cname\u003efs.defaultFS\u003c/name\u003e\n    \u003cvalue\u003ehdfs://master-node:9000\u003c/value\u003e\n\u003c/property\u003e\n```\n\n#### Update `yarn-site.xml` on master:\n```xml\n\u003cproperty\u003e\n    \u003cname\u003eyarn.resourcemanager.hostname\u003c/name\u003e\n    \u003cvalue\u003emaster-node\u003c/value\u003e\n\u003c/property\u003e\n```\n\n#### Configure workers file:\n```bash\n# Edit $HADOOP_HOME/etc/hadoop/workers\nvim $HADOOP_HOME/etc/hadoop/workers\n\n# Add worker node hostnames (one per line)\nworker-node-1\nworker-node-2\nworker-node-3\n```\n\n### Step 2: Configure Slave Nodes\n\n1. Copy the entire Hadoop configuration from master to all slave nodes:\n```bash\nscp -r $HADOOP_HOME/etc/hadoop/ user@worker-node:/opt/hadoop/etc/\n```\n\n2. Update `hdfs-site.xml` on slaves to point to master:\n```xml\n\u003cproperty\u003e\n    \u003cname\u003edfs.namenode.name.dir\u003c/name\u003e\n    \u003cvalue\u003e/opt/hadoop/data/namenode\u003c/value\u003e\n\u003c/property\u003e\n```\n\n### Step 3: Network Configuration\n\nUpdate `/etc/hosts` on all nodes:\n```bash\n# Add entries for all nodes\n192.168.1.100   master-node\n192.168.1.101   worker-node-1\n192.168.1.102   worker-node-2\n192.168.1.103   worker-node-3\n```\n\n### Step 4: Start Multi-Node Cluster\n\nFrom the master node:\n```bash\n# Format namenode (first time only)\nhdfs namenode -format\n\n# Start the cluster\nstart-all.sh\n```\n\n## Starting and Stopping Hadoop Cluster\n\n### 🎯 **Quick Start (Recommended)**\n\nUse the convenient control script included in this repository:\n\n```bash\n# Start the cluster\n./hadoop-control.sh start\n\n# Stop the cluster  \n./hadoop-control.sh stop\n\n# Restart the cluster\n./hadoop-control.sh restart\n\n# Check cluster status\n./hadoop-control.sh status\n\n# Show help\n./hadoop-control.sh help\n```\n\nThis script provides:\n- ✅ **Colored output** for easy reading\n- ✅ **Automatic error checking** and validation\n- ✅ **Smart service detection** - knows what's running\n- ✅ **Web interface links** when cluster is healthy\n- ✅ **Safe start/stop procedures** with proper sequencing\n\n### 🚀 Manual Service Management\n\n#### Method 1: Start All Services at Once\n\n```bash\n# Start all Hadoop services\nstart-all.sh\n\n# Check if all services are running\njps\n```\n\n#### Method 2: Start Services Individually\n\n```bash\n# 1. Start HDFS services (NameNode, DataNode, SecondaryNameNode)\nstart-dfs.sh\n\n# 2. Start YARN services (ResourceManager, NodeManager)\nstart-yarn.sh\n\n# 3. Start MapReduce Job History Server (optional)\nmapred --daemon start historyserver\n```\n\n#### Method 3: Start Services One by One\n\n```bash\n# Start NameNode\nhdfs --daemon start namenode\n\n# Start DataNode\nhdfs --daemon start datanode\n\n# Start SecondaryNameNode\nhdfs --daemon start secondarynamenode\n\n# Start ResourceManager\nyarn --daemon start resourcemanager\n\n# Start NodeManager\nyarn --daemon start nodemanager\n\n# Start Job History Server\nmapred --daemon start historyserver\n```\n\n### ⏹️ Stopping Hadoop Services\n\n#### Stop All Services\n\n```bash\n# Stop all Hadoop services\nstop-all.sh\n```\n\n#### Stop Services Individually\n\n```bash\n# Stop YARN services\nstop-yarn.sh\n\n# Stop HDFS services\nstop-dfs.sh\n\n# Stop Job History Server\nmapred --daemon stop historyserver\n```\n\n#### Stop Services One by One\n\n```bash\n# Stop Job History Server\nmapred --daemon stop historyserver\n\n# Stop NodeManager\nyarn --daemon stop nodemanager\n\n# Stop ResourceManager\nyarn --daemon stop resourcemanager\n\n# Stop SecondaryNameNode\nhdfs --daemon stop secondarynamenode\n\n# Stop DataNode\nhdfs --daemon stop datanode\n\n# Stop NameNode\nhdfs --daemon stop namenode\n```\n\n### 🔄 Restart Services\n\n```bash\n# Restart all services\nstop-all.sh \u0026\u0026 start-all.sh\n\n# Restart HDFS only\nstop-dfs.sh \u0026\u0026 start-dfs.sh\n\n# Restart YARN only\nstop-yarn.sh \u0026\u0026 start-yarn.sh\n```\n\n### ✅ Verify Services are Running\n\n#### Check Running Java Processes\n\n```bash\n# List all Hadoop-related Java processes\njps\n\n# Expected output should include:\n# 12345 NameNode\n# 12346 DataNode\n# 12347 SecondaryNameNode\n# 12348 ResourceManager\n# 12349 NodeManager\n# 12350 JobHistoryServer (if started)\n```\n\n#### Check Specific Service Status\n\n```bash\n# Check if NameNode is running\nhdfs dfsadmin -report\n\n# Check if YARN is running\nyarn node -list\n\n# Check cluster health\nhdfs dfsadmin -safemode get\n```\n\n### 🧪 Testing Hadoop Installation\n\n#### Test 1: Basic HDFS Operations\n\n```bash\n# Create your user directory in HDFS\nhdfs dfs -mkdir -p /user/$USER\n\n# Create a test directory\nhdfs dfs -mkdir /user/$USER/test\n\n# List HDFS root directory\nhdfs dfs -ls /\n\n# List your user directory\nhdfs dfs -ls /user/$USER\n\n# Check HDFS health\nhdfs fsck /\n```\n\n#### Test 2: File Upload and Download\n\n```bash\n# Create a test file locally\necho \"Hello Hadoop World!\" \u003e test.txt\necho \"This is a test file for Hadoop HDFS\" \u003e\u003e test.txt\n\n# Upload file to HDFS\nhdfs dfs -put test.txt /user/$USER/\n\n# List files in HDFS\nhdfs dfs -ls /user/$USER/\n\n# View file content in HDFS\nhdfs dfs -cat /user/$USER/test.txt\n\n# Download file from HDFS\nhdfs dfs -get /user/$USER/test.txt downloaded_test.txt\n\n# Verify the downloaded file\ncat downloaded_test.txt\n\n# Clean up test files\nrm test.txt downloaded_test.txt\nhdfs dfs -rm /user/$USER/test.txt\n```\n\n#### Test 3: Run Sample MapReduce Job\n\n```bash\n# Create input directory for MapReduce\nhdfs dfs -mkdir /input\n\n# Copy Hadoop configuration files as input\nhdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /input/\n\n# Run the word count example\nhadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar wordcount /input /output\n\n# Check the output\nhdfs dfs -ls /output/\nhdfs dfs -cat /output/part-r-00000 | head -20\n\n# Clean up\nhdfs dfs -rm -r /output\nhdfs dfs -rm -r /input\n```\n\n#### Test 4: YARN Application Test\n\n```bash\n# Run a simple YARN application (Pi calculation)\nyarn jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 2 100\n\n# Check application history\nyarn application -list -appStates ALL\n```\n\n#### Test 5: Performance Benchmark Tests\n\n```bash\n# TestDFSIO Write Test\nhadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar TestDFSIO -write -nrFiles 4 -fileSize 128MB\n\n# TestDFSIO Read Test\nhadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar TestDFSIO -read -nrFiles 4 -fileSize 128MB\n\n# Clean up test data\nhadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-*-tests.jar TestDFSIO -clean\n```\n\n### 📊 Monitoring and Health Checks\n\n#### Web Interface Access\n\n```bash\n# Open web interfaces (run these commands to get URLs)\necho \"NameNode Web UI: http://localhost:9870\"\necho \"ResourceManager Web UI: http://localhost:8088\"\necho \"Job History Server: http://localhost:19888\"\necho \"DataNode Web UI: http://localhost:9864\"\necho \"NodeManager Web UI: http://localhost:8042\"\n```\n\n#### Command Line Monitoring\n\n```bash\n# Check cluster summary\nhdfs dfsadmin -report\n\n# Check filesystem\nhdfs fsck /\n\n# Monitor YARN applications\nyarn top\n\n# Check node status\nyarn node -list -showDetails\n\n# View cluster metrics\nyarn cluster -lnl\n```\n\n## 🔧 Cluster Node Management\n\n### Node Management Script\n\nI've created a comprehensive script to manage cluster nodes:\n\n```bash\n# Make the script executable (if not already)\nchmod +x ./manage-cluster-nodes.sh\n\n# Show all available commands\n./manage-cluster-nodes.sh help\n```\n\n### Key Features\n\n#### 📋 **Node Information**\n```bash\n# List all current nodes\n./manage-cluster-nodes.sh list\n\n# Show detailed cluster status\n./manage-cluster-nodes.sh status\n```\n\n#### ➕ **Adding Nodes**\n```bash\n# Add a new worker node\n./manage-cluster-nodes.sh add worker-node-1\n\n# Add node by IP address\n./manage-cluster-nodes.sh add 192.168.1.100\n```\n\n#### ➖ **Removing Nodes (Safe Process)**\n```bash\n# Step 1: Safely decommission the node\n./manage-cluster-nodes.sh decommission worker-node-1\n\n# Step 2: Remove from cluster (after decommissioning completes)\n./manage-cluster-nodes.sh remove worker-node-1\n```\n\n#### 🔄 **Node Management**\n```bash\n# Bring back a decommissioned node\n./manage-cluster-nodes.sh recommission worker-node-1\n```\n\n#### 🏗️ **Cluster Conversion**\n```bash\n# Convert single-node to multi-node setup\n./manage-cluster-nodes.sh convert-multi\n\n# Convert multi-node back to single-node\n./manage-cluster-nodes.sh convert-single\n```\n\n#### 💾 **Configuration Backup \u0026 Restore**\n```bash\n# Backup current configuration\n./manage-cluster-nodes.sh backup\n\n# Restore from backup\n./manage-cluster-nodes.sh restore\n```\n\n### Adding a New Worker Node - Complete Process\n\n1. **Add node to cluster configuration:**\n   ```bash\n   ./manage-cluster-nodes.sh add worker-node-2\n   ```\n\n2. **Set up SSH passwordless access:**\n   ```bash\n   # Copy SSH key to new node\n   ssh-copy-id user@worker-node-2\n   \n   # Test SSH access\n   ssh worker-node-2\n   ```\n\n3. **Install Hadoop on the new node:**\n   ```bash\n   # Copy Hadoop installation to new node\n   scp -r $HADOOP_HOME user@worker-node-2:/opt/\n   ```\n\n4. **Copy configuration files:**\n   ```bash\n   # Copy configuration to new node\n   scp -r $HADOOP_HOME/etc/hadoop/* user@worker-node-2:$HADOOP_HOME/etc/hadoop/\n   ```\n\n5. **Refresh cluster nodes:**\n   ```bash\n   # Refresh YARN nodes\n   yarn rmadmin -refreshNodes\n   \n   # Refresh HDFS nodes\n   hdfs dfsadmin -refreshNodes\n   ```\n\n6. **Start services on new node:**\n   ```bash\n   # On the new worker node, start DataNode and NodeManager\n   ssh worker-node-2 \"$HADOOP_HOME/bin/hdfs --daemon start datanode\"\n   ssh worker-node-2 \"$HADOOP_HOME/bin/yarn --daemon start nodemanager\"\n   ```\n\n### Safely Removing a Node - Complete Process\n\n1. **Decommission the node:**\n   ```bash\n   ./manage-cluster-nodes.sh decommission worker-node-2\n   ```\n\n2. **Monitor decommissioning progress:**\n   ```bash\n   # Check HDFS decommissioning status\n   hdfs dfsadmin -report\n   \n   # Check YARN node status\n   yarn node -list -all\n   ```\n\n3. **Wait for decommissioning to complete** (data blocks are moved to other nodes)\n\n4. **Remove the node:**\n   ```bash\n   ./manage-cluster-nodes.sh remove worker-node-2\n   ```\n\n5. **Stop services on the removed node:**\n   ```bash\n   ssh worker-node-2 \"$HADOOP_HOME/bin/yarn --daemon stop nodemanager\"\n   ssh worker-node-2 \"$HADOOP_HOME/bin/hdfs --daemon stop datanode\"\n   ```\n\n### Configuration Files Modified\n\nThe script automatically manages these files:\n- `$HADOOP_HOME/etc/hadoop/workers` - List of worker nodes\n- `$HADOOP_HOME/etc/hadoop/core-site.xml` - Core configuration\n- `$HADOOP_HOME/etc/hadoop/yarn-site.xml` - YARN configuration  \n- `$HADOOP_HOME/etc/hadoop/dfs.exclude` - HDFS decommission list\n- `$HADOOP_HOME/etc/hadoop/yarn.exclude` - YARN decommission list\n\n### Backup and Recovery\n\nAll configuration changes are automatically backed up to:\n```\n$HADOOP_HOME/backups/hadoop_config_YYYYMMDD_HHMMSS.tar.gz\n```\n\nYou can restore any backup using:\n```bash\n./manage-cluster-nodes.sh restore\n```\n\n### Quick Reference\n\nFor a complete quick reference guide, see: `CLUSTER-MANAGEMENT.md`\n\n---\n## Verification and Testing\n\n### Check Cluster Status\n\n```bash\n# Check HDFS status\nhdfs dfsadmin -report\n\n# Check YARN nodes\nyarn node -list\n\n# Check running processes\njps\n```\n\n### Web Interfaces\n\n- **HDFS NameNode**: http://localhost:9870\n- **YARN ResourceManager**: http://localhost:8088\n- **MapReduce Job History**: http://localhost:19888\n\n### Basic HDFS Operations\n\n```bash\n# Create directories in HDFS\nhdfs dfs -mkdir /user\nhdfs dfs -mkdir /user/$USER\n\n# List HDFS contents\nhdfs dfs -ls /\n\n# Copy file to HDFS\nhdfs dfs -put /path/to/local/file /user/$USER/\n\n# Copy file from HDFS\nhdfs dfs -get /user/$USER/file /path/to/local/\n```\n\n### Run Sample MapReduce Job\n\n```bash\n# Create input directory\nhdfs dfs -mkdir /input\n\n# Copy input files\nhdfs dfs -put $HADOOP_HOME/etc/hadoop/*.xml /input\n\n# Run word count example\nhadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.1.jar wordcount /input /output\n\n# Check output\nhdfs dfs -cat /output/part-r-00000\n```\n\n## Troubleshooting\n\n### Common Issues and Solutions\n\n#### 1. Java-related Errors\n```bash\n# Verify Java installation\njava -version\necho $JAVA_HOME\n\n# Ensure JAVA_HOME is set in hadoop-env.sh\n```\n\n#### 2. SSH Connection Issues\n```bash\n# Test SSH connectivity\nssh localhost\nssh worker-node-1\n\n# Check SSH key setup\nls -la ~/.ssh/\n```\n\n#### 3. Permission Denied Errors\n```bash\n# Fix directory permissions\nsudo chown -R $USER:$USER $HADOOP_HOME\nchmod 755 $HADOOP_HOME/data/*\n```\n\n#### 4. Port Already in Use\n```bash\n# Check what's using the port\nnetstat -tulpn | grep :9000\nlsof -i :9000\n\n# Kill the process if necessary\nkill -9 \u003cPID\u003e\n```\n\n#### 5. DataNode Not Starting\n```bash\n# Check logs\ntail -f $HADOOP_HOME/logs/hadoop-*-datanode-*.log\n\n# Common solution: Remove and reformat\nstop-all.sh\nrm -rf $HADOOP_HOME/data/datanode/*\nhdfs namenode -format -force\nstart-all.sh\n```\n\n### Log Files Location\n\n```bash\n# Hadoop logs directory\n$HADOOP_HOME/logs/\n\n# Important log files\nhadoop-*-namenode-*.log\nhadoop-*-datanode-*.log\nyarn-*-resourcemanager-*.log\nyarn-*-nodemanager-*.log\n```\n\n## Useful Commands\n\n### HDFS Commands\n```bash\n# File system check\nhdfs fsck /\n\n# Safe mode operations\nhdfs dfsadmin -safemode leave\nhdfs dfsadmin -safemode enter\n\n# Balance cluster\nhdfs balancer\n\n# Decommission nodes\nhdfs dfsadmin -refreshNodes\n```\n\n### YARN Commands\n```bash\n# List applications\nyarn application -list\n\n# Kill application\nyarn application -kill \u003capplication_id\u003e\n\n# Node management\nyarn node -list -all\nyarn rmadmin -refreshNodes\n```\n\n### Cluster Administration\n```bash\n# Check cluster health\nhdfs dfsadmin -report\nyarn node -list -showDetails\n\n# Monitor cluster\nhadoop dfsadmin -printTopology\nyarn top\n```\n\n### Performance Monitoring\n```bash\n# Check disk usage\nhdfs dfs -du -h /\n\n# Monitor system resources\ntop\nhtop\niostat -x 1\n```\n\n## References\n\n- [Official Hadoop Documentation](https://hadoop.apache.org/docs/current/)\n- [Cluster Setup Guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)\n- [HDFS Architecture](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)\n- [YARN Architecture](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)\n\n## Support\n\nFor issues and questions:\n- Check the [Hadoop Troubleshooting Guide](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Troubleshooting.html)\n- Visit [Apache Hadoop User Mailing List](https://hadoop.apache.org/mailing_lists.html)\n- Submit issues to [Apache Hadoop JIRA](https://issues.apache.org/jira/projects/HADOOP)\n\n---\n\n**Note**: This guide is based on Hadoop 3.4.1. Configuration may vary slightly for different versions.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpramithamj%2Fhadoop-cluster-manager","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fpramithamj%2Fhadoop-cluster-manager","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fpramithamj%2Fhadoop-cluster-manager/lists"}