{"id":26055754,"url":"https://github.com/lovnishverma/bigdataecosystem","last_synced_at":"2025-08-19T02:33:52.873Z","repository":{"id":270940060,"uuid":"911913637","full_name":"lovnishverma/bigdataecosystem","owner":"lovnishverma","description":"Complete Big Data  Ecosystem on Docker Desktop","archived":false,"fork":false,"pushed_at":"2025-03-24T23:22:53.000Z","size":348,"stargazers_count":4,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-03-24T23:23:19.056Z","etag":null,"topics":["bigdata","docker","flume","hadoop","hdfs","hive","mapreduce","spark","sqoop"],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lovnishverma.png","metadata":{"files":{"readme":"README.md","changelog":"historyserver/Dockerfile","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-04T07:06:40.000Z","updated_at":"2025-03-24T23:22:56.000Z","dependencies_parsed_at":null,"dependency_job_id":"d6a9ea00-24b1-4816-b4b7-accb66d05853","html_url":"https://github.com/lovnishverma/bigdataecosystem","commit_stats":null,"previous_names":["lovnishverma/bigdataecosystem"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovnishverma%2Fbigdataecosystem","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovnishverma%2Fbigdataecosystem/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovnishverma%2Fbigdataecosystem/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lovnishverma%2Fbigdataecosystem/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lovnishverma","download_url":"https://codeload.github.com/lovnishverma/bigdataecosystem/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248333947,"owners_count":21086283,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["bigdata","docker","flume","hadoop","hdfs","hive","mapreduce","spark","sqoop"],"created_at":"2025-03-08T10:21:17.841Z","updated_at":"2025-04-11T03:20:23.860Z","avatar_url":"https://github.com/lovnishverma.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"![image](https://github.com/user-attachments/assets/2b0a8b29-8287-446a-8a0c-8c1820ea0971) ![image](https://github.com/user-attachments/assets/343cfd7e-73b7-4eb2-a9a4-76c31f5703c8).![image](https://github.com/user-attachments/assets/04ad8a37-c3a0-4e62-a5c4-70c023992209)![image](https://github.com/user-attachments/assets/5a5fc24a-bc9d-4cc2-aab4-b651c59197d5)![image](https://github.com/user-attachments/assets/10b26b1e-614f-4ad7-966c-505e54825680)\n\n\n\n# Docker Multi-Container Environment with Hadoop, Spark, and Hive\n\nThis guide helps you set up a multi-container environment using Docker for Hadoop (HDFS), Spark, and Hive. The setup is lightweight, without the large memory requirements of a Cloudera sandbox.\n\n## **Prerequisites**\n\nBefore you begin, ensure you have the following installed:\n\n- **Docker**: [Install Docker Desktop for Windows](https://docs.docker.com/desktop/setup/install/windows-install/)\n\n- IMPORTANT:\n  ******- Enable the \"Expose daemon on tcp://localhost:2375 without TLS\" option if you're using Docker Desktop for compatibility.******\n\n![image](https://github.com/user-attachments/assets/398451cd-46bb-4ba8-876f-9e85f8c0d632)\n\n\n - **Git**: [Download Git](https://git-scm.com/downloads/win)\n   - Git is used to download the required files from a repository.\n  \n   Create a newfolder and open it in terminal or go inside it using CD Command\n  \n ![image](https://github.com/user-attachments/assets/28602a4b-52e2-4265-bfb5-a08301fda7b8)\n\n\n## **Step 1: Clone the Repository**\n\nFirst, clone the GitHub repository that contains the necessary Docker setup files.\n\n```bash\ngit clone https://github.com/lovnishverma/bigdataecosystem.git\n```\n \n[or Directly download zip from my repo](https://github.com/lovnishverma/BigDataecosystem)\n\nNavigate to the directory:\n\n```bash\ncd bigdataecosystem\n```\n\n![image](https://github.com/user-attachments/assets/e4d6a8ab-3f36-424a-bf13-9402bc1c13a2)\n\nif downloaded zip than cd bigdataecosystem-main\n\n## **Step 2: Start the Cluster**\n\nUse Docker Compose to start the containers in the background.\n\n```bash\ndocker-compose up -d\n```\n\nThis command will launch the Hadoop, Spark, and Hive containers.\n\n![image](https://github.com/user-attachments/assets/8dc3ec44-84af-40f2-8056-92e5f3449919)\n\n\n## **Step 3: Verify Running Containers**\n\nTo check if the containers are running, use the following command:\n\n```bash\ndocker ps\n```\n![image](https://github.com/user-attachments/assets/f6897172-d14f-462a-95dd-ba46401b5dd7)\n\n\n## **Step 4: Stop and Remove Containers**\n\nWhen you are done, stop and remove the containers with:\n\n```bash\ndocker-compose down\n```\n![image](https://github.com/user-attachments/assets/fd1f2298-7d65-4055-a929-12de4d01c428)\n\n\n### Step 5: Access the NameNode container\nEnter the NameNode container to interact with Hadoop:\n```bash\ndocker exec -it namenode bash\n```\n** -it refers to (interactive terminal)**\n---\n\n## **Running Hadoop Code** \n\nTo View NameNode UI Visit:   [http://localhost:9870/](http://localhost:9870/)\n\n![image](https://github.com/user-attachments/assets/c4f708cb-7976-49f8-ba79-8b985bcd6a10)\n\n\nTo View Resource Manager UI Visit [http://localhost:8088/](http://localhost:8088/)\n\n![image](https://github.com/user-attachments/assets/a65f2495-293e-440c-8366-9e1bed605b29)\n\n\n### ** MAPREDUCE WordCount program**\n### Step 1: Copy the `code` folder into the container\nUse the following command in your windows cmd to copy the `code` folder to the container:\n```bash\ndocker cp code namenode:/\n```\n\n![image](https://github.com/user-attachments/assets/7acdebdc-2b20-41bf-b92d-8555091d570c)\n\n\n### Step 2: Locate the `data.txt` file\nInside the container, navigate to the `code/input` directory where the `data.txt` file is located.\n\n### Step 3: Create directories in the Hadoop file system\nRun the following commands to set up directories in Hadoop's file system:\n```bash\nhdfs dfs -mkdir /user\nhdfs dfs -mkdir /user/root\nhdfs dfs -mkdir /user/root/input\n```\n\n### Step 4: Upload the `data.txt` file\nCopy `data.txt` into the Hadoop file system:\n```bash\nhdfs dfs -put /code/input/data.txt /user/root/input\n```\n![image](https://github.com/user-attachments/assets/31fadc17-1c8c-4621-bdee-39d818f3da2c)\n\n\n### Step 5: Navigate to the directory containing the `wordCount.jar` file\nReturn to the directory where the `wordCount.jar` file is located:\n```bash\ncd /code/\n```\n![image](https://github.com/user-attachments/assets/4242e3b2-c954-4faf-ab75-825906eeafc5)\n\n\n### Step 6: Execute the WordCount program \n\nTo View NameNode UI Visit:   [http://localhost:9870/](http://localhost:9870/)\n\n![image](https://github.com/user-attachments/assets/20681490-0fcc-41dd-874a-8fe0376dc981)\n\n\nRun the WordCount program to process the input data:\n```bash\nhadoop jar wordCount.jar org.apache.hadoop.examples.WordCount input output\n```\n![image](https://github.com/user-attachments/assets/2bafcdd5-be22-471c-bf9a-6b8a48d88d44)\n\n\nTo View YARN Resource Manager UI Visit [http://localhost:8088/](http://localhost:8088/)\n\n![image](https://github.com/user-attachments/assets/89f47e9f-c92f-456c-b89e-0e6025df80e2)\n\n### Step 7: Display the output\nView the results of the WordCount program:\n```bash\nhdfs dfs -cat /user/root/output/*\n```\n![image](https://github.com/user-attachments/assets/8a20f77f-71bd-423b-a501-c9514ec9f825)\n\n---\n\n**or**\n\n```bash\nhdfs dfs -cat /user/root/output/part-r-00000\n```\n\n![image](https://github.com/user-attachments/assets/a4ef5293-1018-4c5e-a314-91681d430715)\n\n\n## **Summary**\n\nThis guide simplifies setting up and running Hadoop on Docker. Each step ensures a smooth experience, even for beginners without a technical background. Follow the instructions carefully, and you’ll have a working Hadoop setup in no time!\n\nCertainly! Here’s the explanation of your **MapReduce process** using the input example `DOG CAT RAT`, `CAR CAR RAT`, and `DOG CAR CAT`.\n---\n\n## 🐾 **Input Data**\n\nThe `data.txt` file contains the following lines:\n\n```\nDOG CAT RAT\nCAR CAR RAT\nDOG CAR CAT\n```\n\nThis text file is processed by the **MapReduce WordCount program** to count the occurrences of each word.\n\n---\n\n## 💡 **What is MapReduce?**\n\n- **MapReduce** is a two-step process:\n  1. **Map Phase** 🗺️: Splits the input into key-value pairs.\n  2. **Reduce Phase** ➕: Combines the key-value pairs to produce the final result.\n\nIt's like dividing a big task (word counting) into smaller tasks and then combining the results. 🧩\n\n---\n\n## 🔄 **How MapReduce Works in Your Example**\n\n### **1. Map Phase** 🗺️\n\nThe mapper processes each line of the input file, splits it into words, and assigns each word a count of `1`.\n\nFor example:\n```\nDOG CAT RAT  -\u003e (DOG, 1), (CAT, 1), (RAT, 1)\nCAR CAR RAT  -\u003e (CAR, 1), (CAR, 1), (RAT, 1)\nDOG CAR CAT  -\u003e (DOG, 1), (CAR, 1), (CAT, 1)\n```\n\n**Mapper Output**:\n```\n(DOG, 1), (CAT, 1), (RAT, 1)\n(CAR, 1), (CAR, 1), (RAT, 1)\n(DOG, 1), (CAR, 1), (CAT, 1)\n```\n\n---\n\n### **2. Shuffle and Sort Phase** 🔄\n\nThis step groups all values for the same key (word) together and sorts them.\n\nFor example:\n```\n(CAR, [1, 1, 1])\n(CAT, [1, 1])\n(DOG, [1, 1])\n(RAT, [1, 1])\n```\n\n---\n\n### **3. Reduce Phase** ➕\n\nThe reducer sums up the counts for each word to get the total number of occurrences.\n\n**Reducer Output**:\n```\nCAR 3  🏎️\nCAT 2  🐱\nDOG 2  🐶\nRAT 2  🐭\n```\n\n---\n\n### **Final Output** 📋\n\nThe final word count is saved in the HDFS output directory. You can view it using:\n```bash\nhdfs dfs -cat /user/root/output/*\n```\n\n**Result**:\n```\nCAR 3\nCAT 2\nDOG 2\nRAT 2\n```\n\n---\n\n## 🗂️ **HDFS Commands You Used**\n\nHere are the basic HDFS commands you used and their purpose:\n\n1. **Upload a file to HDFS** 📤:\n   ```bash\n   hdfs dfs -put data.txt /user/root/input\n   ```\n   - **What it does**: Uploads `data.txt` to the HDFS directory `/user/root/input`.\n   - **Output**: No output, but the file is now in HDFS.\n\n2. **List files in a directory** 📁:\n   ```bash\n   hdfs dfs -ls /user/root/input\n   ```\n   - **What it does**: Lists all files in the `/user/root/input` directory.\n   - **Output**: Something like this:\n     ```\n     Found 1 items\n     -rw-r--r--   1 root supergroup        50  2024-12-12  /user/root/input/data.txt\n     ```\n\n3. **View the contents of a file** 📄:\n   ```bash\n   hdfs dfs -cat /user/root/input/data.txt\n   ```\n   - **What it does**: Displays the contents of the `data.txt` file in HDFS.\n   - **Output**:\n     ```\n     DOG CAT RAT\n     CAR CAR RAT\n     DOG CAR CAT\n     ```\n\n4. **Run the MapReduce Job** 🚀:\n   ```bash\n   hadoop jar wordCount.jar org.apache.hadoop.examples.WordCount input output\n   ```\n   - **What it does**: Runs the WordCount program on the input directory and saves the result in the output directory.\n\n5. **View the final output** 📊:\n   ```bash\n   hdfs dfs -cat /user/root/output/*\n   ```\n   - **What it does**: Displays the word count results.\n   - **Output**:\n     ```\n     CAR 3\n     CAT 2\n     DOG 2\n     RAT 2\n     ```\n\n---\n\n## 🛠️ **How You Utilized MapReduce**\n\n1. **Input**:  \n   You uploaded a small text file (`data.txt`) to HDFS.\n\n2. **Process**:  \n   The `WordCount` program processed the file using MapReduce:\n   - The **mapper** broke the file into words and counted each occurrence.\n   - The **reducer** aggregated the counts for each word.\n\n3. **Output**:  \n   The results were saved in HDFS and displayed using the `cat` command.\n\n---\n\n## 🧩 **Visualization of the Entire Process**\n\n### **Input** (HDFS file):\n```\nDOG CAT RAT\nCAR CAR RAT\nDOG CAR CAT\n```\n\n### **Map Phase Output** 🗺️:\n```\n(DOG, 1), (CAT, 1), (RAT, 1)\n(CAR, 1), (CAR, 1), (RAT, 1)\n(DOG, 1), (CAR, 1), (CAT, 1)\n```\n\n### **Shuffle \u0026 Sort** 🔄:\n```\n(CAR, [1, 1, 1])\n(CAT, [1, 1])\n(DOG, [1, 1])\n(RAT, [1, 1])\n```\n\n### **Reduce Phase Output** ➕:\n```\nCAR 3\nCAT 2\nDOG 2\nRAT 2\n```\n\n---\n\n![image](https://github.com/user-attachments/assets/a037fc47-7639-48b8-b3f7-5d9f2d5c51ac)\n\n### 🔑 **Key Takeaways**\n- **MapReduce** splits the task into small, manageable pieces and processes them in parallel.\n- It’s ideal for large datasets but works the same for smaller ones (like your example).\n- Hadoop is designed for distributed systems, making it powerful for big data processing.\n\n\n\n\n\n\n### . **Stopping the Containers**  \nTo stop the Docker containers when done:\n```bash\ndocker-compose down\n```\nThis will stop and remove the containers and networks created by `docker-compose up`.\n\n### 4. **Permissions Issue with Copying Files**  \nIf you face permission issues while copying files to containers ensure the correct directory permissions in Docker by using:\n```bash\ndocker exec -it namenode bash\nchmod -R 777 /your-directory\n```\n\n### 5. **Additional Debugging Tips**  \nSometimes, containers might not start or might throw errors related to Hadoop configuration. A small troubleshooting section or references to common issues (e.g., insufficient memory for Hadoop) would be helpful.\n\n### 6. **Final Output File Path**  \nThe output of the WordCount job will be written to `/user/root/output/` in HDFS. This is clearly explained, but you could also include a note that the output directory might need to be created beforehand to avoid errors.\n\n---\n\n### **Example Additions:**\n\n1. **Network Issues:**\n   ```\n   If you can't access the NameNode UI, ensure that your Docker container's ports are correctly exposed. For example, if you're running a local machine, the UI should be accessible via http://localhost:9870.\n   ```\n   \n2. **Stopping Containers:**\n   ```bash\n   docker-compose down  # Stop and remove the containers\n   ```\n\n3. **Permissions Fix:**\n   ```bash\n   docker exec -it namenode bash\n   chmod -R 777 /your-directory  # If you face any permission errors\n   ```\n\n4. **Handling HDFS Directory Creation:**\n   If `hdfs dfs -mkdir` gives an error, it may be because the directory already exists. Consider adding:\n   ```bash\n   hdfs dfs -rm -r /user/root/input  # If the directory exists, remove it first\n   hdfs dfs -mkdir /user/root/input\n   ```\n\n---\n\n😊 References\n\nhttps://data-flair.training/blogs/top-hadoop-hdfs-commands-tutorial/\n\nhttps://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html\n\nhttps://medium.com/@traininghub.io/hadoop-mapreduce-architecture-7e167e264595\n\n\n## **Step 5: Set Up HDFS**\n\n### **Upload Files to HDFS**\n\nTo copy a file (e.g., `police.csv`) to the Hadoop cluster:\n\n1. Copy the file into the namenode container:\n    ```bash\n    docker cp police.csv namenode:/police.csv\n    ```\n![image](https://github.com/user-attachments/assets/496c7e6a-41d6-44d2-9557-b6004fe986c4)\n\n\n2. Access the namenode container's bash shell:\n    ```bash\n    docker exec -it namenode bash\n    ```\n![image](https://github.com/user-attachments/assets/d501a9b3-d2d9-4e2d-aecb-8e3eb7ccf678)\n\n\n3. Create a directory in HDFS and upload the file:\n    ```bash\n    hdfs dfs -mkdir -p /data/crimerecord/police\n    hdfs dfs -put /police.csv /data/crimerecord/police/\n    ```\n![image](https://github.com/user-attachments/assets/ab68bba9-92f2-4b15-a50e-f3ee1a0f998e)\n\n\n\n![image](https://github.com/user-attachments/assets/6b27db66-a111-4c2f-a701-2cef8aaa3344)\n\n\n### **Start Spark Shell**\n\nTo interact with Spark, start the Spark shell in the master container:\n\n```bash\ndocker exec -it spark-master bash\n\nspark/bin/spark-shell --master spark://spark-master:7077\n```\n### **Access the Spark Master UI**\n\n- Open `http://localhost:8080` in your web browser to view the Spark Master UI.\n- **You can monitor processes here**\n\n- ![image](https://github.com/user-attachments/assets/8fa7e525-d601-4dad-b5b4-0477d47ec4dd)\n\n\n![image](https://github.com/user-attachments/assets/45765d5e-b1e7-4726-a60c-ddd5dd278c93)\n\n![image](https://github.com/user-attachments/assets/b071335b-4928-491a-8bed-321995881d83)\n\n# **Working with Apache Spark**\n\n## **1. Introduction to Apache Spark**\n\n- **Overview**: Apache Spark is an open-source distributed computing system known for its speed, ease of use, and general-purpose capabilities for big data processing.\n\n- **Key Features**:\n  - Fast processing using in-memory computation.\n  - Supports multiple languages: Scala, Python, Java, and R.\n  - Unified framework for batch and streaming data processing.\n\n---\n\n## **2. Introduction to DataFrames**\n\n- **What are DataFrames?**\n  - Distributed collections of data organized into named columns, similar to a table in a database or a DataFrame in Python's pandas.\n  - Optimized for processing large datasets using Spark SQL.\n\n- **Key Operations**:\n  - Creating DataFrames from structured data sources (CSV, JSON, Parquet, etc.).\n  - Performing transformations and actions on the data.\n\n---\n\n## **3. Introduction to Scala for Apache Spark**\n\n- **Why Scala?**\n  - Apache Spark is written in Scala, offering the best compatibility and performance.\n  - Concise syntax and functional programming support.\n\n- **Basic Syntax**:\n\n```scala\nval numbers = List(1, 2, 3, 4, 5)      // Creates a list of numbers.\nval doubled = numbers.map(_ * 2)       // Doubles each element in the list using map.\nprintln(doubled)                        // Prints the doubled list.\n```\nThe output will be:\nList(2, 4, 6, 8, 10)\n\n---\n\n## **4. Spark SQL**\n\n- **Need for Spark SQL**:\n  - Provides a declarative interface to query structured data using SQL-like syntax.\n  - Supports seamless integration with other Spark modules.\n  - Allows for optimization through Catalyst Optimizer.\n\n- **Key Components**:\n  - SQL Queries on DataFrames and temporary views.\n  - Hive integration for legacy SQL workflows.\n  - Support for structured data sources.\n\n---\n## **5. Hands-On: Spark SQL**\n\n### **Objective**:\nTo create DataFrames, load data from different sources, and perform transformations and SQL queries.\n\n\n#### **Step 1: Create DataFrames**\n\n```scala\nval data = Seq(\n  (\"Alice\", 30, \"HR\"),\n  (\"Bob\", 25, \"Engineering\"),\n  (\"Charlie\", 35, \"Finance\")\n)\n\nval df = data.toDF(\"Name\", \"Age\", \"Department\")\n\ndf.show()\n```\n![image](https://github.com/user-attachments/assets/06c2c14f-cf8e-4b38-8944-7844e75ee5d6)\n\n\n#### **Step 3: Perform Transformations Using Spark SQL**\n\n```scala\ndf.createOrReplaceTempView(\"employees\")\nval result = spark.sql(\"SELECT Department, COUNT(*) as count FROM employees GROUP BY Department\")\nresult.show()\n```\n![image](https://github.com/user-attachments/assets/c9125138-63dd-4c29-82c4-6d04bc531508)\n\n\n#### **Step 4: Save Transformed Data**\n\n```scala\nresult.write.option(\"header\", \"true\").csv(\"hdfs://namenode:9000/output_employees\")\n```\n\nReading from HDFS:\nOnce the data is written to HDFS, you can read it back into Spark using:\n\n```scala\nval outputDF = spark.read.option(\"header\", \"true\").csv(\"hdfs://namenode:9000/output_employees\")\n ```\n\nView output_employees.csv from HDFS\n\n```scala\noutputDF.show()\n ```\n![image](https://github.com/user-attachments/assets/a4bb7af6-2ee6-485f-a306-371165e5bf37)\n\n\n#### **Step 5: Load Data from HDFS**\n\n```scala\n// Load CSV from HDFS\nval df = spark.read.option(\"header\", \"false\").csv(\"hdfs://namenode:9000/data/crimerecord/police/police.csv\")\ndf.show()\n```\n\n![image](https://github.com/user-attachments/assets/f6dfde78-f44a-4554-9c0f-f11cb9173e6c)\n\n\n#### **Step 6: Scala WordCount using Apache Spark**\n\n\n### Docker Command to Copy File\n*Copy File**: Use `docker cp` to move or create the file inside the namenode Docker container.\nUse the following command to copy the `data.txt` file from your local system to the Docker container:\n\n```bash\ndocker cp data.txt nodemanager:/data.txt\n```\n![image](https://github.com/user-attachments/assets/73a84d9a-af1c-45f0-9504-a24b192e598d)\n\n*Copy File to HDFS**: Use `hdfs dfs -put` to move the file inside the HDFS filesystem.\nUse the following command to put the `data.txt` file from your Docker container to HDFS:\n\n```bash\nhdfs dfs -mkdir /data\nhdfs dfs -put data.txt /data\n```\n![image](https://github.com/user-attachments/assets/b4d93f36-f1b1-4056-a4af-d4dbb418634e)\n\n**Scala WordCount program.**\n\n**WordCount Program**: The program reads the file, splits it into words, and counts the occurrences of each word.\n\n```scala\nimport org.apache.spark.{SparkConf}\nval conf = new SparkConf().setAppName(\"WordCountExample\").setMaster(\"local\")\nval input = sc.textFile(\"hdfs://namenode:9000/data/data.txt\")\nval wordPairs = input.flatMap(line =\u003e line.split(\" \")).map(word =\u003e (word, 1))\nval wordCounts = wordPairs.reduceByKey((a, b) =\u003e a + b)\nwordCounts.collect().foreach { case (word, count) =\u003e\n  println(s\"$word: $count\")\n}\n```\n\n**Output**: The word counts will be printed to the console when the program is executed.\n\n![image](https://github.com/user-attachments/assets/428e0d99-f0e0-4edd-8f3c-4543130c8a47)\n\n\n**Stop Session**:\n\n```scala\nsc.stop()\n```\n\n---\n\n## **6. Key Takeaways**\n\n- Spark SQL simplifies working with structured data.\n- DataFrames provide a flexible and powerful API for handling large datasets.\n- Apache Spark is a versatile tool for distributed data processing, offering scalability and performance.\n\n---\n\n\n![image](https://github.com/user-attachments/assets/fada1eec-5349-4382-8d1a-96940c124064)\n\n## **Step 7: Set Up Hive** \n\n### **Start Hive Server**\n\nAccess the Hive container and start the Hive Server:\n\n```bash\ndocker exec -it hive-server bash\n```\n\n```bash\nhive\n```\n\nCheck if Hive is listening on port 10000:\n![image](https://github.com/user-attachments/assets/dc1e78d4-d903-4ac5-9eaa-eff0b893d6fb)\n\n\n```bash\nnetstat -anp | grep 10000\n```\n![image](https://github.com/user-attachments/assets/9ac08fd3-f515-448d-83b3-c620fa3b15c2)\n\n\n### **Connect to Hive Server**\n\nUse Beeline to connect to the Hive server:\n\n```bash\nbeeline -u jdbc:hive2://localhost:10000 -n root\n```\n![image](https://github.com/user-attachments/assets/d2dce309-0334-4a64-b8df-8cb6206b1432)\n\n\nAlternatively, use the following command for direct connection:\n\n```bash\nbeeline\n```\n\n```bash\n!connect jdbc:hive2://127.0.0.1:10000 scott tiger\n```\n\n![image](https://github.com/user-attachments/assets/77fadb1f-118e-4d15-8a78-e9783baa9690)\n\n\n### **Create Database and Table in Hive**\n\n1. Create a new Hive database:\n    ```sql\n    CREATE DATABASE punjab_police;\n    USE punjab_police;\n    ```\n![image](https://github.com/user-attachments/assets/73227817-b2d5-4df0-a392-6927750d7220)\n\n\n2. Create a table based on the schema of the `police.csv` dataset:\n    ```sql\n    CREATE TABLE police_data (\n        Crime_ID INT,\n        Crime_Type STRING,\n        Location STRING,\n        Reported_Date STRING,\n        Status STRING\n    )\n    ROW FORMAT DELIMITED\n    FIELDS TERMINATED BY ','\n    STORED AS TEXTFILE;\n    ```\n    ![image](https://github.com/user-attachments/assets/13faa21a-5242-4f1e-bd69-4d98dc318400)\n\n\n3. Load the data into the Hive table:\n    ```sql\n    LOAD DATA INPATH '/data/crimerecord/police/police.csv' INTO TABLE police_data;\n    ```\n![image](https://github.com/user-attachments/assets/e0fcbe55-d5fd-4a8c-a17b-df888204915f)\n\n\n### **Query the Data in Hive**\n\nRun SQL queries to analyze the data in Hive:\n\n1. **View the top 10 rows:**\n    ```sql\n    SELECT * FROM police_data LIMIT 10;\n    ```\n![image](https://github.com/user-attachments/assets/6f189765-24f4-47db-ad70-42fbcfb4068e)\n\n\n2. **Count total crimes:**\n    ```sql\n    SELECT COUNT(*) AS Total_Crimes FROM police_data;\n    ```\n![image](https://github.com/user-attachments/assets/8b56a8b5-6b0b-4306-82da-4cce52b50e95)\n\n\n3. **Find most common crime types:**\n    ```sql\n    SELECT Crime_Type, COUNT(*) AS Occurrences\n    FROM police_data\n    GROUP BY Crime_Type\n    ORDER BY Occurrences DESC;\n    ```\n\n   ![image](https://github.com/user-attachments/assets/54f000f7-36ec-4672-8bc6-996ac7b4004b)\n\n\n4. **Identify locations with the highest crime rates:**\n    ```sql\n    SELECT Location, COUNT(*) AS Total_Crimes\n    FROM police_data\n    GROUP BY Location\n    ORDER BY Total_Crimes DESC;\n    ```\n![image](https://github.com/user-attachments/assets/fb418097-97ff-46aa-941a-4b72a0702d3d)\n\n\n5. **Find unresolved cases:**\n    ```sql\n    SELECT Status, COUNT(*) AS Count\n    FROM police_data\n    WHERE Status != 'Closed'\n    GROUP BY Status;\n    ```\n![image](https://github.com/user-attachments/assets/9b3b32df-38c9-45bd-85dc-c4ac2b16b246)\n\n\n**********There you go: your private Hive server to play with.**********\n\nshow databases;\n\n![image](https://github.com/user-attachments/assets/7e8e65b1-cb98-41e2-b655-ddf941b614d5)\n\n#### **📂 Part 2: Creating a Simple Hive Project**\n\n---\n\n##### **🎯 Objective**\nWe will:\n1. Create a database.\n2. Create a table inside the database.\n3. Load data into the table.\n4. Run queries to retrieve data.\n\n---\n\n##### **💾 Step 1: Create a Database**\nIn the Beeline CLI:\n```sql\nCREATE DATABASE mydb;\nUSE mydb;\n```\n- 📝 *`mydb` is the name of the database. Replace it with your preferred name.*\n\n---\n\n##### **📋 Step 2: Create a Table**\nStill in the Beeline CLI, create a simple table:\n```sql\nCREATE TABLE employees (\n    id INT,\n    name STRING,\n    age INT\n);\n```\n- This creates a table named `employees` with columns `id`, `name`, and `age`.\n\n---\n\n##### **📥 Step 3: Insert Data into the Table**\nInsert sample data into your table:\n```sql\nINSERT INTO employees VALUES (1, 'Prince', 30);\nINSERT INTO employees VALUES (2, 'Ram Singh', 25);\n```\n\n---\n\n##### **🔍 Step 4: Query the Table**\nRetrieve data from your table:\n```sql\nSELECT * FROM employees;\n```\n- Output:\n\n![image](https://github.com/user-attachments/assets/63529cb9-c74d-453e-a4d7-9f176762a8bc)\n\n\n  ```\n  +----+----------+-----+\n  | id |   name   | age |\n  +----+----------+-----+\n  | 2  | Ram Singh |  25 |\n  | 1  | Prince     | 30 |\n  +----+----------+-----+\n  ```\n\n---\n\n#### **🌟 Tips \u0026 Knowledge**\n\n1. **What is Hive?**\n   - Hive is a data warehouse tool on top of Hadoop.\n   - It allows SQL-like querying over large datasets.\n\n2. **Why Docker for Hive?**\n   - Simplifies setup by avoiding manual configurations.\n   - Provides a pre-configured environment for running Hive.\n\n3. **Beeline CLI**:\n   - A lightweight command-line tool for running Hive queries.\n\n4. **Use Cases**:\n   - **Data Analysis**: Run analytics on large datasets.\n   - **ETL**: Extract, Transform, and Load data into your Hadoop ecosystem.\n\n---\n\n#### **🎉 You're Ready!**\nYou’ve successfully:\n1. Set up Apache Hive.\n2. Created and queried a sample project.  🐝\n\n### **🐝 Apache Hive Basic Commands**\n\nHere is a collection of basic Apache Hive commands with explanations that can help you while working with Hive:\n\n---\n\n#### **1. Database Commands**\n\n- **Show Databases:**\n  Displays all the databases available in your Hive environment.\n  ```sql\n  SHOW DATABASES;\n  ```\n\n- **Create a Database:**\n  Create a new database.\n  ```sql\n  CREATE DATABASE \u003cdatabase_name\u003e;\n  ```\n  Example:\n  ```sql\n  CREATE DATABASE mydb;\n  ```\n  In Hive, you can find out which database you are currently using by running the following command:\n\n```sql\nSELECT current_database();\n```\n\nThis will return the name of the database that is currently in use.\n\nAlternatively, you can use this command:\n\n```sql\nUSE database_name;\n```\n\nIf you want to explicitly switch to a specific database or verify the database context, you can use this command before running your queries.\n\n- **Use a Database:**\n  Switch to the specified database.\n  ```sql\n  USE \u003cdatabase_name\u003e;\n  ```\n  Example:\n  ```sql\n  USE mydb;\n  ```\n  \n\n- **Drop a Database:**\n  Deletes a database and its associated data.\n  ```sql\n  DROP DATABASE \u003cdatabase_name\u003e;\n  ```\n\n---\n\n#### **2. Table Commands**\n\n- **Show Tables:**\n  List all the tables in the current database.\n  ```sql\n  SHOW TABLES;\n  ```\n\n- **Create a Table:**\n  Define a new table with specific columns.\n  ```sql\n  CREATE TABLE \u003ctable_name\u003e (\n      column_name column_type,\n      ...\n  );\n  ```\n  Example:\n  ```sql\n  CREATE TABLE employees (\n      id INT,\n      name STRING,\n      age INT\n  );\n  ```\n\n- **Describe a Table:**\n  Get detailed information about a table, including column names and types.\n  ```sql\n  DESCRIBE \u003ctable_name\u003e;\n  ```\n\n- **Drop a Table:**\n  Deletes a table and its associated data.\n  ```sql\n  DROP TABLE \u003ctable_name\u003e;\n  ```\n\n- **Alter a Table:**\n  Modify a table structure, like adding new columns.\n  ```sql\n  ALTER TABLE \u003ctable_name\u003e ADD COLUMNS (\u003cnew_column\u003e \u003ctype\u003e);\n  ```\n  Example:\n  ```sql\n  ALTER TABLE employees ADD COLUMNS (salary DOUBLE);\n  ```\n\n---\n\n#### **3. Data Manipulation Commands**\n\n- **Insert Data:**\n  Insert data into a table.\n  ```sql\n  INSERT INTO \u003ctable_name\u003e VALUES (\u003cvalue1\u003e, \u003cvalue2\u003e, ...);\n  INSERT INTO employees VALUES (1, 'Prince', 30), (2, 'Ram Singh', 25), (3, 'John Doe', 28), (4, 'Jane Smith', 32);\n  ```\n  Example:\n  ```sql\n  INSERT INTO employees VALUES (1, 'John Doe', 30);\n  \n  ```\n\n- **Select Data:**\n  Retrieve data from a table.\n  ```sql\n  SELECT * FROM \u003ctable_name\u003e;\n  ```\n\n- **Update Data:**\n  Update existing data in a table.\n  ```sql\n  UPDATE \u003ctable_name\u003e SET \u003ccolumn_name\u003e = \u003cnew_value\u003e WHERE \u003ccondition\u003e;\n  ```\n\n- **Delete Data:**\n  Delete rows from a table based on a condition.\n  ```sql\n  DELETE FROM \u003ctable_name\u003e WHERE \u003ccondition\u003e;\n  ```\n\n---\n\n#### **4. Querying Commands**\n\n- **Select Specific Columns:**\n  Retrieve specific columns from a table.\n  ```sql\n  SELECT \u003ccolumn1\u003e, \u003ccolumn2\u003e FROM \u003ctable_name\u003e;\n  ```\n\n- **Filtering Data:**\n  Filter data based on conditions using the `WHERE` clause.\n  ```sql\n  SELECT * FROM \u003ctable_name\u003e WHERE \u003ccolumn_name\u003e \u003coperator\u003e \u003cvalue\u003e;\n  ```\n  Example:\n  ```sql\n  SELECT * FROM employees WHERE age \u003e 25;\n  ```\n\n- **Sorting Data:**\n  Sort the result by a column in ascending or descending order.\n  ```sql\n  SELECT * FROM \u003ctable_name\u003e ORDER BY \u003ccolumn_name\u003e ASC|DESC;\n  ```\n  Example:\n  ```sql\n  SELECT * FROM employees ORDER BY age DESC;\n  SELECT * FROM employees ORDER BY age ASC;\n  ```\n\n- **Group By:**\n  Group data by one or more columns and aggregate it using functions like `COUNT`, `AVG`, `SUM`, etc.\n  ```sql\n  SELECT \u003ccolumn_name\u003e, COUNT(*) FROM \u003ctable_name\u003e GROUP BY \u003ccolumn_name\u003e;\n  ```\n  Example:\n  ```sql\n  SELECT age, COUNT(*) FROM employees GROUP BY age;\n  ```\n\n---\n\n#### **5. File Format Commands**\n\n- **Create External Table:**\n  Create a table that references data stored externally (e.g., in HDFS).\n  ```sql\n  CREATE EXTERNAL TABLE \u003ctable_name\u003e (\u003ccolumn_name\u003e \u003cdata_type\u003e, ...) \n  ROW FORMAT DELIMITED\n  FIELDS TERMINATED BY '\u003cdelimiter\u003e'\n  LOCATION '\u003cfile_path\u003e';\n  ```\n  Example:\n  ```sql\n  CREATE EXTERNAL TABLE employees (\n      id INT,\n      name STRING,\n      age INT\n  ) ROW FORMAT DELIMITED\n  FIELDS TERMINATED BY ','\n  LOCATION '/user/hive/warehouse/employees';\n  ```\n\n- **Load Data into Table:**\n  Load data from a file into an existing Hive table.\n  ```sql\n  LOAD DATA LOCAL INPATH '\u003cfile_path\u003e' INTO TABLE \u003ctable_name\u003e;\n  ```\n\n---\n\n#### **6. Other Useful Commands**\n\n- **Show Current User:**\n  Display the current user running the Hive session.\n  ```sql\n  !whoami;\n  ```\n\n- **Exit Hive:**\n  Exit from the Hive shell.\n  ```sql\n  EXIT;\n  ```\n\n- **Set Hive Variables:**\n  Set Hive session variables.\n  ```sql\n  SET \u003cvariable_name\u003e=\u003cvalue\u003e;\n  ```\n\n- **Show Hive Variables:**\n  Display all the set variables.\n  ```sql\n  SET;\n  ```\n\n- **Show the Status of Hive Jobs:**\n  Display the status of running queries.\n  ```sql\n  SHOW JOBS;\n  ```\n\n---\n\n#### **🌟 Tips \u0026 Best Practices**\n\n- **Partitioning Tables:**\n  When dealing with large datasets, partitioning your tables can help improve query performance.\n  ```sql\n  CREATE TABLE sales (id INT, amount DOUBLE)\n  PARTITIONED BY (year INT, month INT);\n  ```\n\n- **Bucketing:**\n  Bucketing splits your data into a fixed number of files or \"buckets.\"\n  ```sql\n  CREATE TABLE sales (id INT, amount DOUBLE)\n  CLUSTERED BY (id) INTO 4 BUCKETS;\n  ```\n\n- **Optimization:**\n  Use columnar formats like `ORC` or `Parquet` for efficient storage and performance.\n  ```sql\n  CREATE TABLE sales (id INT, amount DOUBLE)\n  STORED AS ORC;\n  ```\n\nThese basic commands will help you interact with Hive and perform common operations like creating tables, querying data, and managing your Hive environment efficiently.\n\nWhile **Hive** and **MySQL** both use SQL-like syntax for querying data, there are some key differences in their commands, especially since Hive is designed for querying large datasets in a Hadoop ecosystem, while MySQL is a relational database management system (RDBMS).\n\n##**Here’s a comparison of **Hive** and **MySQL** commands in terms of common operations:**\n\n### **1. Creating Databases**\n- **Hive**:\n   ```sql\n   CREATE DATABASE mydb;\n   ```\n\n- **MySQL**:\n   ```sql\n   CREATE DATABASE mydb;\n   ```\n\n   *Both Hive and MySQL use the same syntax to create a database.*\n\n---\n\n### **2. Switching to a Database**\n- **Hive**:\n   ```sql\n   USE mydb;\n   ```\n\n- **MySQL**:\n   ```sql\n   USE mydb;\n   ```\n\n   *The syntax is the same for selecting a database in both systems.*\n\n---\n\n### **3. Creating Tables**\n- **Hive**:\n   ```sql\n   CREATE TABLE employees (\n       id INT,\n       name STRING,\n       age INT\n   );\n   ```\n\n- **MySQL**:\n   ```sql\n   CREATE TABLE employees (\n       id INT,\n       name VARCHAR(255),\n       age INT\n   );\n   ```\n\n   **Differences**:\n   - In Hive, **STRING** is used for text data, while in MySQL, **VARCHAR** is used.\n   - Hive also has some specialized data types for distributed storage and performance, like `ARRAY`, `MAP`, `STRUCT`, etc.\n\n---\n\n### **4. Inserting Data**\n- **Hive**:\n   ```sql\n   INSERT INTO employees VALUES (1, 'John', 30);\n   INSERT INTO employees VALUES (2, 'Alice', 25);\n   ```\n\n- **MySQL**:\n   ```sql\n   INSERT INTO employees (id, name, age) VALUES (1, 'John', 30);\n   INSERT INTO employees (id, name, age) VALUES (2, 'Alice', 25);\n   ```\n\n   **Differences**:\n   - Hive allows direct `INSERT INTO` with values, while MySQL explicitly lists column names in the insert statement (though this is optional in MySQL if the columns match).\n\n---\n\n### **5. Querying Data**\n- **Hive**:\n   ```sql\n   SELECT * FROM employees;\n   ```\n\n- **MySQL**:\n   ```sql\n   SELECT * FROM employees;\n   ```\n\n   *Querying data using `SELECT` is identical in both systems.*\n\n---\n\n### **6. Modifying Data**\n- **Hive**:\n   Hive doesn’t support traditional **UPDATE** or **DELETE** commands directly, as it is optimized for batch processing and is more suited for append operations. However, it does support **INSERT** and **INSERT OVERWRITE** operations.\n\n   Example of replacing data:\n   ```sql\n   INSERT OVERWRITE TABLE employees SELECT * FROM employees WHERE age \u003e 30;\n   ```\n\n- **MySQL**:\n   ```sql\n   UPDATE employees SET age = 31 WHERE id = 1;\n   DELETE FROM employees WHERE id = 2;\n   ```\n\n   **Differences**:\n   - Hive does not allow direct **UPDATE** or **DELETE**; instead, it uses **INSERT OVERWRITE** to modify data in batch operations.\n\n---\n\n### **7. Dropping Tables**\n- **Hive**:\n   ```sql\n   DROP TABLE IF EXISTS employees;\n   ```\n\n- **MySQL**:\n   ```sql\n   DROP TABLE IF EXISTS employees;\n   ```\n\n   *The syntax for dropping tables is the same in both systems.*\n\n---\n\n### **8. Query Performance**\n- **Hive**:\n   - Hive is designed to run on large datasets using the Hadoop Distributed File System (HDFS), so it focuses more on **batch processing** rather than real-time queries. Query performance in Hive may be slower than MySQL because it’s optimized for scale, not for low-latency transaction processing.\n\n- **MySQL**:\n   - MySQL is an RDBMS, designed to handle **transactional workloads** with low-latency queries. It’s better suited for OLTP (Online Transaction Processing) rather than OLAP (Online Analytical Processing) workloads.\n\n---\n\n### **9. Indexing**\n- **Hive**:\n   - Hive doesn’t support traditional indexing as MySQL does. However, you can create **partitioned** or **bucketed** tables in Hive to improve query performance for certain types of data.\n\n- **MySQL**:\n   - MySQL supports **indexes** (e.g., **PRIMARY KEY**, **UNIQUE**, **INDEX**) to speed up query performance on large datasets.\n\n---\n\n### **10. Joins**\n- **Hive**:\n   ```sql\n   SELECT a.id, a.name, b.age\n   FROM employees a\n   JOIN employee_details b ON a.id = b.id;\n   ```\n\n- **MySQL**:\n   ```sql\n   SELECT a.id, a.name, b.age\n   FROM employees a\n   JOIN employee_details b ON a.id = b.id;\n   ```\n\n   *The syntax for **JOIN** is the same in both systems.*\n\n---\n\n### **Summary of Key Differences**:\n- **Data Types**: Hive uses types like `STRING`, `TEXT`, `BOOLEAN`, etc., while MySQL uses types like `VARCHAR`, `CHAR`, `TEXT`, etc.\n- **Data Modification**: Hive does not support **UPDATE** or **DELETE** in the traditional way, and is generally used for **batch processing**.\n- **Performance**: Hive is designed for querying large-scale datasets in Hadoop, so queries tend to be slower than MySQL.\n- **Indexing**: Hive does not natively support indexing but can use partitioning and bucketing for performance optimization. MySQL supports indexing for faster queries.\n- **ACID Properties**: MySQL supports full ACID compliance for transactional systems, whereas Hive is not transactional by default (but can support limited ACID features starting from version 0.14 with certain configurations).\n\nIn conclusion, while **Hive** and **MySQL** share SQL-like syntax, they are designed for very different use cases, and not all commands work the same way in both systems.\n\n### **Visualize the Data (Optional)**\n\nExport the query results to a CSV file for analysis in visualization tools:\n\n```bash\nhive -e \"SELECT * FROM police_data;\" \u003e police_analysis_results.csv\n```\n\nYou can use tools like Tableau, Excel, or Python (Matplotlib, Pandas) for data visualization.\n\n## **Step 8: Configure Environment Variables (Optional)**\n\nIf you need to customize configurations, you can specify parameters in the `hadoop.env` file or as environmental variables for services (e.g., namenode, datanode, etc.). For example:\n\n```bash\nCORE_CONF_fs_defaultFS=hdfs://namenode:8020\n```\n\nThis will be transformed into the following in the `core-site.xml` file:\n\n```xml\n\u003cproperty\u003e\n    \u003cname\u003efs.defaultFS\u003c/name\u003e\n    \u003cvalue\u003ehdfs://namenode:8020\u003c/value\u003e\n\u003c/property\u003e\n```\n\n## **Conclusion**\n\nYou now have a fully functional Hadoop, Spark, and Hive cluster running in Docker. This environment is great for experimenting with big data processing and analytics in a lightweight, containerized setup.\n\n---\n\nI hope you have fun with this Hadoop-Spark-Hive cluster.\n\n\n\n![image](https://github.com/user-attachments/assets/1347d354-a160-4cc6-8547-eb0857a72ba5)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flovnishverma%2Fbigdataecosystem","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flovnishverma%2Fbigdataecosystem","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flovnishverma%2Fbigdataecosystem/lists"}