{"id":15199017,"url":"https://github.com/ansh-info/Hadoop-Pipeline","last_synced_at":"2025-10-02T14:32:03.889Z","repository":{"id":255643622,"uuid":"841986914","full_name":"ansh-info/Hadoop-DataPipeline","owner":"ansh-info","description":"An end-to-end data engineering pipeline to collect, store, process, and analyze property and crime data using Hadoop, Docker, MySQL, Tailscale, and Selenium","archived":false,"fork":false,"pushed_at":"2024-09-13T20:05:01.000Z","size":17934,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-09-28T02:01:15.228Z","etag":null,"topics":["docker","docker-compose","hadoop","jupyter-notebook","mapreduce","python","selenium","sql","tailscale"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ansh-info.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-08-13T12:50:23.000Z","updated_at":"2024-09-11T09:54:53.000Z","dependencies_parsed_at":"2024-09-28T02:00:38.216Z","dependency_job_id":null,"html_url":"https://github.com/ansh-info/Hadoop-DataPipeline","commit_stats":{"total_commits":24,"total_committers":2,"mean_commits":12.0,"dds":0.04166666666666663,"last_synced_commit":"ef0503704b701195f7857ec7345bdbf1b64db84f"},"previous_names":["ansh-info/hadoop-datapipeline"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ansh-info%2FHadoop-DataPipeline","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ansh-info%2FHadoop-DataPipeline/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ansh-info%2FHadoop-DataPipeline/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ansh-info%2FHadoop-DataPipeline/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ansh-info","download_url":"https://codeload.github.com/ansh-info/Hadoop-DataPipeline/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":219875714,"owners_count":16554705,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["docker","docker-compose","hadoop","jupyter-notebook","mapreduce","python","selenium","sql","tailscale"],"created_at":"2024-09-28T02:00:29.452Z","updated_at":"2025-10-02T14:32:02.091Z","avatar_url":"https://github.com/ansh-info.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Property and Locality Data Analysis\n\n## Introduction\nThis project involves setting up a data engineering pipeline to collect, store, process, and analyze Property and Locality data using Hadoop, Docker, MySQL, Tailscale, and Selenium.\n\n## Project Overview\n- **Objective:** Analyze Property and Locality data to derive meaningful insights.\n- **Scope:** Collect data through web scraping, store in HDFS, process using Hadoop, and analyze with MySQL.\n\n# Video\n\nHere is a demo video of the Pipeline:\n\n[![Watch the demo video](https://img.youtube.com/vi/0sdQdd7E67o/maxresdefault.jpg)](https://www.youtube.com/watch?v=0sdQdd7E67o)\n\n## Setup and Environment\n### Virtual Machines Setup\n- **Step:** Installed Ubuntu on VirtualBox for each VM.\n- **Action:** Configured each VM with necessary packages including Docker and Docker Compose.\n\n### Networking with Tailscale\n- **Step:** Installed and configured Tailscale on all VMs.\n- **Action:** Created a secure virtual network to enable communication between VMs.\n\n### Docker Swarm Initialization\n- **Step:** Initialized Docker Swarm on the master node and joined worker nodes.\n- **Action:** Used Docker Swarm for container orchestration.\n\n### Image of Setup Process\n![Setup Process - hidden name for privacy concern](/images/setup.png)\n\n## Data Collection\n### Web Scraping with Selenium\n- **Step:** Installed Selenium and Chrome WebDriver.\n- **Action:** Developed scripts to scrape Property and Locality data from various websites.\n\n### Image of Collection Process\n![Data Collection Process](/images/collection.png)\n\n## Data Storage in HDFS\n- **Step:** `cd spark cluster` folder to use, A ready to go Big Data cluster (Hadoop + Hadoop Streaming + Spark + PySpark + Jupyter Notebook) with Docker and Docker Swarm! \nConfigured HDFS on the Hadoop cluster provided by `Prof. Dr.-Ing.` [Binh Vu](https://github.com/binhvd). Check the README.md file in the spark cluster folder to begin with the setup process`\n- **Action:** Stored scraped data in HDFS with appropriate partitioning and replication.\n\n### Image of Data Storage in HDFS Process\n![Data Storage in HDFS Process](/images/datastorage.png)\n\n## Data Processing\n### Hadoop Job Development\n- **Step:** Developed and executed Hadoop jobs for data cleaning and transformation.\n- **Action:** Used MapReduce for distributed processing.\n\n### Image of Data Processing Process\n![Data Processing Process](/images/dataprocessing.png)\n\n## Failure Test\n- **Step:** Conducted data read/write operations while intentionally shutting down a worker node.\n- **Action:** Verified system resilience and fault tolerance.\n\n### Image of Failure Test Process - node down\n![Failure Test Process - node down](/images/failuretest.png)\n\n### Image of Failure Test Process - ingestion to mysql\n![Failure Test Process - ingestion to mysql](/images/ingestion.png)\n\n## Data Ingestion into MySQL\n### Database Design\n- **Step:** Created a relational database schema in MySQL.\n- **Action:** Developed scripts to ingest data from HDFS to MySQL.\n\n### Image of Database Schema\n![Database Schema](/images/datamodeling.png)\n\n## Business Insights\n### Query Development\n- **Step:** Developed SQL queries to extract insights from the database.\n- **Action:** Generated graphs and tables to present the results.\n\n### Image of Business Insights Visualization\n![Business Insights](/images/crimessolved.png)\n\n## Acknowledgements\n- **Note:** A special thank you to `Prof. Dr.-Ing.` [Binh Vu](https://github.com/binhvd) for providing the ready-to-go Spark cluster image used in this project.\n---\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fansh-info%2FHadoop-Pipeline","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fansh-info%2FHadoop-Pipeline","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fansh-info%2FHadoop-Pipeline/lists"}