{"id":27699743,"url":"https://github.com/spidey-acer/python-project-big-data","last_synced_at":"2026-02-12T02:40:25.401Z","repository":{"id":249059922,"uuid":"830489838","full_name":"Spidey-Acer/Python-Project-Big-Data","owner":"Spidey-Acer","description":"Big Data","archived":false,"fork":false,"pushed_at":"2024-12-16T16:32:40.000Z","size":28523,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-04-25T17:53:21.917Z","etag":null,"topics":["data-science","machine-learning","machine-learning-algorithms","python"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Spidey-Acer.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-07-18T11:26:23.000Z","updated_at":"2025-01-24T03:13:26.000Z","dependencies_parsed_at":"2024-12-13T08:27:39.339Z","dependency_job_id":"3cec1923-ef36-4e66-8b35-e17eecb01f58","html_url":"https://github.com/Spidey-Acer/Python-Project-Big-Data","commit_stats":null,"previous_names":["peter-kibet/python-projects","spidey-acer/python-projects","spidey-acer/python-project-big-data"],"tags_count":0,"template":false,"template_full_name":"microverseinc/readme-template","purl":"pkg:github/Spidey-Acer/Python-Project-Big-Data","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spidey-Acer%2FPython-Project-Big-Data","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spidey-Acer%2FPython-Project-Big-Data/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spidey-Acer%2FPython-Project-Big-Data/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spidey-Acer%2FPython-Project-Big-Data/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Spidey-Acer","download_url":"https://codeload.github.com/Spidey-Acer/Python-Project-Big-Data/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spidey-Acer%2FPython-Project-Big-Data/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29355264,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-12T01:03:07.613Z","status":"online","status_checked_at":"2026-02-12T02:00:06.911Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-science","machine-learning","machine-learning-algorithms","python"],"created_at":"2025-04-25T17:29:25.185Z","updated_at":"2026-02-12T02:40:25.396Z","avatar_url":"https://github.com/Spidey-Acer.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Data Processing using PySpark\n\n## Task 1: DataFrame Creation with REGEX\n\nEach member will define a custom schema using REGEX to extract specific metrics from the dataset.\n\n### Student Metrics to Extract\n\n- **Student 1**: IP Address, Timestamp, HTTP Method  \n   **REGEX Example**: `(\\d+\\.\\d+\\.\\d+\\.\\d+) - - \\[(.*?)\\] \\\"([A-Z]+)`\n- **Student 2**: HTTP Status Code, Response Size, Timestamp  \n   **REGEX Example**: `\\\".*\\\" (\\d+) (\\d+) \\[(.*?)\\]`\n- **Student 3**: URL Path, IP Address, Response Size  \n   **REGEX Example**: `\\\"[A-Z]+ (\\/.*?) HTTP.* (\\d+\\.\\d+\\.\\d+\\.\\d+) (\\d+)`\n- **Student 4**: Log Message, HTTP Status Code, Timestamp  \n   **REGEX Example**: `\\\".*\\\" (\\d+) .* \\[(.*?)\\] (.*)`\n\n## Task 2: Two Advanced DataFrame Analysis\n\nEach member will write unique SQL queries for the analysis:\n\n### SQL Query 1: Window Functions\n\n- **Student 1**: Rolling hourly traffic per IP  \n   **Description**: Calculate traffic count per IP over a sliding window.\n- **Student 2**: Session identification  \n   **Description**: Identify sessions based on timestamp gaps.\n- **Student 3**: Unique visitors per hour  \n   **Description**: Count distinct IPs for each hour.\n- **Student 4**: Average response size per status code  \n   **Description**: Compute averages grouped by status codes.\n\n### SQL Query 2: Aggregation Functions\n\n- **Student 1**: Traffic patterns by URL path  \n   **Description**: Analyze URL visits by hour.\n- **Student 2**: Top 10 failed requests by size  \n   **Description**: Identify the largest failed requests.\n- **Student 3**: Response size distribution by status  \n   **Description**: Show min, max, and avg sizes for each status.\n- **Student 4**: Daily unique visitors  \n   **Description**: Count unique IPs per day.\n\n## Task 3: Data Visualization using Matplotlib and Seaborn\n\nEach member will visualize the results of their unique SQL queries using different chart types.\n\n### Student Visualization Type Examples\n\n- **Student 1**: Line Chart (Hourly Traffic)  \n   **Tool**: Matplotlib for rolling traffic visualization.\n- **Student 2**: Bar Chart (Top 10 Failed Requests)  \n   **Tool**: Seaborn for aggregated failure counts.\n- **Student 3**: Heatmap (Hourly Unique Visitors)  \n   **Tool**: Seaborn for visualizing traffic density.\n- **Student 4**: Pie Chart (Response Code Distribution)  \n   **Tool**: Matplotlib for status code proportions.\n\n# Data Processing using PySpark RDD\n\n## Task 1: Basic RDD Analysis\n\nEach member will create a custom function to parse and process the log entries.\n\n### Student Basic Extraction Examples\n\n- **Student 1**: Extract Timestamp and IP  \n   **Description**: Parse timestamp and IP address from logs.\n- **Student 2**: Extract URL and HTTP Method  \n   **Description**: Parse URL path and HTTP method from logs.\n- **Student 3**: Extract Status Code and Response Size  \n   **Description**: Parse HTTP status and response size from logs.\n- **Student 4**: Extract Log Message and IP Address  \n   **Description**: Parse log messages and corresponding IP addresses.\n\n## Task 2: Two Advanced RDD Analysis\n\nEach member will perform unique advanced processing tasks.\n\n### Student Advanced Analysis Examples\n\n- **Student 1**: Calculate hourly visit counts per IP  \n   **Description**: Count visits grouped by hour and IP.\n- **Student 2**: Identify top 10 URLs by visit count  \n   **Description**: Aggregate visit counts and rank top URLs.\n- **Student 3**: Find average response size per URL  \n   **Description**: Compute average response size for each URL.\n- **Student 4**: Detect failed requests per IP  \n   **Description**: Identify IPs with the most failed requests.\n\n## Optimization and LSEPI Considerations\n\nEach member chooses two unique methods for optimization.\n\n### Student Optimization Methods\n\n- **Student 1**: Partition Strategies, Caching\n- **Student 2**: Caching, Bucketing \u0026 Indexing\n- **Student 3**: Partition Strategies, Bucketing \u0026 Indexing\n- **Student 4**: Caching, Partition Strategies\n\n## Starter Code\n\n```json\n{\n  \"nbformat\": 4,\n  \"nbformat_minor\": 0,\n  \"metadata\": {\n    \"colab\": {\n      \"provenance\": [],\n      \"collapsed_sections\": [\"Lx9-Fre4FMda\"]\n    },\n    \"kernelspec\": {\n      \"name\": \"python3\",\n      \"display_name\": \"Python 3\"\n    }\n  },\n  \"cells\": [\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"Lx9-Fre4FMda\"\n      },\n      \"source\": [\n        \"# Big Data Analytics [CN7031] CRWK 2024-25\\n\",\n        \"# Group ID: [Your Group ID]\\n\",\n        \"1.   Student 1: Name and ID\\n\",\n        \"2.   Student 2: Name and ID\\n\",\n        \"3.   Student 3: Name and ID\\n\",\n        \"4.   Student 4: Name and ID\\n\",\n        \"\\n\",\n        \"---\\n\",\n        \"\\n\",\n        \"If you want to add comments on your group work, please write it here for us:\"\n      ]\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"GdMZR-9QTwG3\"\n      },\n      \"source\": [\"\\n\", \"# Initiate and Configure Spark\\n\", \"\\n\", \"---\\n\"]\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"2wbXV70D6xbl\"\n      },\n      \"source\": [\"!pip3 install pyspark\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"_z0p88Xtw_3-\"\n      },\n      \"source\": [\"# linking with Spark\\n\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"6P2CZVl6TOQX\"\n      },\n      \"source\": [\"# Load Unstructured Data\\n\", \"\\n\", \"---\\n\"]\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\n        \"# Load the unstructured data: (1) drag and drop data on the \\\"Files\\\" section or (2) use Google Drive\"\n      ],\n      \"metadata\": {\n        \"id\": \"efdQkCg_soaq\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"-RjT7_UHAqic\"\n      },\n      \"source\": [\n        \"\\n\",\n        \"# Task 1: Data Processing using PySpark DF [40 marks]\\n\",\n        \"\\n\",\n        \"---\\n\",\n        \"\\n\"\n      ]\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 1 (Name and ID)\\n\",\n        \"\\n\",\n        \"- DF Creation with REGEX (10 marks)\\n\",\n        \"- Two advanced DF Analysis (20 marks)\\n\",\n        \"- Utilize data visualization (10 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"LSE7bNND4caH\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"7fCFTcOQBLD2\"\n      },\n      \"source\": [\"# Task 1 - Student 1\\n\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 2 (Name and ID)\\n\",\n        \"\\n\",\n        \"- DF Creation with REGEX (10 marks)\\n\",\n        \"- Two advanced DF Analysis (20 marks)\\n\",\n        \"- Utilize data visualization (10 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"QkJNiyVu4tKK\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 1 - Student 2\\n\"],\n      \"metadata\": {\n        \"id\": \"kHPoRpSD4vYW\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 3 (Name and ID)\\n\",\n        \"\\n\",\n        \"- DF Creation with REGEX (10 marks)\\n\",\n        \"- Two advanced DF Analysis (20 marks)\\n\",\n        \"- Utilize data visualization (10 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"JFiwgD4H4vph\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 1 - Student 3\\n\"],\n      \"metadata\": {\n        \"id\": \"-TZIFMZB4xFZ\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 4 (Name and ID)\\n\",\n        \"\\n\",\n        \"- DF Creation with REGEX (10 marks)\\n\",\n        \"- Two advanced DF Analysis (20 marks)\\n\",\n        \"- Utilize data visualization (10 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"F7AQAa574xnx\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 1 - Student 4\\n\"],\n      \"metadata\": {\n        \"id\": \"5KsnRrDK4zRB\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"dcJhGbI2BKpx\"\n      },\n      \"source\": [\n        \"\\n\",\n        \"# Task 2 - Data Processing using PySpark RDD [40 marks]\\n\",\n        \"\\n\",\n        \"---\\n\"\n      ]\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 1 (Name and ID)\\n\",\n        \"\\n\",\n        \"- One Basic RDD Analysis (10 marks)\\n\",\n        \"- Two Advanced RDD Analysis (30 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"mDEDGQOh450o\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"V3eiN9geBQRf\"\n      },\n      \"source\": [\"# Task 2 - Student 1\\n\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 2 (Name and ID)\\n\",\n        \"\\n\",\n        \"- One Basic RDD Analysis (10 marks)\\n\",\n        \"- Two Advanced RDD Analysis (30 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"92RPdoeV5SHz\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"FQ_-hgdeiMle\"\n      },\n      \"source\": [\"# Task 2 - Student 2\\n\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 3 (Name and ID)\\n\",\n        \"\\n\",\n        \"- One Basic RDD Analysis (10 marks)\\n\",\n        \"- Two Advanced RDD Analysis (30 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"y7MY1leq5USZ\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"2JGQHXYliMK5\"\n      },\n      \"source\": [\"# Task 2 - Student 3\\n\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 4 (Name and ID)\\n\",\n        \"\\n\",\n        \"- One Basic RDD Analysis (10 marks)\\n\",\n        \"- Two Advanced RDD Analysis (30 marks)\"\n      ],\n      \"metadata\": {\n        \"id\": \"n8G2vN3g5Vua\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"A5mwMvIsBQlX\"\n      },\n      \"source\": [\"# Task 2 - Student 4\\n\"],\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"wHft1Jht1Qxl\"\n      },\n      \"source\": [\n        \"# (3) Optimization and LSEPI (Legal, Social, Ethical, and Professional Issues) Considerations [10 marks]\\n\",\n        \"\\n\",\n        \"---\\n\"\n      ]\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 1 (Name and ID)\\n\",\n        \"\\n\",\n        \"Choose two out of the following three methods to apply. Compare results with and without optimization for the chosen methods.\\n\",\n        \"\\n\",\n        \"- Different Partition Strategies (5 Marks)\\n\",\n        \"  - Explore and evaluate various strategies for partitioning data.\\n\",\n        \"\\n\",\n        \"- Caching vs. No Caching (5 Marks)\\n\",\n        \"  - Analyze the impact of caching data versus not caching.\\n\",\n        \"\\n\",\n        \"- Bucketing and Indexing (5 Marks)\\n\",\n        \"  - Investigate the effects of bucketing and indexing on data operations.\"\n      ],\n      \"metadata\": {\n        \"id\": \"95m9jb8f5d_s\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 3 - Student 1\\n\"],\n      \"metadata\": {\n        \"id\": \"8dbo5dG25ra2\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 2 (Name and ID)\\n\",\n        \"\\n\",\n        \"Choose two out of the following three methods to apply. Compare results with and without optimization for the chosen methods.\\n\",\n        \"\\n\",\n        \"- Different Partition Strategies (5 Marks)\\n\",\n        \"  - Explore and evaluate various strategies for partitioning data.\\n\",\n        \"\\n\",\n        \"- Caching vs. No Caching (5 Marks)\\n\",\n        \"  - Analyze the impact of caching data versus not caching.\\n\",\n        \"\\n\",\n        \"- Bucketing and Indexing (5 Marks)\\n\",\n        \"  - Investigate the effects of bucketing and indexing on data operations.\"\n      ],\n      \"metadata\": {\n        \"id\": \"cQpYG-4k5rrq\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 3 - Student 2\\n\"],\n      \"metadata\": {\n        \"id\": \"8ZTAGJiz5tIX\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 3 (Name and ID)\\n\",\n        \"\\n\",\n        \"Choose two out of the following three methods to apply. Compare results with and without optimization for the chosen methods.\\n\",\n        \"\\n\",\n        \"- Different Partition Strategies (5 Marks)\\n\",\n        \"  - Explore and evaluate various strategies for partitioning data.\\n\",\n        \"\\n\",\n        \"- Caching vs. No Caching (5 Marks)\\n\",\n        \"  - Analyze the impact of caching data versus not caching.\\n\",\n        \"\\n\",\n        \"- Bucketing and Indexing (5 Marks)\\n\",\n        \"  - Investigate the effects of bucketing and indexing on data operations.\"\n      ],\n      \"metadata\": {\n        \"id\": \"thZJwceS5tX7\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 3 - Student 3\\n\"],\n      \"metadata\": {\n        \"id\": \"WOFn2U7F5urh\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"source\": [\n        \"# Student 4 (Name and ID)\\n\",\n        \"\\n\",\n        \"Choose two out of the following three methods to apply. Compare results with and without optimization for the chosen methods.\\n\",\n        \"\\n\",\n        \"- Different Partition Strategies (5 Marks)\\n\",\n        \"  - Explore and evaluate various strategies for partitioning data.\\n\",\n        \"\\n\",\n        \"- Caching vs. No Caching (5 Marks)\\n\",\n        \"  - Analyze the impact of caching data versus not caching.\\n\",\n        \"\\n\",\n        \"- Bucketing and Indexing (5 Marks)\\n\",\n        \"  - Investigate the effects of bucketing and indexing on data operations.\"\n      ],\n      \"metadata\": {\n        \"id\": \"uX-rH0Uz5u-2\"\n      }\n    },\n    {\n      \"cell_type\": \"code\",\n      \"source\": [\"# Task 3 - Student 4\\n\"],\n      \"metadata\": {\n        \"id\": \"Gu3ere9c5wJ4\"\n      },\n      \"execution_count\": null,\n      \"outputs\": []\n    },\n    {\n      \"cell_type\": \"markdown\",\n      \"metadata\": {\n        \"id\": \"mIM6uLApSxi2\"\n      },\n      \"source\": [\n        \"# Convert ipynb to HTML for Turnitin submission [10 marks]\\n\",\n        \"\\n\",\n        \"---\\n\",\n        \"\\n\"\n      ]\n    },\n    {\n      \"cell_type\": \"code\",\n      \"metadata\": {\n        \"id\": \"ZrQu11N_DCfZ\"\n      },\n      \"source\": [\n        \"# install nbconvert\\n\",\n        \"#!pip3 install nbconvert\\n\",\n        \"\\n\",\n        \"\\n\",\n        \"# convert ipynb to html\\n\",\n        \"# file name: \\\"Your_Group_ID_CN7031.ipynb\\n\",\n        \"!jupyter nbconvert --to html Your_Group_ID_CN7031.ipynb\"\n      ],\n      \"execution_count\": null,\n      \"outputs\": []\n    }\n  ]\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspidey-acer%2Fpython-project-big-data","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspidey-acer%2Fpython-project-big-data","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspidey-acer%2Fpython-project-big-data/lists"}