{"id":19682501,"url":"https://github.com/rayyan9477/dep","last_synced_at":"2026-05-08T06:07:10.382Z","repository":{"id":250763594,"uuid":"835403682","full_name":"Rayyan9477/DEP","owner":"Rayyan9477","description":null,"archived":false,"fork":false,"pushed_at":"2024-07-29T19:28:57.000Z","size":3390,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-10T06:07:26.823Z","etag":null,"topics":["data","data-science","machine-learning","python","visualization","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Rayyan9477.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-07-29T19:06:22.000Z","updated_at":"2024-07-29T19:50:11.000Z","dependencies_parsed_at":"2024-07-30T01:26:48.609Z","dependency_job_id":"6a4b1ff2-1ca5-4140-a9a0-a267b7596bd8","html_url":"https://github.com/Rayyan9477/DEP","commit_stats":null,"previous_names":["rayyan9477/dep"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rayyan9477%2FDEP","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rayyan9477%2FDEP/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rayyan9477%2FDEP/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Rayyan9477%2FDEP/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Rayyan9477","download_url":"https://codeload.github.com/Rayyan9477/DEP/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240990959,"owners_count":19889983,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data","data-science","machine-learning","python","visualization","web-scraping"],"created_at":"2024-11-11T18:11:04.469Z","updated_at":"2026-05-08T06:07:05.363Z","avatar_url":"https://github.com/Rayyan9477.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Digital Empowerment Network\n\n## Task 1: Data Extraction and Initial Analysis (`DEP_Task1.ipynb`)\n**Objective**: Extract data from given sources and perform initial analysis.\n\n### Steps:\n1. **Data Collection**:\n   - Extracted data from various sources using web scraping techniques.\n   - Libraries used: `BeautifulSoup`, `requests`.\n\n2. **Data Cleaning**:\n   - Processed raw data to remove noise and irrelevant information.\n   - Handled missing values and standardized data formats.\n\n3. **Initial Analysis**:\n   - Conducted exploratory data analysis (EDA) to understand data distribution and identify key patterns.\n   - Visualized data using libraries like `matplotlib` and `seaborn`.\n\n### Key Achievements:\n- Successfully extracted data from multiple sources.\n- Cleaned and preprocessed data for further analysis.\n- Identified initial insights and patterns in the data.\n\n## Task 2: Text Analysis and NLP (`DEP_TASK_2.ipynb`)\n**Objective**: Perform textual analysis and compute various NLP metrics.\n\n### Steps:\n1. **Text Extraction**:\n   - Extracted article text from URLs provided in an Excel file.\n   - Ensured extraction of only relevant content (title and body).\n\n2. **Text Analysis**:\n   - Computed various NLP metrics such as positive score, negative score, polarity score, subjectivity score, etc.\n   - Libraries used: `TextBlob`, `nltk`.\n\n3. **Output Generation**:\n   - Structured the computed metrics as per the provided output format.\n   - Saved the results in an Excel file.\n\n### Key Achievements:\n- Efficiently extracted and processed textual data.\n- Computed and analyzed various NLP metrics.\n- Generated structured output for further use.\n\n## Task 3: Advanced Data Processing (`Task_3.ipynb`)\n**Objective**: Advanced data processing and analysis using Python.\n\n### Steps:\n1. **Advanced Data Cleaning**:\n   - Applied advanced data cleaning techniques to handle complex datasets.\n   - Used regular expressions and custom functions for specific cleaning tasks.\n\n2. **Feature Engineering**:\n   - Created new features to enhance data analysis.\n   - Techniques used: aggregation, normalization, and transformation.\n\n3. **Data Analysis and Visualization**:\n   - Conducted in-depth analysis using advanced statistical methods.\n   - Visualized complex relationships and trends using `plotly` and `seaborn`.\n\n### Key Achievements:\n- Applied advanced data cleaning and feature engineering techniques.\n- Conducted comprehensive data analysis.\n- Created interactive and insightful visualizations.\n\n## Task 4: Anomaly Detection in Network Traffic (`Task_4.ipynb`)\n**Objective**: Detect anomalies in network traffic data using machine learning techniques.\n\n### Steps:\n1. **Data Loading and Preprocessing**:\n   - Loaded network traffic data from a CSV file.\n   - Converted time columns to datetime format and calculated the duration of network sessions.\n\n2. **Feature Extraction and Normalization**:\n   - Extracted relevant features such as bytes_in, bytes_out, and duration.\n   - Normalized the features using StandardScaler for consistent scaling.\n\n3. **Anomaly Detection**:\n   - Applied the Isolation Forest algorithm to detect anomalies in the data.\n   - Identified anomalies based on the algorithm's predictions.\n\n4. **Evaluation and Visualization**:\n   - Evaluated the performance of the anomaly detection using a confusion matrix.\n   - Visualized the results using Plotly and Seaborn for better understanding and presentation.\n\n### Key Achievements:\n- Successfully detected anomalies in network traffic data using machine learning.\n- Evaluated and validated the anomaly detection results.\n- Visualized anomalies effectively to highlight patterns and insights.\n\n## Conclusion\nThroughout the internship, I gained hands-on experience in data extraction, cleaning, and analysis using Python. I successfully completed tasks involving web scraping, NLP, advanced data processing, and anomaly detection, enhancing my skills in data science and analytics.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayyan9477%2Fdep","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frayyan9477%2Fdep","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frayyan9477%2Fdep/lists"}