{"id":26667652,"url":"https://github.com/neelka96/havocmapper","last_synced_at":"2026-02-15T04:31:51.820Z","repository":{"id":271442992,"uuid":"913070341","full_name":"Neelka96/HavocMapper","owner":"Neelka96","description":"Data Visualization Project 1 Shared Repo","archived":false,"fork":false,"pushed_at":"2025-02-21T03:52:13.000Z","size":43310,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-07-13T10:45:08.197Z","etag":null,"topics":["2u","api","api-rest","dataviz","dfi","edx","etl","geolocation","geolocation-api","geoviews","holoviews","hvplot","jupyter-notebook","matplotlib","matplotlib-pyplot","pandas","pandas-dataframe","python","seaborn"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Neelka96.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-07T01:13:53.000Z","updated_at":"2025-04-10T04:37:32.000Z","dependencies_parsed_at":"2025-01-07T19:58:27.430Z","dependency_job_id":"e1e02c4d-7a0c-4196-85b1-8f6c0685b16d","html_url":"https://github.com/Neelka96/HavocMapper","commit_stats":null,"previous_names":["neelka96/havocmapper"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/Neelka96/HavocMapper","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neelka96%2FHavocMapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neelka96%2FHavocMapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neelka96%2FHavocMapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neelka96%2FHavocMapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Neelka96","download_url":"https://codeload.github.com/Neelka96/HavocMapper/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Neelka96%2FHavocMapper/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29469214,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-15T04:31:33.243Z","status":"ssl_error","status_checked_at":"2026-02-15T04:30:38.729Z","response_time":118,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["2u","api","api-rest","dataviz","dfi","edx","etl","geolocation","geolocation-api","geoviews","holoviews","hvplot","jupyter-notebook","matplotlib","matplotlib-pyplot","pandas","pandas-dataframe","python","seaborn"],"created_at":"2025-03-25T19:40:05.011Z","updated_at":"2026-02-15T04:31:51.791Z","avatar_url":"https://github.com/Neelka96.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# HavocMapper: FEMA Responses to Extreme Weather  \n\n## Introduction  \n\nHavocMapper is a data analysis project exploring FEMA's Disaster Declarations Summary dataset and using  \nGeoapify's geocoding API to uncover insights into disaster responses across the United States. This  \nproject applies foundational Python, Pandas, Matplotlib, and API skills to clean, analyze, and visualize  \ndata trends. Learn more about FEMA's work on their [website](https://www.fema.gov/) and Geoapify's API [here](https://www.geoapify.com/).  \n\n\n## Table of Contents  \n\n- [Introduction](#introduction)  \n- [Setup](#setup)  \n  - [Quick-Run Instructions](#quick-run-instructions)  \n  - [Notebook Execution Order](#notebook-execution-order)  \n  - [Reading Notebooks Effectively](#reading-notebooks-effectively)  \n- [General Overview](#general-overview)  \n  - [Major Technologies Used](#major-technologies-used)  \n  - [System Requirements](#system-requirements)  \n    - [Hardware](#hardware)\n    - [Software](#software)\n    - [Manual Dependencies](#manual-dependencies)\n- [Detailed Description](#detailed-description)  \n  - [Workflow](#workflow)  \n  - [Purpose and Goals](#purpose-and-goals)  \n  - [Techniques Applied](#techniques-applied)  \n  - [Preferred Viewing Method](#preferred-viewing-method)  \n- [Key Challenges](#key-challenges)  \n- [Directory Structures](#directory-structures)  \n  - [Original Cloned Repo](#original-cloned-repo)  \n  - [New Expected Structure](#new-expected-structure)  \n- [Deliverables](#deliverables)  \n  - [Visualizations](#visualizations)  \n  - [Analyses](#analyses)  \n  - [Saved Figures and Models](#saved-figures-and-models)  \n  - [Google Slides Presentation](#google-slides-presentation)  \n- [Contributors](#contributors)  \n- [Data Sources and Citations](#data-sources-and-citations)  \n- [Long Form Directory Structure](#long-form-directory-structure)\n\n\n## Setup  \n\n### Quick-Run Instructions  \n\n1. Clone the repository in the local directory of your choosing with `cd your/path/here`  \n    - Using the HTTPS web URL:  \n        ```bash  \n        git clone https://github.com/neelka96/HavocMapper.git  \n        cd HavocMapper  \n        ```  \n    - Using a password protected SSH Key (if you have it enabled other wise use method above):  \n        ```bash  \n        git clone git@github.com:Neelka96/HavocMapper.git  \n        cd HavocMapper  \n        ```  \n2. Create a virtual environment (optional but recommended)  \n    - With Conda:  \n        ```bash  \n        conda create --name venv python=3.12.2  \n        conda activate venv  \n        ```  \n    - With Built-in Python venv Library:  \n        ```bash  \n        python -m venv venv  \n        source venv/bin/activate  # On Windows: venv\\Scripts\\activate  \n        ```  \n3. Install dependencies:  \n    ```bash  \n    pip install -r requirements.txt  \n    ```  \n4. Feel free to deactivate your virtual environment (conda or otherwise) after installing dependencies!  \n    - With Conda:  \n        ```bash  \n        conda deactivate  \n        ```  \n    - With venv:  \n        ```bash  \n        deactivate  \n        ```  \n5. Add API keys in a file named `api_keys.py`:  \n    ```python  \n    key1 = \"your_api_key_here\"  \n    key2 = \"your_optional_second_key_here\"\n    key3 = \"your_optional_third_key_here\"\n    ...\n    # Add additional keys as needed  \n    # Native setup utilizes up to 9 keys  \n    # Must have 1 key with no upwards limit  \n    ```  \n6. When running notebooks, be sure to select the kernel/environment the dependencies were installed in.  \n\u003e [!NOTE]  \n\u003e While the native system all code was built on runs through the conda python interpreter   \n\u003e (Anaconda Distribution), all dependencies were loaded through pip to ensure cross-compatibility  \n\u003e with other Python interpreters. Anaconda's official statement is that they can not ensure the  \n\u003e resolution of dependency conflicts from packages installed through Pypi, BUT conda does naturally  \n\u003e house pip installed dependencies separately from each other when installed in different conda  \n\u003e environments. When creating a new environment from scratch, this process will almost always  \n\u003e function as intended!\n\n\n### Notebook Execution Order  \n\n1. Run `havoc_ETL.ipynb` to set up directory, prepare data and geocode locations.  \n2. Follow with `havoc_analysis.ipynb` for visualizations and analyses, and export images.  \n\n### Reading Notebooks Effectively  \n\n- Use Jupyter Notebook's Outline feature (e.g., in the VS Code extension) to collapse and expand sections for better navigation.  \n- Outputs and markdown are organized with headings and foldable cells for clarity.  \n\u003e [!TIP]\n\u003e If running cell by cell, collapse all grouped cells first and then expand and collapse as needed!  \n\u003e Markdown has been thoroughly labeled and grouped for enhanced traversal.  \n\n\n## General Overview  \n\nThis project follows a structured workflow to explore and analyze FEMA's disaster declarations. Data is extracted,  \ncleaned, geocoded, and visualized to identify trends in extreme weather responses, leveraging modern Python  \nlibraries and API integrations.  \n\n\n### Major Technologies Used  \n\n- Python 3.12.2  \n- Jupyter Notebook  \n- pandas  \n- matplotlib  \n- requests  \n\n\n### System Requirements  \n\n#### Hardware  \n\n- A modern computer with at least 8GB RAM and 2GHz multi-core CPU.  \n- Internet access for API calls.  \n\n#### Software  \n\n- Python 3.12 or higher  \n- Recommended: Conda for environment management.  \n\n#### Manual Dependencies \nIf not using `requirements.txt` please pip install the following if you don't already have these libraries:  \n- pandas  \n- pathlib  \n- matplotlib  \n- holoviews  \n- geoviews  \n- hvplot  \n- seaborn  \n- dataframe\\_image  \n\n\n## Detailed Description  \n\n### Workflow  \n1. **Data Extraction**:  \n  - Download and clean FEMA dataset.  \n  - Apply minor and major cleaning steps, including regex operations and vector mappings.  \n  - Sets up HavocMapper/assets directory  \n2. **API Integration**:  \n  - Geocode locations using Geoapify API.  \n  - Utilize `ThreadPoolExecutor` for multi-threaded API calls.  \n3. **Data Aggregation**:  \n  - Summarize FEMA merged geocoded data via groupby to reduce data points.  \n  - Export multiple intermediate and final CSV files.  \n4. **Analysis and Visualization**:  \n  - Load cleaned data for analysis.  \n  - Analyze via different lenses of time, states, types of disasters, and response times.\n  - Generate visualizations such as maps, line plots, and bar charts, and scatter plots.  \n\n### Purpose and Goals  \n\n- Provide insights into FEMA disaster responses over time.  \n- Highlight the efficiency and accuracy of FEMA's dataset.  \n- Identify trends and anomalies in disaster declarations.  \n\n### Techniques Applied  \n\n- Regex for text data cleaning.  \n- Multi-threading for API efficiency.  \n- Vectorized operations for improved performance.  \n- Exploratory data analysis (EDA) with Pandas and Matplotlib.  \n\n### Preferred Viewing Method  \n\n- View notebooks in VS Code or Jupyter Notebook.  \n- Check saved figures in the `assets/fig/` directory.  \n- Review intermediate CSVs in `assets/csv/`.  \n\n\n## Key Challenges  \n\n1. Deciding on project goals and API usage.\n2. Understanding and parsing FEMA's dataset.  \n3. Handling API rate limits and multi-threading with a large dataset.  \n4. Adapting workflow to handle large datasets efficiently.  \n5. Creating efficient workflow that allowed everyone to contribute at the same time\n\n\n## Directory Structures  \n\n### Original Cloned Repo  \n\n```  \nHavocMapper/\n|-- docs/\n|-- havoc_ETL.ipynb\n|-- havoc_analysis.ipynb\n|-- requirements.txt\n```  \n\n### New Expected Structure  \n\n```  \nHavocMapper/\n|-- assets/\n|   |-- csv/\n|   |   |-- raw/\n|   |   |-- clean/\n|   |-- fig/\n```  \n\n\n## Deliverables  \n\n### Visualizations  \n\n- Line plots, scatter plots, and bar charts.  \n- Choropleth maps with Holoviews and Bokeh.  \n\n### Analyses  \n\n- FEMA disaster trends by state and time.  \n- Severe weather/storm trends by years.  \n- Cycle times and distributions of responses.  \n- Geographic distribution of disaster events by type and disaster itself.\n\n### Saved Figures and Models  \n\n- Figures saved in the `assets/fig/` directory, organized by contributor.  \n\n### Google Slides Presentation  \n\n- Overview of findings, shared in PDF format in `docs/PDFs/`.  \n\n\n## Contributors  \n\n- **Alex**: Disaster frequencies and trends.  \n- **Avenika**: Storm trends and summaries.  \n- **Claudia**: Disaster cycles and state responses.  \n- **Neel**: GitHub management, API integration, README creation.  \n\n\n## Data Sources and Citations  \n\n- [FEMA Disaster Declarations Summary Dataset](https://www.fema.gov/openfema-data-page/disaster-declarations-summaries-v2)  \n- [Geoapify API](https://www.geoapify.com/)  \n\n\n## Long Form Directory Structure\n\nStructure of GitHub Repo:\n```\nHavocMapper/\n|\n|-- docs/\n|   |-- Expected Assets/\n|   |   |-- csv/\n|   |   |   |-- clean/\n|   |   |   |   |-- distilled_summary.csv\n|   |   |   |   |-- femaDisasters_geocode.csv\n|   |   |   |-- raw/\n|   |   |   |   |-- DisastersDeclarationsSummaries.csv\n|   |   |   |   |-- femaDisasters_rough.csv\n|   |   |   |   |-- location_geocode.csv\n|   |   |-- fig/\n|   |   |   |-- alex/\n|   |   |   |   |-- MostCommonDisasters_byTopStates.png\n|   |   |   |   |-- Top10_MostFrequent_Disasters.png\n|   |   |   |   |-- Top10_States_MostDisasters.png\n|   |   |   |-- avenika/\n|   |   |   |   |-- FloodTrend_byYear.png\n|   |   |   |   |-- HurricaneTrend_byYear.png\n|   |   |   |   |-- SevereStormTrend_byYear.png\n|   |   |   |-- claudia/\n|   |   |   |   |-- AvgCycleTime_byType.png\n|   |   |   |   |-- CycleTimeDistr_byType.png\n|   |   |   |   |-- DisasterCount_byState.png\n|   |   |   |   |-- DisasterNumber_overTime.png\n|   |   |   |   |-- DisasterNumbers_byYear.png\n|   |   |   |   |-- TimeForDeclaration_byState.png\n|   |   |   |   |-- Year_over_year_disasters.png\n|   |   |   |-- neel/\n|   |   |   |   |-- cleaned_df.png\n|   |   |   |   |-- original_df.png\n|   |   |   |   |-- summary_df.png\n|   |-- Original Notebooks/\n|   |   |-- Alex_Notebook.ipynb\n|   |   |-- Avenika_Notebook.ipynb\n|   |   |-- Claudia_Notebook.ipynb\n|   |   |-- Neel_Notebook.ipynb\n|   |-- PDFs/\n|   |   |-- disaster_declaration_process.pdf\n|   |   |-- FEMA Weather Impact Analysis.pdf\n|   |   |-- InternationalCountryCodes.pdf\n|   |-- fact_sheet.md\n|- .gitignore\n|- DisastersDeclarationsSummaries.csv\n|- havoc_ETL.ipynb\n|- havoc_analysis.ipynb\n|- README.md\n|- requirements.txt\n```\n\nStructure of Intended Output Directory with new assets directory - should mimic the Expected Assets directory:\n```\nHavocMapper/\n|\n|-- assets/\n|   |-- csv/\n|   |   |-- clean/\n|   |   |   |-- distilled_summary.csv\n|   |   |   |-- femaDisasters_geocode.csv\n|   |   |-- raw/\n|   |   |   |-- DisastersDeclarationsSummaries.csv\n|   |   |   |-- femaDisasters_rough.csv\n|   |   |   |-- location_geocode.csv\n|   |-- fig/\n|   |   |-- alex/\n|   |   |   |-- MostCommonDisasters_byTopStates.png\n|   |   |   |-- Top10_MostFrequent_Disasters.png\n|   |   |   |-- Top10_States_MostDisasters.png\n|   |   |-- avenika/\n|   |   |   |-- FloodTrend_byYear.png\n|   |   |   |-- HurricaneTrend_byYear.png\n|   |   |   |-- SevereStormTrend_byYear.png\n|   |   |-- claudia/\n|   |   |   |-- AvgCycleTime_byType.png\n|   |   |   |-- CycleTimeDistr_byType.png\n|   |   |   |-- DisasterCount_byState.png\n|   |   |   |-- DisasterNumber_overTime.png\n|   |   |   |-- DisasterNumbers_byYear.png\n|   |   |   |-- TimeForDeclaration_byState.png\n|   |   |   |-- Year_over_year_disasters.png\n|   |   |-- neel/\n|   |   |   |-- cleaned_df.png\n|   |   |   |-- original_df.png\n|   |   |   |-- summary_df.png\n|-- docs/\n|   |-- Expected Assets/\n|   |   |-- csv/\n|   |   |   |-- clean/\n|   |   |   |   |-- distilled_summary.csv\n|   |   |   |   |-- femaDisasters_geocode.csv\n|   |   |   |-- raw/\n|   |   |   |   |-- DisastersDeclarationsSummaries.csv\n|   |   |   |   |-- femaDisasters_rough.csv\n|   |   |   |   |-- location_geocode.csv\n|   |   |-- fig/\n|   |   |   |-- alex/\n|   |   |   |   |-- MostCommonDisasters_byTopStates.png\n|   |   |   |   |-- Top10_MostFrequent_Disasters.png\n|   |   |   |   |-- Top10_States_MostDisasters.png\n|   |   |   |-- avenika/\n|   |   |   |   |-- FloodTrend_byYear.png\n|   |   |   |   |-- HurricaneTrend_byYear.png\n|   |   |   |   |-- SevereStormTrend_byYear.png\n|   |   |   |-- claudia/\n|   |   |   |   |-- AvgCycleTime_byType.png\n|   |   |   |   |-- CycleTimeDistr_byType.png\n|   |   |   |   |-- DisasterCount_byState.png\n|   |   |   |   |-- DisasterNumber_overTime.png\n|   |   |   |   |-- DisasterNumbers_byYear.png\n|   |   |   |   |-- TimeForDeclaration_byState.png\n|   |   |   |   |-- Year_over_year_disasters.png\n|   |   |   |-- neel/\n|   |   |   |   |-- cleaned_df.png\n|   |   |   |   |-- original_df.png\n|   |   |   |   |-- summary_df.png\n|   |-- Original Notebooks/\n|   |   |-- Alex_Notebook.ipynb\n|   |   |-- Avenika_Notebook.ipynb\n|   |   |-- Claudia_Notebook.ipynb\n|   |   |-- Neel_Notebook.ipynb\n|   |-- PDFs/\n|   |   |-- disaster_declaration_process.pdf\n|   |   |-- FEMA Weather Impact Analysis.pdf\n|   |   |-- InternationalCountryCodes.pdf\n|   |-- fact_sheet.md\n|- .gitignore\n|- DisastersDeclarationsSummaries.csv\n|- havoc_ETL.ipynb\n|- havoc_analysis.ipynb\n|- README.md\n|- requirements.txt\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneelka96%2Fhavocmapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fneelka96%2Fhavocmapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fneelka96%2Fhavocmapper/lists"}