https://github.com/rmkenv/climate-disaster-data-pipeline
Automated pipeline for synchronizing NOAA Billion-Dollar Disasters data via NWS and FEMA APIs.
https://github.com/rmkenv/climate-disaster-data-pipeline
climate-change data-engineering disaster-management fema-data noaa-api
Last synced: 4 months ago
JSON representation
Automated pipeline for synchronizing NOAA Billion-Dollar Disasters data via NWS and FEMA APIs.
- Host: GitHub
- URL: https://github.com/rmkenv/climate-disaster-data-pipeline
- Owner: rmkenv
- License: gpl-3.0
- Created: 2025-05-08T20:40:56.000Z (about 1 year ago)
- Default Branch: main
- Last Pushed: 2026-01-27T00:08:26.000Z (5 months ago)
- Last Synced: 2026-01-27T11:36:56.417Z (5 months ago)
- Topics: climate-change, data-engineering, disaster-management, fema-data, noaa-api
- Language: Python
- Size: 25.4 KB
- Stars: 4
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
This is a high-end, professionally structured `README.md` designed to impress recruiters by highlighting technical proficiency, domain impact, and engineering best practices.
***
# 🌪️ disaster-data-replicator
[](https://www.python.org/)
[](https://opensource.org/licenses/MIT)
[](https://www.noaa.gov/)
[](https://www.fema.gov/)
[](http://makeapullrequest.com)
> **Bridging the gap between climate events and actionable insights through automated, high-fidelity data engineering.**
`disaster-data-replicator` is a production-grade ETL pipeline designed to synchronize and normalize "Billion-Dollar Disaster" data. By orchestrating data from the National Weather Service (NWS) and FEMA APIs, this tool provides a unified view of climate-driven economic impacts, enabling researchers and developers to analyze disaster trends with precision.
---
## 💡 Key Features
- **🔄 Multi-Source Synchronization:** Seamlessly merges NOAA Billion-Dollar Disaster datasets with real-time FEMA disaster declarations.
- **🛠 Automated ETL Pipeline:** Handles data extraction, schema normalization, and validation without manual intervention.
- **📉 Intelligent Rate Limiting:** Built-in backoff algorithms and request throttling to respect NWS and FEMA API constraints.
- **🧪 Data Integrity:** Uses Pydantic for strict schema enforcement, ensuring that inconsistent API responses don't break downstream analytics.
- **📂 Flexible Export:** Supports high-performance output formats including Parquet (for Big Data), CSV, and JSON.
---
## 🛠 Tech Stack
| Category | Tools |
| :--- | :--- |
| **Language** |  |
| **Data Handling** |   |
| **APIs** | `noaa-api`, `fema-data-api`, `requests` |
| **DevOps** |   |
| **Domain** | Climate Change, Disaster Management, Data Engineering |
---
## 🏁 Quick Start
### Prerequisites
- Python 3.9+
- API keys for NOAA/FEMA (if required for higher rate limits)
### Installation
1. **Clone the repository:**
```bash
git clone https://github.com/yourusername/disaster-data-replicator.git
cd disaster-data-replicator
```
2. **Set up a virtual environment:**
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. **Install dependencies:**
```bash
pip install -r requirements.txt
```
### Usage
**Run the full replication pipeline:**
```bash
python main.py --start-year 2000 --output-format parquet
```
**Sync specific agency data:**
```bash
# Sync only FEMA disaster declarations
python scripts/sync_fema.py --region "FL"
```
---
## 📊 Project Architecture
```mermaid
graph LR
A[NOAA NWS API] --> E[Ingestion Engine]
B[FEMA API] --> E
E --> F{Normalization}
F --> G[Validation/Pydantic]
G --> H[(Local Storage / S3)]
H --> I[Analytics & Visualization]
```
---
## 🤝 How to Contribute
Contributions make the open-source community an amazing place to learn, inspire, and create.
1. **Fork** the Project
2. Create your **Feature Branch** (`git checkout -b feature/AmazingFeature`)
3. **Commit** your Changes (`git commit -m 'Add some AmazingFeature'`)
4. **Push** to the Branch (`git push origin feature/AmazingFeature`)
5. Open a **Pull Request**
---
## 📄 License
Distributed under the MIT License. See `LICENSE` for more information.
---
## 📧 Contact
**Your Name** - [@YourTwitter](https://twitter.com/YourHandle) - your.email@example.com
*Project Link: [https://github.com/yourusername/disaster-data-replicator](https://github.com/yourusername/disaster-data-replicator)*
---
*This project was developed to provide transparent access to climate-related economic data, supporting the global effort to understand and mitigate the impacts of climate change.*