{"id":27326513,"url":"https://github.com/ryanlevee/nexrad-mapbox-backend","last_synced_at":"2026-04-09T11:34:22.283Z","repository":{"id":287140004,"uuid":"950412933","full_name":"ryanlevee/nexrad-mapbox-backend","owner":"ryanlevee","description":"Python and Flask backend services for the NEXRAD Mapbox Viewer","archived":false,"fork":false,"pushed_at":"2025-04-10T06:10:36.000Z","size":157,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-04-10T07:22:01.185Z","etag":null,"topics":["api","async","aws","aws-s3","backend","boto3","concurrency","flask","matplotlib","multiprocessing","nexrad","noaa","numpy","pyart","python","pytz","rest-api","s3","unidata","waitress"],"latest_commit_sha":null,"homepage":"https://nexradmapbox.netlify.app/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ryanlevee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-18T05:52:45.000Z","updated_at":"2025-04-10T06:10:39.000Z","dependencies_parsed_at":"2025-04-10T07:22:05.302Z","dependency_job_id":"470a6161-812b-4dd1-846a-faa55eb0deb8","html_url":"https://github.com/ryanlevee/nexrad-mapbox-backend","commit_stats":null,"previous_names":["ryanlevee/nexrad-mapbox-backend"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanlevee%2Fnexrad-mapbox-backend","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanlevee%2Fnexrad-mapbox-backend/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanlevee%2Fnexrad-mapbox-backend/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ryanlevee%2Fnexrad-mapbox-backend/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ryanlevee","download_url":"https://codeload.github.com/ryanlevee/nexrad-mapbox-backend/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248563722,"owners_count":21125339,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["api","async","aws","aws-s3","backend","boto3","concurrency","flask","matplotlib","multiprocessing","nexrad","noaa","numpy","pyart","python","pytz","rest-api","s3","unidata","waitress"],"created_at":"2025-04-12T11:42:27.321Z","updated_at":"2025-12-30T23:05:06.607Z","avatar_url":"https://github.com/ryanlevee.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NEXRAD Weather Radar Viewer with Mapbox - Backend\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n**Live Demo:** [**NEXRAD Mapbox on Netlify**](https://nexradmapbox.netlify.app/)\n###### **NOTE**: Current demo displays data from March 21, 2025 for the KPDT site only. Real-time data functionality is available, but turned off for the moment due to cost. \n\n**Frontend Repository:** [**NEXRAD Mapbox frontend repository**](https://github.com/ryanlevee/nexrad-mapbox)\n\n\n## Overview\n\nThis repository contains the backend services for the NEXRAD Mapbox Viewer. It consists of two main parts:\n\n1.  **Data Processing Scripts:** Python scripts that fetch raw NEXRAD Level 2 and Level 3 data from public AWS S3 buckets (NOAA and Unidata), process them using Py-ART and Matplotlib to generate PNG image overlays and JSON metadata, and upload these processed files to a dedicated project S3 bucket.\n2.  **Flask API:** A Python Flask web server that serves the processed data (images, metadata, file lists) from the project S3 bucket to the frontend application via a REST API.\n\n\n## Architecture Overview\n\nThe backend utilizes a standard Python `src` layout for organization:\n\n* **`src/nexrad_backend/`**: The main Python package containing all core logic.\n    * **`api/`**: Handles the Flask web application setup (`app_factory.py`) and defines the HTTP API endpoints/routes (`routes.py`) using a Flask Blueprint.\n    * **`services/`**: Contains modules that interact with external systems or manage specific data domains:\n        * `s3_service.py`: Low-level functions for all interactions with AWS S3 (get, put, list, delete, check existence). Handles both project and public bucket clients.\n        * `nexrad_fetcher.py`: Functions responsible for finding and downloading raw NEXRAD data (Level 2 from NOAA, Level 3 from Unidata) from their public S3 buckets.\n        * `metadata_service.py`: Functions for managing the application's metadata stored in the *project's* S3 bucket (JSON file lists, product code options, update flags).\n    * **`processing/`**: Contains the core scientific data processing logic:\n        * `common.py`: Shared helper functions used by different processing steps (e.g., bounding box calculation, saving plots to buffer, file cleanup).\n        * `level2.py`: Logic specific to reading Level 2 files, processing each radar sweep (tilt), generating plots/metadata using Py-ART/Matplotlib, and preparing results for S3 upload.\n        * `level3.py`: Logic specific to reading Level 3 files, normalizing filenames, generating plots/metadata, and preparing results for S3 upload.\n    * **`utils/`**: General utility functions (e.g., `list_helpers.py`).\n    * **`config.py`**: Centralized configuration loading from environment variables (`.env`) and definition of constants (bucket names, paths, sites, etc.).\n* **`scripts/`**: Contains standalone Python scripts that act as entry points for the data processing workflows. These scripts import and orchestrate calls to functions within the `nexrad_backend` services and processing modules.\n    * `process_level2.py`: Runs the complete workflow for fetching, processing, and updating Level 2 data.\n    * `process_level3.py`: Runs the complete workflow for fetching, processing, and updating Level 3 data.\n* **`server.py`**: The main entry point at the project root for starting the Flask API server using Waitress. It uses the app factory from `nexrad_backend.api.app_factory`.\n* **`pyproject.toml`**: Defines project metadata, dependencies, and build system configuration, enabling standard Python packaging and installation workflows.\n\n**Data Processing Flow (Triggered Periodically):**\n\n```ascii\n+-------------------------------+\n| Public S3 (NOAA L2/Unidata L3)|  Raw Data Source\n+-------------┬-----------------+\n              │ Finds/Downloads\n              │ [nexrad_fetcher service]\n              ▼\n+-------------------------------+\n| scripts/*.py                  |  Orchestration (Scheduled)\n| (Downloads to Local Temp)     |\n+-------------┬-----------------+\n              │ Processes Files\n              │ [processing.L2/L3 modules] using PyArt/Matplotlib\n              ▼\n+-------------------------------+\n| s3_service                    |  Uploads Processed Data \u0026 Metadata\n| (Handles S3 Put/Delete)       |  (Also called by metadata_service)\n+-------------┬-----------------+\n              │ Stores Processed Files \u0026 Updates Metadata Files\n              ▼\n+-------------------------------+\n| Project S3 (nexrad-mapbox)    |  Central Storage\n|  - plots_level*/ (PNG/JSON)   |\n|  - lists/*.json               |\n|  - codes/options.json         |\n|  - flags/update_flags.json    |\n+-------------------------------+\n```\n\n**API Serving Flow (Handles Frontend Requests):**\n```ascii\n+-----------------------------+\n| Project S3 (nexrad-mapbox)  |  Central Storage\n+-------------┬---------------+\n              │ Reads Data \u0026 Metadata Files\n              │ (via metadata_service \u0026 s3_service)\n              ▼\n+-----------------------------+\n| Flask API (server.py)       |  Serves Data via HTTP Endpoints\n+-------------┬---------------+\n              │ HTTP Requests / JSON \u0026 PNG Responses\n              ▼\n+-----------------------------+\n| Frontend (SolidJS App)      |  Consumes API\n+-----------------------------+\n```\n\n**Explanation of Flow:**\n\n1.  **Processing:** The `scripts/process_level*.py` files are run on a schedule. They use the `nexrad_fetcher` service to find and download recent raw data from public S3 buckets to temporary local storage. These scripts then use the `processing` modules (which utilize Py-ART/Matplotlib) to generate PNG plots and JSON metadata. The `s3_service` is used to upload these processed files to your project's S3 bucket. The `metadata_service` (also using `s3_service`) updates the JSON file lists, code counts, and update flags within the project S3 bucket. Finally, `s3_service` cleans up old processed files from your bucket.\n2.  **Serving:** The `Flask API` (running via `server.py`) receives HTTP requests from the Frontend. It uses the `metadata_service` to get file lists/codes/flags (which reads JSON files from the Project S3 via `s3_service`) and uses the `s3_service` directly to retrieve specific plot (PNG) or metadata (JSON) files requested by the frontend, sending them back as HTTP responses.\n\n\n## Features\n\n* **Automated NEXRAD Data Processing:**\n    * Fetches Level 2 (Reflectivity) and Level 3 (Hydrometeor, Precipitation) data.\n    * Uses `Py-ART` for scientific radar data interpretation.\n    * Generates transparent PNG plot overlays using `Matplotlib`.\n    * Calculates and saves geographic bounding box JSON metadata for each plot.\n    * Handles multi-sweep Level 2 data, generating plots/JSON per tilt angle.\n    * Utilizes `asyncio` and `ProcessPoolExecutor` for concurrent downloading and processing.\n* **REST API for Frontend:**\n    * Serves processed PNG images and JSON metadata.\n    * Provides lists of available files for different products/levels.\n    * Provides lists of available Level 3 product codes and their file counts.\n    * Handles CORS for frontend integration.\n* **AWS S3 Integration:**\n    * Leverages public S3 buckets for raw data input.\n    * Uses a dedicated S3 bucket for storing processed output (plots, JSON, lists).\n    * Includes utility for cleaning up old files from the S3 bucket.\n* **Configuration:** Uses environment variables for AWS credentials and other settings.\n* **Modular Structure:** Code is organized into services, processing modules, and API components for better maintainability and testability using a standard `src` layout.\n\n\n## Technology Stack\n\n* **Language:** Python (3.9+ recommended)\n* **API Framework:** Flask\n* **WSGI Server:** Waitress\n* **AWS SDK:** Boto3\n* **Scientific Computing:** Py-ART, NumPy\n* **Plotting:** Matplotlib\n* **Asynchronous Programming:** asyncio, concurrent.futures\n* **Environment Management:** python-dotenv\n* **Cloud Storage:** AWS S3\n* **Timezone Handling:** pytz\n* **Packaging:** Setuptools (via `pyproject.toml`)\n\n\n## API Endpoints (`src/nexrad_backend/api/routes.py`)\n\n| Method    | Path                                   | Description                                                                                                | Success Response | Error Responses |\n| :-------- | :------------------------------------- | :--------------------------------------------------------------------------------------------------------- | :--------------- | :-------------- |\n| `GET`     | `/code/`                               | Retrieves the `codes/options.json` file containing Level 3 product code options and counts.                | 200 (JSON Body)  | 404, 500        |\n| `GET`     | `/flag/`                               | Retrieves the `flags/update_flags.json` file.                                                              | 200 (JSON Body)  | 404, 500        |\n| `POST`    | `/flag/`                               | Updates the `flags/update_flags.json` file with the provided JSON body.                                    | 200 (JSON Body)  | 400, 500        |\n| `GET`     | `/list/\u003clevel\u003e/\u003cproduct\u003e/`             | Retrieves the JSON file list for a specific level and product (e.g., `/list/2/reflectivity/`).             | 200 (JSON Body)  | 400, 404, 500   |\n| `GET`     | `/list-all/`                           | Retrieves and combines JSON file lists for all primary products (reflectivity, hydrometeor, precipitation).| 200 (JSON Body)  | 500             |\n| `GET`     | `/data/\u003clevel\u003e/\u003cpath:file_key\u003e/\u003cext\u003e`  | Retrieves a specific data file (e.g., `/data/2/..._idx0/png`). `ext` is `png` or `json`.                   | 200 (PNG/JSON)   | 400, 404, 500   |\n| `OPTIONS` | `/*`                                   | Handles CORS preflight requests (managed automatically by Flask-CORS).                                     | 200              | -               |\n\n\n## Data Processing Scripts\n\nThe data processing logic is initiated via scripts in the `/scripts` directory:\n\n* **`scripts/process_level2.py`:** Orchestrates the workflow for Level 2 data:\n    * Finds recent raw files on the public NOAA S3 bucket using the `nexrad_fetcher` service.\n    * Filters out files already processed based on lists managed by the `metadata_service`.\n    * Downloads new files concurrently using `nexrad_fetcher`.\n    * Processes downloaded files concurrently using the `processing.level2` module, which generates/uploads individual sweep plots (PNG) and metadata (JSON) to the project S3 via the `s3_service`.\n    * Updates the Level 2 file list and processing flag in the project S3 via `metadata_service`.\n    * Cleans up old processed L2 files from the project S3 via `s3_service`.\n* **`scripts/process_level3.py`:** Orchestrates the workflow for Level 3 data (per product type like 'hydrometeor', 'precipitation'):\n    * Fetches product code configuration using `metadata_service`.\n    * Finds recent raw files on the public Unidata S3 bucket for relevant codes using `nexrad_fetcher`.\n    * Filters out files already processed (using normalized keys).\n    * Downloads new files concurrently.\n    * Processes downloaded files concurrently using the `processing.level3` module, which generates/uploads plots (PNG) and metadata (JSON) to the project S3.\n    * Updates the Level 3 file list, product code counts (`options.json`), and processing flag for the specific product type via `metadata_service`.\n    * Cleans up old processed L3 files from the project S3 via `s3_service`.\n\nThese scripts are designed to be run periodically (e.g., every 5-15 minutes) using a scheduler like `cron` or a cloud scheduling service to keep the data served by the API up-to-date.\n\n\n## Setup \u0026 Configuration\n\n1.  **Clone the repository:**\n    ```bash\n    git clone [https://github.com/your-username/nexrad-mapbox-backend.git](https://github.com/your-username/nexrad-mapbox-backend.git)\n    cd nexrad-mapbox-backend\n    ```\n\n2.  **Create a Python Virtual Environment:** (Python 3.9+ recommended)\n    ```bash\n    # Using venv\n    python -m venv venv\n    source venv/bin/activate  # Linux/macOS\n    # or Venv\\Scripts\\activate # Windows (cmd)\n    # or Venv\\Scripts\\Activate.ps1 # Windows (PowerShell)\n\n    # OR Using Conda (Recommended for easier handling of pygrib/eccodes on Windows)\n    # conda create --name nexrad_env python=3.9\n    # conda activate nexrad_env\n    # conda install -c conda-forge eccodes pygrib pyart-mch matplotlib numpy pytz # Install key deps via conda\n    ```\n\n3.  **Install Dependencies:** The project uses `pyproject.toml`. Install the package itself in editable mode along with development dependencies (like Ruff linter/formatter):\n    ```bash\n    # Make sure you are in the project root (nexrad-mapbox-backend/)\n    # Ensure pip is up-to-date: python -m pip install --upgrade pip\n\n    # If using Venv:\n    pip install -e .[dev]\n\n    # If using Conda (after conda install step above):\n    # Install remaining deps (Flask, Boto3, etc.) AND the local package\n    pip install flask flask-cors boto3 python-dotenv waitress # Add any others not installed via conda\n    pip install -e .[dev] # Installs local package and dev tools from pyproject.toml\n    ```\n    *(The `-e` flag installs your `nexrad_backend` package in editable mode, linking to your `src` directory. The `[dev]` installs optional dependencies listed under `project.optional-dependencies.dev` in `pyproject.toml`)*.\n\n4.  **Set up Environment Variables:**\n    * Create a `.env` file in the project root (`nexrad-mapbox-backend/.env`).\n    * Add your AWS credentials and desired region. Ensure these credentials have S3 read/write/delete permissions for your target bucket (`PROJECT_S3_BUCKET` defined in config, defaults to `nexrad-mapbox`).\n        ```dotenv\n        # nexrad-mapbox-backend/.env\n\n        AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY_ID\n        AWS_SECRET_ACCESS_KEY=YOUR_AWS_SECRET_ACCESS_KEY\n        AWS_REGION=us-west-1 # Or your preferred region\n\n        # Optional: Override defaults from config.py if needed\n        # PROJECT_S3_BUCKET=my-custom-bucket-name\n        # RADAR_SITE_L2=KSEA\n        # RADAR_SITE_L3=SEA\n        # PROCESSING_WINDOW_MINUTES=120\n        # API_PORT=5000\n        ```\n    * **Security:** Add `.env` to your `.gitignore` file to avoid committing credentials.\n\n5.  **AWS S3 Bucket Setup:**\n    * Ensure the target S3 bucket (e.g., `nexrad-mapbox`) exists in your specified AWS region.\n    * The processing scripts create necessary prefixes (`plots_level2/`, `lists/`, etc.) if they don't exist.\n\n\n## Running the Application\n\n1.  **Run the API Server:**\n    * Activate your virtual environment (`source venv/bin/activate` or `conda activate nexrad_env`).\n    * Run from the project root:\n        ```bash\n        python server.py\n        ```\n    * The API will be available at `http://\u003cAPI_HOST\u003e:\u003cAPI_PORT\u003e` (defaults to `http://0.0.0.0:4000`). Access via `http://localhost:4000` or `http://127.0.0.1:4000` from your local machine.\n\n2.  **Run the Data Processing Scripts:**\n    * Activate your virtual environment.\n    * Run manually from the project root:\n        ```bash\n        python scripts/process_level2.py\n        python scripts/process_level3.py\n        ```\n    * **Note:** These scripts should ideally be run periodically via a scheduler (`cron`, Task Scheduler, cloud service) rather than manually long-term.\n\n\n## Deployment\n\n* **API Server:** The API (`server.py` using Waitress) can be deployed to various PaaS/server environments (Render, Heroku, AWS EC2/ECS, etc.).\n    * Ensure the deployment environment has Python and access to the necessary environment variables (AWS credentials, etc.).\n    * The build process should install dependencies via `pip install .` (reading `pyproject.toml`) or `pip install -r requirements.txt` (if generated from `pyproject.toml` for pinning).\n    * **Render.com Start Command Example:** `python server.py`\n* **Processing Scripts:** The scripts (`scripts/process_level*.py`) need to be executed on a schedule. Suitable options include:\n    * `cron` jobs on a Linux server.\n    * Scheduled Tasks on Windows Server.\n    * Cloud-based schedulers triggering container tasks or serverless functions (e.g., AWS EventBridge Scheduler + Lambda/Fargate, Google Cloud Scheduler + Cloud Run/Functions, Render Cron Jobs).\n\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](https://github.com/ryanlevee/nexrad-mapbox-backend/blob/main/LICENSE) file for details.\n\n\n## Contact\n\nRyan Levee - [GitHub](https://github.com/ryanlevee) | [LinkedIn](https://www.linkedin.com/in/ryanlevee/) | [Email](mailto:ryanlevee@gmail.com)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanlevee%2Fnexrad-mapbox-backend","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fryanlevee%2Fnexrad-mapbox-backend","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fryanlevee%2Fnexrad-mapbox-backend/lists"}