{"id":47892543,"url":"https://github.com/cipher982/floodmap","last_synced_at":"2026-04-04T03:10:52.294Z","repository":{"id":260183883,"uuid":"869193828","full_name":"cipher982/floodmap","owner":"cipher982","description":"check your elevation and location in flood maps for storms","archived":false,"fork":false,"pushed_at":"2025-12-13T00:50:39.000Z","size":6571,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-14T07:05:19.555Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cipher982.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2024-10-07T21:56:04.000Z","updated_at":"2025-12-13T00:50:42.000Z","dependencies_parsed_at":"2025-09-09T02:30:38.538Z","dependency_job_id":"89a316df-9910-4bba-96bd-56f186bd7236","html_url":"https://github.com/cipher982/floodmap","commit_stats":null,"previous_names":["cipher982/floodmap"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cipher982/floodmap","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cipher982%2Ffloodmap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cipher982%2Ffloodmap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cipher982%2Ffloodmap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cipher982%2Ffloodmap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cipher982","download_url":"https://codeload.github.com/cipher982/floodmap/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cipher982%2Ffloodmap/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31386001,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T01:22:39.193Z","status":"online","status_checked_at":"2026-04-04T02:00:07.569Z","response_time":60,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T03:10:51.653Z","updated_at":"2026-04-04T03:10:52.287Z","avatar_url":"https://github.com/cipher982.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# FloodMap 🗺️🌊\n\nFloodMap is **an end-to-end, high-performance flood-risk mapping platform** covering the entire United States.\nIt combines a FastAPI micro-service, an optimized elevation–processing pipeline and a zero-dependency HTML/JS viewer to deliver near-real-time flood visualisations down to 256 × 256 px map tiles.\n\n\n[floodmap.drose.io](https://floodmap.drose.io)\n\n---\n\n## ✨ Key Capabilities\n\n- **Nation-wide coverage** – SRTM 1-arc-second DEM is tiled, compressed (Zstandard) and cached for millisecond look-ups.\n- **Water-level visualisation** – Generate colour-mapped PNG tiles for *any* requested water-level between **–10 m** and **+1 000 m** (configurable).\n- **Topographical mode** – Switch to an absolute-elevation colour scale for terrain inspection.\n- **Vector overlays** – Optional road / city / boundary tiles are served from MBTiles (OpenStreetMap extract).\n- **Predictive pre-loading \u0026 persistent caches** greatly reduce cold-start latency.\n- **Observability built-in** – Prometheus metrics, structured logging and configurable OTLP traces.\n\n\u003cp align=\"center\"\u003e\u003cimg src=\"docs/figures/demo.gif\" width=\"700\" alt=\"Demo animation\"\u003e\u003c/p\u003e\n\n---\n\n## 🏗️ Project Layout\n\n```\n├── src/                      # Python source code\n│   ├── api/                  # FastAPI application (floodmap.api)\n│   │   ├── routers/          # Versioned HTTP endpoints\n│   │   ├── services/         # (future) business logic helpers\n│   │   ├── middleware/       # ASGI middleware (rate-limiter, tracing…)\n│   │   ├── color_mapping.py  # NumPy-vectorised colour LUTs\n│   │   ├── elevation_loader.py  # Thread-safe on-demand DEM loader\n│   │   ├── tile_cache.py         # LRU in-memory PNG cache\n│   │   └── …\n│   ├── process_elevation_usa.py  # One-off DEM compression pipeline\n│   ├── process_maps_usa.py       # Vector MBTiles generation helper\n│   └── web/                  # Lightweight static front-end\n│\n├── tests/                    # Pytest suite (unit + integration)\n├── utils/                    # Misc. notebooks \u0026 ad-hoc scripts\n├── docs/                     # Additional background docs \u0026 figures\n└── Dockerfile                # Production image (multi-stage)\n```\n\n---\n\n## ⚡ Quick-start (macOS / Linux)\n\n```bash\n# 1. Clone \u0026 install deps (uses uv – https://github.com/astral-sh/uv)\nuv sync\n\n\n# 2. Start everything\nmake start   # == docker-compose up ‑d tileserver \u0026\u0026 uvicorn src.api.main:app --reload\n\n# 3. Open the viewer\nopen http://localhost:8000   # or your preferred browser\n```\n\nThe default Makefile recipe spins up:\n\n| Service          | Port | Description                                  |\n| ---------------- | ---- | -------------------------------------------- |\n| fastapi          | 8000 | JSON /tiles/* PNG endpoints \u0026 REST API       |\n| static-frontend  | 8000 | Served via FastAPI `StaticFiles`             |\n| tileserver-gl    | 8080 | Optional vector-tile server (Docker)         |\n\n---\n\n## ⚙️ Configuration\n\nAll tunables live in **`src/api/config.py`** and can be overridden via environment variables or a *.env* file.\n\nVariable              | Purpose                                  | Default\n--------------------- | ---------------------------------------- | ------------------------------\n`PROJECT_ROOT`        | Absolute project path                    | repo root\n`COMPRESSED_DATA_DIR` | Directory containing `*.zst` DEM tiles   | `compressed_data/`\n`ELEVATION_CACHE_SIZE`| In-RAM LRU entries (tiles)               | `50`\n`TILE_CACHE_SIZE`     | In-RAM LRU entries (PNG responses)       | `1000`\n`MIN_WATER_LEVEL` / `MAX_WATER_LEVEL` | Allowed request range     | `-10` / `1000`\n\nSee *.env.example* for a full list.\n\n---\n\n## 🐳 Docker Compose\n\n- Dev (ports published): `docker compose up -d`\n- Prod (internal-only): `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d`\n\nDetails:\n- Dev maps to host for easy access (clickable logs). Override file publishes `${API_PORT}:8000` and `${TILESERVER_PORT}:8080`.\n- Prod does not publish any host ports. The API listens on `8000` inside the compose network and the tileserver on `8080`.\n- Network name defaults to `${COMPOSE_PROJECT_NAME}-network` (e.g., `floodmap-network`).\n\nReverse proxy:\n- Run your proxy on the same Docker network and target `webapp:8000`.\n- Example: `docker run -d --network floodmap-network your-proxy-image`\n- Nginx upstream example: `upstream floodmap { server webapp:8000; }`\n\nVerify prod:\n- `docker compose -f docker-compose.yml -f docker-compose.prod.yml config` (no `ports:` on webapp)\n- From a container on the network: `curl http://webapp:8000/api/health`\n\nNote: Only one Dockerfile is used. `Dockerfile.prod` has been removed to avoid confusion.\n\n## 🏞️ Data Pipeline\n\n### 1. Elevation (DEM) Compression\n\nThe raw US SRTM 1-arc-second GeoTIFFs are *huge* (≈45 GB).\n`process_elevation_usa.py` chunks them into 256 × 256 tiles, serialises each tile as *little-endian int16* bytes and compresses with **Zstandard (level 3)**.\nMetadata (bounds, CRS, nodata, etc.) is persisted per-tile in a side-car JSON.\n\n```bash\nuv run python src/process_elevation_usa.py \\\n  --input /mnt/raw_srtm/us/ \\\n  --output compressed_data/usa \\\n  --workers 8                \\\n  --compression-level 5\n```\n\nResults: **≈4 × space saving** and ~25–30 ms decompression per tile on a laptop.\n\n### 2. Vector Tiles\n\n`process_maps_usa.py` ingests the latest OpenStreetMap extract, filters layers (roads, admin boundaries, landuse, places) and writes an **MBTiles** database with tiles up to `z14`.\n\n### 3. Precompressed Elevation Tiles (.u16.br)\n\nPrecompressed raw elevation tiles eliminate server CPU work and reduce bandwidth by sending compact **uint16** arrays to the browser. These are served by the v1 API and consumed by the client renderer.\n\n- Endpoint: `/api/v1/tiles/elevation-data/{z}/{x}/{y}.u16?method=precompressed`\n- Encoding: negotiate `br` (Brotli) or `gzip` via `Accept-Encoding`; server uses sendfile for `.u16.br/.gz`.\n- Runtime fallback: if a precompressed file is missing, the API generates a runtime tile and compresses it on the fly.\n\n#### Generate (Host / Dev)\n\nUse the hardened generator (loud checks; no silent bad writes):\n\n```bash\nuv run python tools/generate_precompressed_elevation_tiles.py \\\n  --source-dir data/elevation-source \\\n  --output-dir data/elevation-tiles \\\n  --zoom-min 8 --zoom-max 11 \\\n  --no-gz         # only .u16.br\n```\n\nTips:\n- Add `--bbox \u003cminLon\u003e \u003cminLat\u003e \u003cmaxLon\u003e \u003cmaxLat\u003e` to constrain coverage (e.g., a metro).\n- Add `--no-skip` to overwrite previously generated tiles.\n\n#### Generate (In Container / Prod)\n\nEnsure the volumes are mounted (see docker-compose.prod.yml) and the output mount is writable during generation.\n\n```bash\ndocker compose run --rm webapp \\\n  python tools/generate_precompressed_elevation_tiles.py \\\n    --output-dir /app/data/elevation-tiles \\\n    --zoom-min 8 --zoom-max 11 \\\n    --no-gz --no-skip\n```\n\n#### Loud Safety Checks (What The Script Enforces)\n\n- Validates `--source-dir` exists and has thousands of `.zst` files; otherwise aborts with a fatal message.\n- Skips writing when the loader returns no data (ocean/outside coverage). These are counted as `tiles_skipped_missing` in the per‑zoom summary and manifest.\n- Writes a manifest at `output_dir/manifest.json` with totals and variants.\n\n#### Validate A Random Sample\n\nFetch a generated tile using precompressed mode and inspect the payload:\n\n```bash\nAPI=http://127.0.0.1:8000\nZ=9; X=140; Y=215\ncurl -sS -D /tmp/h.txt -o /tmp/tile.br -H 'Accept-Encoding: br' \\\n  \"$API/api/v1/tiles/elevation-data/$Z/$X/$Y.u16?method=precompressed\"\nbrotli -d -f /tmp/tile.br -o /tmp/tile.dec\npython3 - \u003c\u003c'PY'\nfrom array import array\nb=open('/tmp/tile.dec','rb').read(); a=array('H'); a.frombytes(b)\nmn=min(a); mx=max(a); nod=sum(1 for v in a if v==65535); tot=len(a)\nprint('bytes',len(b),'min',mn,'max',mx,'nodata%',round(nod*100.0/tot,2))\nPY\n```\n\nHeuristics:\n- Land tiles should have nodata% near 0 and realistic min/max; precompressed `.br` typically tens of KB.\n- Tiny `.br` (~14 bytes) indicates an all‑65535 tile (pure ocean is OK; large swaths of land are not). Re‑generate if unexpected.\n\n#### Smart Coverage Validation Script\n\nUse the helper to summarise counts/sizes and decode a random sample per zoom.\n\n```bash\n# Local file validation\npython tools/validate_precompressed_tiles.py \\\n  --tiles-dir data/elevation-tiles \\\n  --zooms 8 9 10 11 \\\n  --samples 100\n\n# API validation (fetches via precompressed endpoint)\npython tools/validate_precompressed_tiles.py \\\n  --api http://127.0.0.1:8000 \\\n  --zooms 9 10 11 \\\n  --samples 50\n```\n\nOutput includes per-zoom size buckets and nodata% percentiles for quick sanity.\n\n#### Size Planning (USA Coverage)\n\n- z0–11: ~5–6 GB total (Brotli Q10). Manageable on a laptop.\n- z12 adds ~15–16 GB; z13 adds ~60+ GB; prefer regional bboxes for z12+.\n\n---\n\n## 🚀 Runtime Architecture\n\n1. **HTTP request** `GET /tiles/{water}/{z}/{x}/{y}.png` arrives.\n2. ASGI **rate-limiter** \u0026 **tracing** middleware run.\n3. `tile_cache` is queried (key = water+z+x+y).\n   • **HIT ➜** return cached 256 × 256 PNG immediately.\n   • **MISS ➜** proceed.\n4. `elevation_loader` fetches \u0026 mosaics the required DEM bytes:\n   – Persistent on-disk Zstd *Context* avoids re-allocations.\n   – A tiny `diskcache` layer keeps LRU decompressed arrays on SSD.\n5. `color_mapper` converts the `int16` NumPy array ➜ RGBA via pre-computed LUTs (vectorised, no Python loop).\n6. A *ThreadPoolExecutor* encodes the array to PNG (Pillow), level 1.\n7. Response is cached and streamed back (~50–90 ms cold-path).\n\n![Sequence diagram](docs/figures/sequence.png)\n\n---\n\n## 🔌 Public API\n\n### v1 Tiles\n\nEndpoint                                                   | Description\n---------------------------------------------------------- | ---------------------------------------------\n`/api/v1/tiles/elevation-data/{z}/{x}/{y}.u16`             | Raw uint16 elevation tile (runtime by default)\n`/api/v1/tiles/elevation-data/{z}/{x}/{y}.u16?method=precompressed` | Serve precompressed `.u16.br`/`.gz` if present\n`/api/v1/tiles/vector/usa/{z}/{x}/{y}.pbf`                 | Vector tiles proxy (tileserver-gl)\n`/api/v1/tiles/health`                                     | Health/metadata for tile endpoints\n\n### Risk assessment\n\n`POST /risk/location` → JSON body `{latitude, longitude}` returns:\n\n```jsonc\n{\n  \"elevation_m\": 2.3,\n  \"flood_risk_level\": \"high\",   // very_high | high | moderate | low | estimated | unknown\n  \"risk_description\": \"High flood risk - elevation 2.3m is near flood-prone areas\",\n  \"water_level_m\": 1.0\n}\n```\n\n---\n\n## 🧪 Testing\n\n```bash\nuv run pytest -q         # fast unit tests (\u003c 1 s each)\nuv run pytest -m slow    # include integration \u0026 e2e tests (Playwright)\n```\n\nCI configuration lives in *.github/workflows/*. Pull-requests run the full matrix (unit, lint, mypy, e2e-headless).\n\n---\n\n## 📝 Development Tips\n\n• **Debug helpers** – A set of `debug_*.py` scripts lives at repo root for quick profiling and visual checks.\n• **Hot reload** – `uvicorn src.api.main:app --reload` watches for changes.\n• **Data outside repo** – Large raw datasets are symlinked via `ARCHIVED_DATA → /Volumes/Storage/floodmap-archive`.\n• **Gotchas** – When generating precompressed tiles on host, always pass `--source-dir data/elevation-source` and `--output-dir data/elevation-tiles`. Avoid using container‑only paths (`/app/...`) outside Docker — this will yield empty data and tiny `.br` files.\n\n---\n\n## 📋 Roadmap\n\n- [ ] Switch PNG encoding to **pypng + streaming** saver to cut memory allocation.\n- [ ] WebGL front-end with dynamic slider instead of full tile re-render.\n- [ ] Add per-state AWS S3 public buckets for elevation data.\n- [ ] Terraform module to deploy to Fargate with ALB / CloudFront CDN.\n\n---\n\n## 📄 License\n\nDistributed under the **MIT License**.\n© 2024 FloodMap contributors\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcipher982%2Ffloodmap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcipher982%2Ffloodmap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcipher982%2Ffloodmap/lists"}