{"id":51747824,"url":"https://github.com/hyperphantasia/paris-flood-dataset","last_synced_at":"2026-07-18T21:42:49.186Z","repository":{"id":341412554,"uuid":"1170029112","full_name":"hyperphantasia/paris-flood-dataset","owner":"hyperphantasia","description":"Fluctuat nec mergitur dataset generator. Fetches daily max water levels from the Seine in Paris.","archived":false,"fork":false,"pushed_at":"2026-03-12T20:51:18.000Z","size":384,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-05-16T14:54:02.329Z","etag":null,"topics":["concurrent-fetching","data-analysis","data-cleaning","data-ingestion","data-pipeline","environmental-data","etl","flood","flood-detection","gap-detection","hubeau","hydrology","hydrometry","open-data","open-science","pandas","python","reproducible-research","river-gauges","time-series"],"latest_commit_sha":null,"homepage":"https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"unlicense","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hyperphantasia.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-01T15:43:38.000Z","updated_at":"2026-03-12T20:51:21.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hyperphantasia/paris-flood-dataset","commit_stats":null,"previous_names":["hyperphantasia/paris-flood-dataset","brooks-code/paris-flood-dataset"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hyperphantasia/paris-flood-dataset","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fparis-flood-dataset","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fparis-flood-dataset/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fparis-flood-dataset/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fparis-flood-dataset/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hyperphantasia","download_url":"https://codeload.github.com/hyperphantasia/paris-flood-dataset/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hyperphantasia%2Fparis-flood-dataset/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35632492,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-18T02:00:07.223Z","response_time":61,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["concurrent-fetching","data-analysis","data-cleaning","data-ingestion","data-pipeline","environmental-data","etl","flood","flood-detection","gap-detection","hubeau","hydrology","hydrometry","open-data","open-science","pandas","python","reproducible-research","river-gauges","time-series"],"created_at":"2026-07-18T21:42:48.689Z","updated_at":"2026-07-18T21:42:49.176Z","avatar_url":"https://github.com/hyperphantasia.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# La Seine à Paris, a daily max water levels data collector\n\n**Fluctuat nec mergitur.**\n\n![banner image](img/nivose.gif \"A pixel-art concept about an unfinished adventure game set during the Paris centennial flood.\")\n\u003cbr\u003e[Paris est né](https://en.wikipedia.org/wiki/Fluctuat_nec_mergitur), comme on sait, dans cette vieille île de la Cité qui a la forme d'un berceau. La grève de cette île fut sa première enceinte, la Seine son premier fossé. *(Victor Hugo, Notre-Dame de Paris, 1831).*\n\n## Genesis\n\n\u003e This is a follow-up project inspired by the Hackaviz 2025 [dataset](https://github.com/Toulouse-Dataviz/hackaviz-2025). This repository contains a Python script that builds the Paris flood dataset available on [Kaggle](https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset). The Joy Division inspired data visualization is available in the following [notebook]() and this was my original submission for the Hackathon: [1](https://github.com/hyperphantasia/miniature-fortnight) and [2](https://hyperphantasia.github.io/miniature-fortnight).\n\nThe resulting dataset is useful for studying flood risk indicators and their related seasonal pattern. Curious about some creative applications? The [dataset](https://www.kaggle.com/datasets/grimespoint/paris-flood-dataset) is available on Kaggle with a fancy jupyter [notebook](https://www.kaggle.com/code/grimespoint/paris-flood-dataset-the-design-data-visualization).\n\n## Table of contents\n\n\u003cdetails\u003e\n\u003csummary\u003eContents - click to expand\u003c/summary\u003e\n\n- [La Seine à Paris, a daily max water levels data collector](#la-seine-à-paris-a-daily-max-water-levels-data-collector)\n  - [Genesis](#genesis)\n  - [Table of contents](#table-of-contents)\n  - [Project overview](#project-overview)\n  - [Features](#features)\n  - [Functional overview](#functional-overview)\n  - [Architecture](#architecture)\n  - [Installation](#installation)\n    - [Clone the repository](#clone-the-repository)\n    - [Create a virtual environment (optional but recommended)](#create-a-virtual-environment-optional-but-recommended)\n    - [Install dependencies](#install-dependencies)\n  - [Configuration](#configuration)\n    - [Available metrics](#available-metrics)\n  - [Usage](#usage)\n  - [Data structure](#data-structure)\n  - [Sample output](#sample-output)\n  - [Data processing pipeline](#data-processing-pipeline)\n    - [Step 1: fetching](#step-1-fetching)\n      - [Fetching is concurrent](#fetching-is-concurrent)\n      - [About the date-based pagination strategy](#about-the-date-based-pagination-strategy)\n      - [Error handling strategy](#error-handling-strategy)\n  - [Step 2: merging](#step-2-merging)\n  - [Step 3: duplicate detection](#step-3-duplicate-detection)\n  - [Step 4: gap detection](#step-4-gap-detection)\n    - [1. Build full date range](#1-build-full-date-range)\n    - [2. Detect missing days](#2-detect-missing-days)\n    - [3. Group consecutive missing days](#3-group-consecutive-missing-days)\n      - [Gap interval construction algorithm](#gap-interval-construction-algorithm)\n  - [Limitations](#limitations)\n  - [Possible enhancements](#possible-enhancements)\n  - [Contributing](#contributing)\n  - [License](#license)\n\n\u003c/details\u003e\n\n## Project overview\n\n\u003e This project demonstrates production-style data ingestion patterns and concurrent API handling using Python.\n\nThe algorithm fetches historical daily maximum water height observations for multiple hydrometric stations from this API endpoint:\n\n```markdown\nhttps://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab\n```\n\nIt then:\n\n- Merges all station's data\n- Detects duplicate dates\n- Detects missing dates across the global time series\n- Builds continuous missing intervals\n- Exports a chronologically sorted CSV file\n\nOutput file:\n\n```text\nparis_flood_dataset.csv\n```\n\n## Features\n\n- **Modular** architecture\n- **Connection reuse** via `requests.Session`\n- **Concurrent** multi-station data fetching\n- Automatic **date-based pagination**\n- **Global duplicate** detection\n- Continuous missing-date **interval detection**\n- **Linear-time gap detection** algorithm\n- **Chronological** CSV export\n\n## Functional overview\n\nThe pipeline runs in three major steps:\n\n```markdown\n+-----------------------------+\n|        Pipeline Flow        |\n+-----------------------------+\n| 1) **Concurrent Collection**|\n|    - `ThreadPoolExecutor`   |\n|    - Fetch stations in parallel\n+-----------------------------+\n| 2)  **Data Consolidation**  |\n|    - Merge into one DataFrame\n+-----------------------------+\n| 3) **Integrity Analysis**   |\n|    - Reconstruct global time range\n|    - *Detect:* Missing days,  |\n|      Continuous missing ranges,\n|      Duplicate days         |\n+-----------------------------+\n```\n\n## Architecture\n\nA Separate ingestion logic from the validation logic.\n\n```text\nMain\n ├── concurrent_fetch()\n │     ├── ThreadPoolExecutor\n │     └── fetch_station()\n │            ├── API calls\n │            ├── Pagination loop\n │            └── DataFrame assembly\n │\n ├── detect_global_gaps()\n |       ├── Duplicate detection\n |       ├── Full date range reconstruction\n |       ├── Missing date detection\n |       └── Interval grouping\n └── CSV sort \u0026 export\n```\n\n## Installation\n\n### Clone the repository\n\n```bash\ngit clone https://github.com/hyperphantasia/paris-flood-dataset.git\ncd paris-flood-dataset\n```\n\n### Create a virtual environment (optional but recommended)\n\n```bash\npython -m venv venv\nsource venv/bin/activate  # macOS/Linux\nvenv\\Scripts\\activate     # Windows\n```\n\n### Install dependencies\n\n- Python 3.10+\n\n```bash\npip install -r requirements.txt\n```\n\n*requirements.txt*\n\n```text\nrequests\npandas\n```\n\n## Configuration\n\n- Default parameters:\n\n```python\nAPI_BASE = \"https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab\"\nMETRIC = \"HIXnJ\"\nOUTPUT = \"paris_flood_dataset.csv\"\nMAX_PER_PAGE = 20000\nMAX_WORKERS = 5\n```\n\n| Parameter | Type | Example | Description |\n|---|---:|---|---|\n| `API_BASE` | string | `\"https://hubeau.eaufrance.fr/api/v2/hydrometrie/obs_elab\"` | Base URL of the API endpoint to query hydrometry observations. Max URL size: *2083 characters*.|\n| `METRIC` | string | `\"HIXnJ\"` | Identifier of the specific metric to request from the API. **HIXnJ**= observed daily max in mm. Other available metrics: [see Metrics](#available-metrics). |\n| `OUTPUT` | string (filename) | `\"paris_flood_dataset.csv\"` | Local filename where the downloaded results will be saved (CSV format). |\n| `MAX_PER_PAGE` | integer | `20000` | Maximum number of records to request per API page (page size / limit parameter). For the *Hub'Eau API*: default is *5000*, max is *20000*. |\n| `MAX_WORKERS` | integer | `5` | Maximum number of concurrent worker threads/processes to use for parallel requests. |\n\n\u003e [!TIP]\n\u003e For I/O-bound (API requests), optimal MAX_WORKERS ≈ $\\min(4c, \\lfloor R \\cdot t \\rfloor, M)$\n\u003e \n\u003e where c = cores, R = requests/sec, t = avg time, M = memory limit\n\n- Hydrometric stations:\n\n```python\nSTATIONS = [\n    \"F700000109\",\n    \"F700000110\",\n    ...\n]\n```\n\nVarious hydrometric stations references can be found on the french hydroportail. Below is the reference for **la Seine à Paris** stations:\n\n```text\nhttps://www.hydro.eaufrance.fr/sitehydro/F7000001/fiche\n```\n\n### Available metrics\n\n| *obs_elab* Code | French | English |\n|---|---|---|\n| **QmnJ** | débit moyen journalier | daily mean flow |\n| **QmM** | débit moyen mensuel | monthly mean flow |\n| **HIXnJ** | **hauteur instantanée maximale journalière en mm** | **daily maximum instantaneous water level in mm** |\n| **HIXM** | hauteur instantanée maximale mensuelle | monthly maximum instantaneous water level |\n| **QIXnJ** | débit instantané maximal journalier | daily maximum instantaneous flow |\n| **QIXM** | débit instantané maximal mensuel | monthly maximum instantaneous flow |\n| **QINnJ** | débit instantané minimal journalier | daily minimum instantaneous flow |\n| **QINM** | débit instantané minimal mensuel | monthly minimum instantaneous flow |\n\nMore [details](https://hubeau.eaufrance.fr/page/api-hydrometrie).\n\n## Usage\n\nRun:\n\n```bash\npython fluctuat_nec_mergitur.py\n```\n\n- Sample console output:\n\n```text\nProcessing station F700000109\nProcessing station F700000110\nProcessing station F700000111\nProcessing station F700000102\nProcessing station F700000103\n   From F700000103 → 7316 rows\n   From F700000111 → 5842 rows\n   From F700000110 → 2920 rows\n   From F700000109 → 20000 rows\n   From F700000109 → 4104 rows\n   From F700000102 → 13147 rows\n\nSorted and saved data to paris_flood_dataset.csv\nstart_date: 1900-01-02 00:00:00\nend_date: 2026-02-27 00:00:00\nexpected_days: 46078\nobserved_days: 46059\nmissing_days_count: 19\nduplicate_records_count: 0\nmissing_ranges: [(Timestamp('1965-12-31 00:00:00'), Timestamp('1966-01-01 00:00:00')), (Timestamp('1973-12-31 00:00:00'), Timestamp('1974-01-01 00:00:00')), (Timestamp('1989-12-31 00:00:00'), Timestamp('1990-01-01 00:00:00')), (Timestamp('1992-06-29 00:00:00'), Timestamp('1992-07-04 00:00:00')), (Timestamp('1994-09-06 00:00:00'), Timestamp('1994-09-06 00:00:00')), (Timestamp('1994-09-23 00:00:00'), Timestamp('1994-09-23 00:00:00')), (Timestamp('1994-09-30 00:00:00'), Timestamp('1994-09-30 00:00:00')), (Timestamp('1995-10-20 00:00:00'), Timestamp('1995-10-20 00:00:00')), (Timestamp('1998-12-31 00:00:00'), Timestamp('1998-12-31 00:00:00')), (Timestamp('1999-05-20 00:00:00'), Timestamp('1999-05-20 00:00:00')), (Timestamp('2000-04-03 00:00:00'), Timestamp('2000-04-03 00:00:00'))]\n```\n\n## Data structure\n\nFinal CSV columns include:\n\n| Column Name           | Description |\n|----------------------|-------------|\n| **code_site**          | Location ID |\n| **code_station**          | Station ID |\n| **date_obs_elab**        | Observation date |\n| **resultat_obs_elab**    | Observed value: daily max water level (in mm) |\n| **grandeur_hydro_elab**  | Metric code (*HIXnJ*) |\n| **date_prod**  | Data production date (processing date)|\n| **code_statut**       | Vamlidation status code |\n| **libelle_statut**         | Validation status label |\n| **code_methode**      | Production method code |\n| **libelle_methode**        | Production method label |\n| **code_qualification**         | Data quality assesment code |\n| **libelle_qualification**         | Data quality assesment label |\n| **longitude**         | Station longitude|\n| **latitude**         |  Station latitude |\n| **grandeur_hydro_elab**         | Observation metric |\n\nCode values are explained in this [document](/doc/codes%20observations%20hydro.pdf) (*french*).\n\n## Sample output\n\n```csv\ncode_site,code_station,date_obs_elab,resultat_obs_elab,date_prod,code_statut,libelle_statut,code_methode,libelle_methode,code_qualification,libelle_qualification,longitude,latitude,grandeur_hydro_elab\nF7000001,F700000109,1900-01-02,1300.0,2025-06-17T09:27:10Z,16,Donnée validée,0,Mesurée,20,Bonne,2.365515502,48.845409133,HIXnJ\n```\n\n## Data processing pipeline\n\n### Step 1: fetching\n\nThe `fetch_station()` function:\n\n1. Starts at a fixed historical date (`1900-01-01`)\n2. Fetches up to `MAX_PER_PAGE` rows\n3. Extracts the last observation date\n4. Resumes from `last_date` + 1 day\n5. Stops when fewer than `MAX_PER_PAGE` rows are returned\n\n#### Fetching is concurrent\n\nUses:\n\n```python\nThreadPoolExecutor(max_workers=5)\n```\n\nWhy threads?\n\n- API calls are I/O-bound\n- Threads improve latency\n- Controlled worker count prevents overload\n\n#### About the date-based pagination strategy\n\nThis avoids inefficient offset-based pagination (using page numbers), and allows full coverage.\n\nStrategy:\n\n- Fetch maximum allowed rows\n- Use last returned date\n- Continue from `last_date` + 1 day\n\nWhy?\n\n- Avoids offset inefficiency\n- Prevents skipped records\n- Robust to changing API data\n\n#### Error handling strategy\n\n- `raise_for_status()` validates HTTP responses\n- Exceptions handled per station\n- Failed station does not crash the pipeline\n\n## Step 2: merging\n\nAll station DataFrames are concatenated:\n\n```python\npd.concat(all_data, ignore_index=True)\n```\n\n## Step 3: duplicate detection\n\nDuplicates across stations are detected using:\n\n```python\ndf.duplicated(date_col)\n```\n\nDuplicates are counted before being dropped.\n\n## Step 4: gap detection\n\n### 1. Build full date range\n\n```python\npd.date_range(start, end, freq=\"D\")\n```\n\nThis reconstructs all expected calendar days.\n\n### 2. Detect missing days\n\n```python\nmissing_days = full_range.difference(observed)\n```\n\n### 3. Group consecutive missing days\n\nExample:\n\nInput missing days:\n\n```text\nJan 1, Jan 2, Jan 3, Jan 10, Jan 11\n```\n\nOutput intervals:\n\n```text\n(Jan 1 - Jan 3)\n(Jan 10 - Jan 11)\n```\n\nThis is done via linear scanning.\n\n#### Gap interval construction algorithm\n\nPseudo-code:\n\n```python\ngap_start = first_missing_day\nfor each current_day in missing_days:\n    if current_day - previous_day \u003e 1:\n        close previous interval\n        start new interval\n```\n\nThis is:\n\n- Single-pass\n- Linear time\n- Memory efficient\n\n## Limitations\n\n\u003e[!WARNING]\n\u003e Memory usage grows linearly with dataset size.\n\n- Consider chunked writing for \u003e10M rows\n- Consider parquet for larger datasets\n\n## Possible enhancements\n\n- [ ] Validate API schema before processing\n- [ ] Implement a retry/backoff mechanism\n- [ ] Timeout configuration per request\n- [ ] Asynchronous I/O (aiohttp)\n- [ ] Control concurrency with a Semaphore or a rate-limiter (token bucket)\n- [ ] Multiprocessing vs threading\n- [x] Write CSV once at end\n\n## Contributing\n\n1) Fork the repository.\n2) Create a feature branch (`git checkout -b feature/your‑feature`).\n3) Commit your changes (`git commit -m \"Add …\"`).\n4) Push and open a Pull Request.\n\n## License\n\nThis project is released into the public domain under the [Unlicense](https://unlicense.org/). See the [LICENSE](/LICENSE) file for details. Regarding the original data, be aware of the [Hub'Eau](https://assistance.brgm.fr/hubeau/quels-sont-droits-dusage-donnees-proposees-apis) platform  usage rights.\n\n\u003e[!NOTE]\n\u003e L’ensemble des données proposées dans le cadre des API sont des données publiques environnementales, déjà diffusées par ailleurs : elles sont donc librement utilisables et réutilisables, dans le cadre de la [licence ouverte interministérielle](https://www.data.gouv.fr/pages/legal/licences/etalab-2.0).\n\n*English translation:*\n\n\u003e All data provided through the APIs are public environmental data, already published elsewhere: they are therefore freely usable and reusable under the Interministerial Open License ([ETALAB](https://www.data.gouv.fr/pages/legal/licences/etalab-2.0)).\n\n♟\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperphantasia%2Fparis-flood-dataset","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhyperphantasia%2Fparis-flood-dataset","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhyperphantasia%2Fparis-flood-dataset/lists"}