{"id":46418980,"url":"https://github.com/aonurakman/demandify","last_synced_at":"2026-03-05T15:02:05.069Z","repository":{"id":336881393,"uuid":"1144266508","full_name":"aonurakman/demandify","owner":"aonurakman","description":"Reproduce real-world traffic congestion with synthetic demand calibration for agent-based traffic scenarios using genetic algorithms.","archived":false,"fork":false,"pushed_at":"2026-03-04T17:49:28.000Z","size":27844,"stargazers_count":0,"open_issues_count":3,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-04T23:38:20.418Z","etag":null,"topics":["fastapi","genetic-algorithm","genetic-algorithms","openstreetmap","python","python3","sumo","tomtom","tomtom-api","traffic-simulation"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aonurakman.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.bib","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-01-28T13:53:24.000Z","updated_at":"2026-02-19T13:10:18.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/aonurakman/demandify","commit_stats":null,"previous_names":["aonurakman/demandify"],"tags_count":3,"template":false,"template_full_name":null,"purl":"pkg:github/aonurakman/demandify","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aonurakman%2Fdemandify","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aonurakman%2Fdemandify/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aonurakman%2Fdemandify/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aonurakman%2Fdemandify/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aonurakman","download_url":"https://codeload.github.com/aonurakman/demandify/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aonurakman%2Fdemandify/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":30132585,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-03-05T14:41:47.141Z","status":"ssl_error","status_checked_at":"2026-03-05T14:41:21.567Z","response_time":93,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["fastapi","genetic-algorithm","genetic-algorithms","openstreetmap","python","python3","sumo","tomtom","tomtom-api","traffic-simulation"],"created_at":"2026-03-05T15:02:00.889Z","updated_at":"2026-03-05T15:02:05.062Z","avatar_url":"https://github.com/aonurakman.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"![demandify](https://github.com/aonurakman/demandify/blob/main/static/banner.png?raw=true)\n\n[![PyPI version](https://badge.fury.io/py/demandify.svg)](https://pypi.org/project/demandify/)\n[![DOI](https://zenodo.org/badge/1144266508.svg)](https://doi.org/10.5281/zenodo.18698977)\n[![Reproducibility Check](https://github.com/aonurakman/demandify/actions/workflows/reproducibility.yml/badge.svg)](https://github.com/aonurakman/demandify/actions/workflows/reproducibility.yml)\n\n\n# Welcome to demandify!\n\n**Turn real-world traffic data into agent-based SUMO traffic scenarios.**\n\nDo you want to recreate real-world city traffic but don't have access to precious driver trip data? **demandify** solves that.\n\nPick a spot on the map and demandify will:\n1.  Fetch real-time congestion data from TomTom 🗺️\n2.  Build a clean SUMO network 🛣️\n3.  Use the Genetic Algorithm to figure out the demand pattern to match that traffic 🧬\n4. Produces a ready-to-run SUMO scenario in agent-level precision that allows you to test your urban routing policies, even for your CAVs! ([wink](https://github.com/COeXISTENCE-PROJECT/URB) [wink](https://github.com/COeXISTENCE-PROJECT/RouteRL)).\n\n![Workflow](https://github.com/aonurakman/demandify/blob/main/static/schema.png?raw=true)\n\n## Features\n\n- 🌍 **Real-world calibration**: Uses TomTom Traffic Flow API for live congestion data\n- 📦 **Offline calibration import**: Run from bundled/offline traffic+network snapshots\n- 🎯 **Seeded \u0026 reproducible**: Same seed = identical results for same congestion and bbox\n- 🚗 **Car-only SUMO networks**: Automatic OSM → SUMO conversion with car filtering, clean networks\n- 🧬 **Genetic algorithm**: Optimizes demand to match observed speeds, with advanced dynamics (elite-slice parent selection, immigrants, assortative mating, adaptive mutation boost)\n- 💾 **Smart caching**: Content-addressed caching for fast re-runs (traffic snapshots bucketed to 5-minute windows)\n- 📊 **Beautiful reports**: HTML reports with visualizations and statistics\n- ⌨️ **CLI native**: Live in the terminal? No problem.\n- 🖥️ **Clean web UI**: Leaflet map, real-time progress stepper, log console\n- ✅ **Data quality labeling**: Feasibility check now reports a quality score/label before calibration starts\n\n![GUI Screenshot](https://github.com/aonurakman/demandify/blob/main/static/gui.png?raw=true)\n\n## Quickstart\n\n### 1. Install demandify\n\n```bash\n# Install from PyPI (Recommended)\npip install demandify\n```\n\nIf you want to contribute or install from source:\n```bash\ngit clone https://github.com/aonurakman/demandify.git\ncd demandify\npip install -e .\n```\n\n### 2. Install SUMO 🚦\n\n**demandify** requires SUMO (Simulation of Urban MObility) to power its simulations.\n\n\u003e [!IMPORTANT] \n\u003e demandify is developed and tested with SUMO version 1.26.0. Ensure that your SUMO version is up to date.\n\n👉 **[Download SUMO from the official website](https://eclipse.dev/sumo/)**\n\nOnce installed, verify it's working:\n```bash\ndemandify doctor\n```\n\n### 3. Get a TomTom API Key\n\n1. Sign up at [https://developer.tomtom.com/](https://developer.tomtom.com/)\n2. Create a new app and copy the API key\n3. The free tier includes 2,500 requests/day\n\n### 4. Run demandify\n\n```bash\ndemandify\n```\n\nThis starts the web server at [http://127.0.0.1:8000](http://127.0.0.1:8000)\n\n### 5. Calibrate a scenario\n\n1. **Choose mode** at the top:\n   - `Create`: live TomTom + OSM fetch\n   - `Import`: select existing offline dataset (bbox auto-loaded and locked)\n2. **Draw a bounding box** on the map (Create mode only)\n3. **Configure parameters** (defaults work well):\n   - Time window: 15, 30, or 60 minutes\n   - Seed: any integer for reproducibility\n   - Warmup: a few minutes to populate the network\n   - GA population/generations: controls quality vs speed\n4. **Paste your API key** (Create mode only; one-time, stored locally)\n5. **Click \"Start Calibration\"**\n6. **Watch the progress** through 8 stages\n7. **Download your scenario** with `demand.csv`, SUMO network, and report\n\nBefore calibration starts, demandify runs a preparation feasibility check and reports:\n- fetched traffic segments\n- matched observed edges\n- total network edges\n- data quality label + score + risk flags\n   \n### 6. Run Headless (Optional) 🤖\n\nYou can run the full calibration pipeline directly from the command line, ideal for automation or remote servers.\n\n```bash\n# Basic usage (defaults: window=15, pop=50, gen=20)\ndemandify run \"2.2961,48.8469,2.3071,48.8532\" --name Paris_Test_01\n\n# Advanced usage with custom parameters\ndemandify run \"2.2961,48.8469,2.3071,48.8532\" \\\n  --name paris_v1 \\\n  --window 30 \\\n  --seed 123 \\\n  --pop 100 \\\n  --gen 50 \\\n  --mutation 0.5 \\\n  --elitism 2\n\n# With advanced GA dynamics\ndemandify run \"2.2961,48.8469,2.3071,48.8532\" \\\n  --name paris_v2 \\\n  --pop 100 \\\n  --gen 100 \\\n  --immigrant-rate 0.05 \\\n  --stagnation-patience 15\n\n# Fully non-interactive (automation/CI)\ndemandify run \"2.2961,48.8469,2.3071,48.8532\" \\\n  --name paris_v3 \\\n  --non-interactive\n\n# Import existing offline dataset (no live TomTom/OSM fetch)\ndemandify run --import krakow_v1 --name krakow_remote\n```\n\n\u003e **Note:** By default, the CLI pauses after fetching/matching data and asks for confirmation, then asks whether to run another calibration. Pass `--non-interactive` to auto-approve and exit immediately after pipeline completion.\n\n### 7. Build Offline Dataset (Optional) 💾\n\nIf you want a reusable prep bundle (for future no-key workflows), open:\n\n- [http://127.0.0.1:8000/dataset-builder](http://127.0.0.1:8000/dataset-builder)\n\nThis dedicated page is separate from calibration runs. It executes preparation only (traffic snapshot + OSM + SUMO network + map matching) and stores files under:\n\n- `demandify_datasets/\u003cdataset_name\u003e/`\n\nEach dataset includes `data/traffic_data_raw.csv`, `data/observed_edges.csv`, `data/map.osm`, `sumo/network.net.xml`, and `dataset_meta.json`.\n\n`dataset_meta.json` now includes a computed data quality block (`score`, `label`, `recommendation`, and metrics) to help decide whether a dataset is strong enough for offline calibration.\n\nBundled snapshot previews:\n\n| Den Haag (`den_haag_v1`) | Krakow (`krakow_v1`) | Eskisehir (`eskisehir_v1`) |\n|---|---|---|\n| ![Den Haag offline network](https://github.com/aonurakman/demandify/blob/main/demandify/offline_datasets/den_haag_v1/plots/network.png?raw=true) | ![Krakow offline network](https://github.com/aonurakman/demandify/blob/main/demandify/offline_datasets/krakow_v1/plots/network.png?raw=true) | ![Eskisehir offline network](https://github.com/aonurakman/demandify/blob/main/demandify/offline_datasets/eskisehir_v1/plots/network.png?raw=true) |\n\n#### Parameters\n\n| Argument | Type | Default | Description |\n|----------|------|---------|-------------|\n| `bbox` | String | Req* | Bounding box (`west,south,east,north`) |\n| `--import` | String | None | Use an offline dataset by name (or `source:name`) |\n| `--name` | String | Auto | Custom Run ID/Name |\n| `--non-interactive` | Flag | off | Disable prompts (auto-approve and exit when pipeline completes) |\n| `--window` | Int | 15 | Simulation duration (min) |\n| `--warmup` | Int | 5 | Warmup duration before scoring (min) |\n| `--seed` | Int | 42 | Random seed |\n| `--step-length`| Float | 1.0 | SUMO step length (seconds) |\n| `--workers` | Int | Auto (CPU count) | Parallel GA workers |\n| `--tile-zoom` | Int | 12 | TomTom vector flow tile zoom |\n| `--pop` | Int | 50 | GA Population size |\n| `--gen` | Int | 20 | GA Generations |\n| `--mutation`| Float | 0.5 | Mutation rate (per individual) |\n| `--crossover`| Float| 0.7 | Crossover rate |\n| `--elitism` | Int | 2 | Top individuals to keep |\n| `--sigma` | Int | 20 | Mutation magnitude (step size) |\n| `--indpb` | Float | 0.3 | Mutation probability (per gene) |\n| `--max-ods` | Int | 50 | Max OD pairs to generate |\n| `--bin-size` | Float | 5 | Time bin size in minutes |\n| `--initial-population` | Int | 1000 | Target initial number of vehicles (controls sparse initialization) |\n\n\\* `bbox` is required in create mode. In import mode, use `--import` and do not pass `bbox`.\n\n`Import` mode constraints:\n- positional `bbox` is rejected\n- `--tile-zoom` is rejected\n- all calibration controls (seed, GA params, warmup/window, etc.) remain available\n\n#### Advanced GA Dynamics\n\nThese parameters control diversity mechanisms and adaptive behavior in the genetic algorithm, addressing local optima stagnation and trip count explosion.\n\n| Argument | Type | Default | Description |\n|----------|------|---------|-------------|\n| `--immigrant-rate` | Float | 0.03 | Fraction of random individuals injected per generation (0–1) |\n| `--elite-top-pct` | Float | 0.1 | Defines the size of the top-by-`E` elite slice per generation: `n=max(1, elite_top_pct * population)` |\n| `--stagnation-patience` | Int | 20 | Generations without improvement before mutation boost activates |\n| `--stagnation-boost` | Float | 1.5 | Multiplier for mutation sigma and rate during stagnation |\n| `--checkpoint-interval` | Int | 10 | Save best-individual checkpoint artifacts every N generations |\n| `--assortative-mating` | Flag | off | Explicitly enable assortative mating |\n| `--no-assortative-mating` | Flag | off | Disable assortative mating (dissimilar parent pairing, on by default) |\n| `--deterministic-crowding` | Flag | off | Explicitly enable deterministic crowding |\n| `--no-deterministic-crowding` | Flag | off | Disable deterministic crowding (diversity-preserving replacement, on by default) |\n\nAll advanced dynamics are **enabled by default** with conservative values. For most use cases, the defaults work well. You can disable features via the corresponding `--no-*` flags or explicitly force-enable them with `--assortative-mating` / `--deterministic-crowding`.\n\n## How It Works\n\ndemandify follows a multi-stage pipeline:\n\n1. **Validate inputs** - Check mode/parameters and feasibility\n2. **Preparation**:\n   - `Create`: fetch traffic + OSM, build network, match edges\n   - `Import`: load/copy network + observed traffic files from offline dataset\n3. **Initialize demand** - Select routable OD pairs (lane-permission aware) and time bins\n4. **Calibrate demand** - Run GA to optimize vehicle counts\n5. **Export scenario** - Generate `demand.csv`, `trips.xml`, config, and report\n\n### Advanced GA Dynamics\n\nThe genetic algorithm includes several mechanisms to avoid common pitfalls like local optima stagnation and trip count explosion:\n\n- **Elite-slice parent selection**: Individuals are first ordered by flow-fit error `E`, and the top slice (`n=max(1, elite_top_pct * population)`) becomes the parent pool. Inside that slice, demandify builds a secondary score from equally weighted normalized ranks of `E`, `fail_total`, and genome magnitude, so main fit, reliability, and total demand all matter on comparable scales.\n- **Random immigrants**: A small fraction of completely random individuals is injected each generation to maintain genetic diversity and escape local optima.\n- **Assortative mating**: Parents are paired by dissimilarity (by genome magnitude) for crossover, promoting exploration of the search space.\n- **Deterministic crowding**: Offspring compete with similar parents for population slots, preserving niche diversity.\n- **Adaptive mutation boost**: If the best fitness stagnates for K generations, mutation sigma and rate are temporarily increased by a configurable multiplier. They reset automatically when improvement resumes.\n\nThe final return policy uses that same **elite-slice secondary score** across generations, so the returned individual comes from the strongest top-by-`E` slice while still balancing failures and total demand.\n\nThe calibration report includes plots for **genotypic diversity** (mean pairwise L2 distance) and **phenotypic diversity** (σ of fitness values) across generations, along with markers indicating when mutation boost was active.\n\n### Variability \u0026 Consistency\n      \nWhile demandify uses seeding (random seed) for all internal stochastic operations (OD selection, GA evolution), **perfect reproducibility is not guaranteed** due to the inherently chaotic nature of traffic microsimulation (SUMO) and real-time data inputs.\n      \nSeeding ensures *consistency* (runs look similar), but small timing differences in OS scheduling or dynamic routing decisions can lead to divergent outcomes. Traffic snapshots are cached in 5-minute buckets; using the same seed, bbox, and time bucket will reproduce demand.csv and SUMO randomness.\n      \n### Caching\n      \ndemandify caches:\n- OSM extracts (by bbox)\n- SUMO networks (by bbox + conversion params)\n- Traffic snapshots (by bbox + provider + style + tile zoom + 5-minute timestamp bucket)\n- Map matching results (by bbox + network key + provider + timestamp bucket)\n      \nCache location: `~/.demandify/cache/`\n\nClear cache: `demandify cache clear`\n\n## CLI Commands\n\n```bash\n# Start web server (default)\ndemandify\n\n# Run headless calibration\ndemandify run \"west,south,east,north\"\n\n# Run headless from bundled offline dataset\ndemandify run --import krakow_v1\n\n# Check system requirements\ndemandify doctor\n\n# Set TomTom API key (CLI)\ndemandify set-key YOUR_KEY_HERE\n\n# Clear cache\ndemandify cache clear\n\n# Show version\ndemandify --version\n```\n\n## Output Files\n\nEach run creates a folder with:\n\n- **`demand.csv`** - Travel demand with exact schema:\n  - `ID`, `origin link id`, `destination link id`, `departure timestep`\n- **`trips.xml`** - SUMO trips file\n- **`network.net.xml`** - SUMO network\n- **`scenario.sumocfg`** - SUMO configuration (ready to run; ignores route errors by default)\n- **`observed_edges.csv`** - Observed traffic speeds\n- **`run_meta.json`** - Complete run metadata\n- **`report.html`** - Calibration report with visualizations\n\nRun the scenario:\n```bash\ncd demandify_runs/run_\u003ctimestamp\u003e\nsumo-gui -c scenario.sumocfg\n```\n\n## Configuration\n\n### API Keys\n\nThree ways to provide your TomTom API key:\n\n1. **Web UI**: Paste in the form (saved to `~/.demandify/config.json`)\n2. **Environment variable**: `export TOMTOM_API_KEY=your_key`\n3. **`.env` file**: Copy `.env.example` to `.env` and add your key\n4. **CLI**: `demandify set-key YOUR_KEY` stores it in `~/.demandify/config.json`\n\n## Development\n\n```bash\n# Install with dev dependencies\npip install -e \".[dev]\"\n\n# Run tests\npytest\n\n# Format code\nblack demandify/\n\n# Lint\nruff check demandify/\n```\n\n## License\n\nMIT\n\n## Acknowledgments\n\n- **SUMO**: [Eclipse SUMO](https://eclipse.dev/sumo/)\n- **TomTom**: [Traffic Flow API](https://developer.tomtom.com/traffic-api)\n- **OpenStreetMap**: [© OpenStreetMap contributors](https://www.openstreetmap.org/copyright)\n\n## Citation\n\nIf you use this software for your research, please consider using the citation below.\nThe canonical metadata for GitHub's \"Cite this repository\" is in `CITATION.cff`.\n\n```bibtex\n@software{demandify_2026,\n  author       = {{Ahmet Onur Akman}},\n  title        = {{demandify: Calibrate SUMO traffic scenarios against real-world congestion using genetic algorithms}},\n  year         = {2026},\n  version      = {0.0.4},\n  publisher    = {PyPI},\n  url          = {https://pypi.org/project/demandify/},\n  repository   = {https://github.com/aonurakman/demandify},\n  doi          = {10.5281/zenodo.18877067}\n}\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faonurakman%2Fdemandify","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faonurakman%2Fdemandify","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faonurakman%2Fdemandify/lists"}