{"id":50474794,"url":"https://github.com/skfd/toronto-2-address-import","last_synced_at":"2026-06-01T12:30:56.461Z","repository":{"id":352815106,"uuid":"1214667164","full_name":"skfd/toronto-2-address-import","owner":"skfd","description":"Conflates Toronto Open Data address points against OpenStreetMap, routes questionable items through a human review UI, and uploads approved additions to OSM.","archived":false,"fork":false,"pushed_at":"2026-05-22T01:02:50.000Z","size":6886,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-05-23T03:37:28.931Z","etag":null,"topics":["address-import","conflation","flask","open-data","openstreetmap","osm","python","toronto"],"latest_commit_sha":null,"homepage":"https://skfd.github.io/toronto-2-address-import/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/skfd.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-04-18T22:09:13.000Z","updated_at":"2026-05-22T01:02:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/skfd/toronto-2-address-import","commit_stats":null,"previous_names":["skfd/toronto-2-address-import"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/skfd/toronto-2-address-import","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skfd%2Ftoronto-2-address-import","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skfd%2Ftoronto-2-address-import/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skfd%2Ftoronto-2-address-import/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skfd%2Ftoronto-2-address-import/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/skfd","download_url":"https://codeload.github.com/skfd/toronto-2-address-import/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/skfd%2Ftoronto-2-address-import/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33775856,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-01T02:00:06.963Z","response_time":115,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["address-import","conflation","flask","open-data","openstreetmap","osm","python","toronto"],"created_at":"2026-06-01T12:30:54.861Z","updated_at":"2026-06-01T12:30:56.455Z","avatar_url":"https://github.com/skfd.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# t2-address-import\n\n[GitHub](https://github.com/skfd/toronto-2-address-import) · [Pilot evidence site](https://skfd.github.io/toronto-2-address-import/) · [OSM community discussion](https://community.openstreetmap.org/t/address-import-for-toronto/119368) · MIT licensed\n\nLocal tool that reads Toronto address points from the sibling\n[`toronto-addresses-import`](https://github.com/skfd/toronto-addresses-import) project's SQLite DB,\nconflates them against live OSM data, routes questionable items to a human\nreviewer via a web UI, and uploads approved candidates to the OpenStreetMap\n**dev sandbox** (`master.apis.dev.openstreetmap.org`). Every auto and manual\naction is written to an append-only audit log.\n\n## Status\n\nLive status of the [import proposal](IMPORT_PROPOSAL.mediawiki) against the [OSM Import Guidelines](https://wiki.openstreetmap.org/wiki/Import/Guidelines) workflow:\n\n| Stage | State |\n|---|---|\n| Draft proposal | Complete (last revised 2026-05-13) |\n| Wiki page (`Toronto/Import/AddressPoints`) | [Published 2026-05-01](https://wiki.openstreetmap.org/wiki/Toronto/Import/AddressPoints) |\n| OSM Community Forum announcement | Posted 2026-05-01 — [thread](https://community.openstreetmap.org/t/address-import-for-toronto/119368) (tagged `import`; the [Import Guidelines](https://wiki.openstreetmap.org/wiki/Import/Guidelines) route announcements through the forum now, not the deprecated `imports@` list) |\n| 14-day feedback window | Closes 2026-05-15 (measured from wiki publication; discussion resolved) |\n| Phase 1 pilot upload (production) | Completed 2026-05-13 — [changeset 182585291](https://www.openstreetmap.org/changeset/182585291) (tile `high-park-swansea-sw-se`, 176 uploaded, 72 skipped, 4 rejected). [Upload manifest](https://skfd.github.io/toronto-2-address-import/pilot/uploads/all.csv). One-week hold before Phase 2. |\n\nAll uploads from this tool to date have used the OSM dev sandbox (`master.apis.dev.openstreetmap.org`). No production edits have been made and none will be made until the proposal has cleared the customary feedback window. The OSM account used for production uploads will be [`skfd imports`](https://www.openstreetmap.org/user/skfd%20imports) (dedicated, not the maintainer's personal account).\n\n## Terminology\n\n**Candidate** and **AddressMatch** are synonyms — both refer to one row from\nthe input CSV paired with its OSM lookup result, the unit flowing through the\npipeline. Code, DB schema, and templates use `candidate`; discussion and new\ndocs may use either term. Each one carries three orthogonal axes:\n\n- **`verdict`** — what conflation decided (`MATCH`, `MATCH_FAR`, `MISSING`, `SKIPPED`)\n- **`status`** — what the operator decided (`OPEN`, `APPROVED`, `REJECTED`, `DEFERRED`); `AUTO_APPROVED` is a synthetic status the review queue derives for clean MISSING rows that bypass manual review\n- **`stage`** — where it sits in the pipeline (`INGESTED`, `CONFLATED`, `CHECKED`, `REVIEW_PENDING`, `APPROVED`, `REJECTED`, `UPLOADED`, `FAILED`, `SKIPPED`)\n\nA **Run** is one execution of the pipeline (produces many candidates) and is\nalso the unit of upload — one run becomes one OSM changeset.\n\n## Setup\n\n1. **Python 3.11+** (uses `tomllib`).\n2. From the project root:\n   ```bash\n   python -m venv .venv\n   .venv\\Scripts\\activate    # PowerShell / cmd\n   pip install -e .\n   ```\n3. **Register an OAuth2 application** on the OSM dev server:\n   - Log into \u003chttps://master.apis.dev.openstreetmap.org/\u003e.\n   - My Settings → OAuth 2 applications → **Register new application**.\n   - Name: anything (e.g. `t2-address-import-dev`).\n   - Redirect URI: `http://127.0.0.1:5000/oauth/callback` (OSM rejects `localhost` as non-HTTPS).\n   - Permissions: tick **read user preferences**, **modify the map**,\n     **comment on changesets**.\n   - Save; copy the resulting Client ID and Client Secret.\n4. **Create `.env.dev`** (copy `.env.dev.example`) and fill in:\n   ```\n   OSM_CLIENT_ID=...\n   OSM_CLIENT_SECRET=...\n   FLASK_SECRET_KEY=\u003cany random string\u003e\n   FERNET_KEY=\u003cgenerate with: python -c \"from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())\"\u003e\n   ```\n   For prod, also create `.env.prod` from `.env.prod.example` with a separate\n   set of OSM creds (registered on real OSM, not the dev sandbox) and its own\n   freshly generated `FERNET_KEY`.\n5. Adjust `config.toml` if your sibling DB lives somewhere else or you want a\n   different default bbox.\n\n## Run\n\n```bash\npython run.py\n```\n\nThen visit \u003chttp://localhost:5000/\u003e.\n\n## Targeting dev vs prod OSM\n\nThe tool defaults to the OSM **dev sandbox**\n(`master.apis.dev.openstreetmap.org`). The header shows a `DEV` / `PROD`\nbadge so you always know which server uploads will go to.\n\nSelection is via the `--env` flag on `run.py` (default `dev`):\n\n- **DEV (default):** loads `.env.dev` → `master.apis.dev.openstreetmap.org`\n- **PROD:** loads `.env.prod` → `api.openstreetmap.org`\n\nEach server has its own OAuth2 application registry, so a prod run also needs\na prod-side `OSM_CLIENT_ID` / `OSM_CLIENT_SECRET` — register a second app on\n\u003chttps://www.openstreetmap.org/oauth2/applications\u003e with the same redirect URI.\n\nTo launch against prod:\n\n```bash\npython run.py --prod        # or: python run.py --env prod\n```\n\nThat's the only switch — `run.py --prod` is what flips the tool to production.\nThe standalone CLIs (`scripts/run_one_tile.py`, `python -m t2.run_for_all`,\n`osm_refresh`, `tiles_build`, the static-export scripts) never touch the OSM\nAPI for uploads, so they always run against dev and take no `--env` flag.\n\nAll config — `OSM_API_BASE`, `OSM_CLIENT_ID`/`OSM_CLIENT_SECRET`,\n`OSM_REDIRECT_URI`, `FLASK_SECRET_KEY`, `FERNET_KEY` — is read **only** from\nthe selected `.env.{dev,prod}` file; the tool reads no environment variables\nfor these. To point at a different server (a local OSM instance, a staging\nhost), edit the relevant `.env` file (or create a third one).\n\nOnly one OAuth token set is stored at a time (`data/tool.db`), so switching\nenv requires re-authorizing on the new server. Each env file gets its own\n`FERNET_KEY` — they don't need to match.\n\nThe Geofabrik extract (Stage 2 read source) is the same in both modes — there\nis no dev-server slice from Geofabrik, and the dev sandbox has no realistic\nToronto data anyway. Only the upload target changes.\n\n## Local OSM extract (default source)\n\nStage 2 reads addresses from a locally-cached Toronto extract instead of\nquerying Overpass every time. First-time setup:\n\n```bash\npython -m t2.osm_refresh\n```\n\nThis downloads the latest Ontario PBF from Geofabrik (~600 MB) into\n`data/osm/ontario-latest.osm.pbf`, filters it to `addr:housenumber`-tagged\nfeatures clipped to the City-of-Toronto bbox in `config.toml`, and writes\n`data/osm/toronto-addresses.json` + a `meta.json` sidecar. Stage 2 then just\nbbox-clips that JSON per run — no network, sub-second.\n\nRe-run whenever you want a fresher snapshot. The tool HEAD-checks Geofabrik\nand skips the download if `Last-Modified` hasn't changed; pass `--force` to\nre-download regardless. `--dry-run` does only the HEAD check.\n\nYou can also trigger a refresh from the web UI at \u003chttp://localhost:5000/osm\u003e.\nThe page shows the extract's freshness, element counts, sha256s, and tails\n`data/osm/refresh.log` so you can watch progress. The button spawns the same\nCLI as a detached subprocess, so Flask stays responsive while the download\nruns.\n\nTo fall back to live Overpass queries (e.g. bbox experiments outside\nToronto), set `[osm] source = \"overpass\"` in `config.toml`.\n\n## Tile layer (run area picker)\n\nToronto is too big to pick by typing lat/lon, so the tool precomputes a tile\nlayer you can click on. Generate it once with:\n\n```bash\npython -m t2.tiles_build\n```\n\nThis downloads the City of Toronto's 158-neighbourhood polygon layer from\n[Open Data](https://open.toronto.ca/dataset/neighbourhoods/), counts active\nsource addresses inside each polygon, quadtree-splits any neighbourhood with\nmore than 500 addresses, then merges any tile under 250 addresses into a\nborder-sharing neighbour (soft ceiling 500, hard ceiling 750) so the operator\nnever reviews a near-empty tile. The result (~1,300 tiles, 250–750 addresses\neach) lands in `data/tiles.json` + a `data/tiles/meta.json` sidecar.\nRegenerate when a new source snapshot lands.\n\nThe dashboard's **Pick on map** button opens `/map` — click any tile to land\non its detail page, which lists prior runs on that tile and has a \"Start new\nrun\" form pre-filled with the tile's bbox. The manual bbox form on the\ndashboard remains as an escape hatch for arbitrary rectangles.\n\n## First end-to-end run\n\n1. **Create a run** from the dashboard. Either **Pick on map** and click a\n   tile, or type a small downtown rectangle like\n   `(43.645, -79.42, 43.665, -79.39)` into the bbox form.\n2. On the run page, click the four pipeline buttons in order:\n   **Ingest → Fetch OSM → Conflate → Run checks**.\n3. Open the **Review queue** — items flagged by any enabled check land here.\n   Approve, reject, or defer each. MISSING candidates with no flags are\n   auto-approved; MATCH candidates are auto-skipped.\n4. Back on the run page, scroll to the **Upload** card and pick one:\n   - `Upload to OSM` opens a changeset on the dev server, uploads the\n     osmChange diff, and closes the changeset. Visit `/oauth/start` first\n     if you haven't authorized yet.\n   - `Download .osm (JOSM)` writes a `.osm` file with the run's APPROVED\n     candidates; open it in JOSM and upload via JOSM's own auth, then click\n     `Mark uploaded (JOSM)` back on the run page.\n5. The **Audit log** at `/runs/\u003cid\u003e/audit` shows every event.\n\n## Resumability\n\nEvery candidate has a `stage` column. Killing the process mid-run and\nrestarting is safe — each stage skips work already done:\n\n- Re-running **Ingest** only adds new rows (`INSERT OR IGNORE`).\n- Re-running **Fetch** reuses the cached `data/osm_current_run\u003cid\u003e.json`.\n- Re-running **Conflate** resumes from any candidate still at `INGESTED`.\n- Re-running **Checks** skips any `(candidate, check_id, check_version)` that\n  already has a result row. Bump a check's `version` in code to force rerun.\n- **Uploads** look up prior changesets by their `import:client_token` tag\n  before opening a new one.\n\n## How conflation decides\n\nMatch targets are **pure address nodes** (`addr:housenumber` + no POI tags) and\n**polygons** (ways/relations with an address — typically buildings, including\namenity-tagged footprints like a hospital).\n\n**POI nodes** (nodes carrying `amenity`, `shop`, `office`, `tourism`, `leisure`,\n`craft`, `healthcare`, `building`, including any lifecycle-qualified form such\nas `disused:amenity` or `amenity:disused` — see `POI_TAG_KEYS` /\n`LIFECYCLE_QUALIFIERS` in `t2/conflate.py`) are **ignored** for matching: their address\nis a courtesy annotation, not the canonical address feature. When a POI sits at\na MISSING candidate's address, the review UI acknowledges it with a pill, and\nany `addr:postcode` on the POI is copied into the proposed upload tags.\n\nEven after that filter, a matched \"pure address\" node can quietly carry\nnon-address tags (`name`, `ref`, `entrance`). The `potential_amenity` check\nflags those with `severity=info` so we can refine the POI filter over time.\nMetadata keys like `source`, `opendata:type`, `check_date`, `note` are on an\nignore list inside the check and don't trigger it.\n\nStreet-name normalization (`STREET` → `ST`, `AVENUE` → `AVE`, etc. — see\n`STREET_SUFFIXES` in `t2/conflate.py`) covers suffix and direction variants\nbut cannot bridge spelling differences inside the proper-noun part of the\nname, including space-vs-no-space splits like source `Deane Field Crescent`\nvs OSM signage `Deanefield Crescent`. Conflation calls those MISSING; the\n`nearby_street_mismatch` check then flags any MISSING candidate whose OSM\nneighbour within ~20 m shares the housenumber under a different street\nname, so a reviewer can decide whether to accept the variant or fix the\nsource. Default radius is in `config.toml` under\n`[check_params.nearby_street_mismatch]`.\n\nOnce a variant is confirmed, it goes into `STREET_NAME_OVERRIDES` in\n`t2/conflate.py` — a hardcoded source-name → OSM-name table applied at\ningest. From then on the candidate carries the OSM name in `street_raw`\n(and the upload tag), so it MATCHes the existing OSM addresses instead of\nduplicating them under a parallel spelling. Current entries cover proper-\nnoun spacing (`Deane Field Cres → Deanefield Cres`, `Golfcrest Rd →\nGolf Crest Rd`, `Forest View Rd → Forestview Rd`, `Greenhouse Rd → Green\nHouse Rd`, `Posthorn Grv → Post Horn Grv`) and one outright suffix\ncorrection (`Kathleen Ave → Kathleen Cres` — the source has the street\ntype wrong; the addresses sit on what OSM and signage call Kathleen\nCrescent).\n\n## Out of scope (possible next phase)\n\nThe current pipeline is one-directional: Toronto source → OSM lookup → upload\nadditions. Two cleanup flows in the opposite direction are **explicitly out of\nscope** and left for a later phase. Documented here so reviewers don't assume\nthey were overlooked.\n\n### Removing OSM addresses absent from Toronto source\n\nIf OSM has an address that Toronto's active snapshot doesn't, we do not flag,\npropose, or remove it.\n\nReasoning — the absence direction is asymmetric. Toronto's open data is\nauthoritative when it asserts an address exists; silence is a weaker signal.\nThe feed has refresh lag, known-missing neighborhoods, and retired-address\nstates that aren't cleanly separable from \"never existed.\" Deleting OSM data\nbased on absence alone would destroy real addresses on worse evidence than we\naccept for additions.\n\nA future phase would need, at minimum: a reverse-sweep stage enumerating OSM\naddresses in the run bbox; a separate review queue (not `Candidate` — the\nverdicts don't fit); a street-level cross-check to suppress the common case\nwhere Toronto's feed is missing a whole street; prioritization by OSM metadata\n(`start_date`, last-edit age, `source`); and human-only approval — no\nautomation, since OSM deletions are high blast radius and hard to reverse.\n\n### Removing `addr:interpolation` ways\n\nOSM `addr:interpolation` ways synthesize housenumbers along a street segment\nbetween two endpoint nodes. When Toronto's per-address points cover the same\nsegment with real data, the interpolation way is technically redundant. We\nstill don't touch them.\n\nReasoning — an interpolation way isn't an address, it's a geometry-anchored\nrange declaration. Our matching model (housenumber + street + point) doesn't\ndescribe what's being replaced. Replacement needs cross-validation: every\ninteger in the interpolation range must have a real Toronto point before\nremoval, otherwise the delete leaves mapped gaps. It's also a bulk structural\nedit to OSM, not an address-import operation — different review bar, different\nchangeset hygiene, different rollback story than what this tool was built for.\n\nA future phase would need: enumeration of `addr:interpolation` ways in the\nbbox; coverage check that every integer in the range has a colocated Toronto\npoint; a proposed delete-way-plus-preserve-endpoints changeset for human\nreview; and care around tags (`addr:street`, `addr:postcode`) that the\ninterpolation way carries on behalf of its endpoints.\n\n### Why defer both\n\nThe shipping scope — \"get Toronto's missing civic addresses into OSM without\ncreating duplicates\" — has standalone value. Folding cleanup into the same\npipeline expands blast radius and review burden without proportional benefit,\nand the two reverse flows have different enough semantics (different data\nsources, different review criteria, different failure modes) that they\ndeserve their own pipelines when we get to them.\n\n## Writing a new check\n\n1. Create `t2/checks/\u003cname\u003e.py` exporting a class that matches the `Check`\n   protocol in `t2/checks/base.py`.\n2. Register it in `t2/checks/__init__.py`.\n3. Restart the app. The new check appears in the run's toggle list.\n\n## Drift back-scan\n\n`scripts/drift_backscan.py` re-evaluates the `match_number_drift` check\nagainst runs that were already conflated — useful for finding OSM positional\ndrift in runs processed before the check existed. It is read-only: it never\nwrites to `tool.db` and never creates review items.\n\n```bash\npython -m scripts.drift_backscan                       # uploaded runs only\npython -m scripts.drift_backscan --status all          # every run\npython -m scripts.drift_backscan --min-flags 3         # widen the summary\npython -m scripts.drift_backscan --out C:/tmp/d.csv    # custom CSV path\n```\n\nIt writes one CSV row per flagged candidate (`data/drift_backscan.csv` by\ndefault) — matched OSM element, the closer different-numbered OSM element,\nand both distances — and prints a console summary of \"systemic\" runs (those\nat or above `--min-flags`) and their drifted streets. The check's `slack_m`\nis read from `config.toml`, so the scan matches a fresh pipeline run.\n\n## Data sources \u0026 attribution\n\nThis tool moves data between three open datasets. Downstream uploads inherit OSM's licence, but the upstream sources each have their own terms:\n\n- **Toronto Open Data** — \"Address Points (Municipal) – Toronto One Address Repository\", published under the [Open Government Licence – Toronto](https://open.toronto.ca/open-data-licence/). Consumed indirectly via the sibling [`toronto-addresses-import`](https://github.com/skfd/toronto-addresses-import) project.\n- **OpenStreetMap** — © OpenStreetMap contributors, [ODbL 1.0](https://www.openstreetmap.org/copyright). All uploads target the OSM **dev sandbox** (`master.apis.dev.openstreetmap.org`); any future production import must separately comply with the OSMF [import guidelines](https://wiki.openstreetmap.org/wiki/Import/Guidelines) and [contributor terms](https://osmfoundation.org/wiki/Licence/Contributor_Terms).\n- **Geofabrik** — Ontario `.osm.pbf` extracts, redistributed under ODbL from OSM.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskfd%2Ftoronto-2-address-import","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fskfd%2Ftoronto-2-address-import","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fskfd%2Ftoronto-2-address-import/lists"}