{"id":51763258,"url":"https://github.com/hadro/city-directory-extraction","last_synced_at":"2026-07-19T16:31:12.968Z","repository":{"id":367409338,"uuid":"1275680146","full_name":"hadro/city-directory-extraction","owner":"hadro","description":"Profiles of the various kinds of NYC city directories","archived":false,"fork":false,"pushed_at":"2026-07-19T02:58:43.000Z","size":1008,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-07-19T04:30:13.624Z","etag":null,"topics":["brooklyn","city-directories","nyc","open-data","small-model"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/hadro.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-06-21T02:28:44.000Z","updated_at":"2026-07-19T02:58:47.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/hadro/city-directory-extraction","commit_stats":null,"previous_names":["hadro/city-directory-extraction"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/hadro/city-directory-extraction","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fcity-directory-extraction","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fcity-directory-extraction/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fcity-directory-extraction/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fcity-directory-extraction/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/hadro","download_url":"https://codeload.github.com/hadro/city-directory-extraction/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hadro%2Fcity-directory-extraction/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35659409,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-19T02:00:06.923Z","response_time":112,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["brooklyn","city-directories","nyc","open-data","small-model"],"created_at":"2026-07-19T16:31:11.046Z","updated_at":"2026-07-19T16:31:12.699Z","avatar_url":"https://github.com/hadro.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# city-directory-extraction\n\nOne project, two halves, aimed at **extracting structured person records from historical US city\ndirectories** and releasing the results to the community:\n\n- **(A) The model** — fine-tune a small open model (Qwen3.5) to turn one OCR'd directory line\n  into a structured 8-field record: train on synthetic data, evaluate on real hand-labeled gold,\n  publish models + datasets on Hugging Face.\n- **(B) The data** — everything the model trains and is judged on: a 449-volume multi-institution\n  **catalog** of digitized NYC directories (~1786–1925), per-publisher×era **style profiles**,\n  a hand-labeled **real-OCR gold eval panel**, and the **synthetic training-data generator** that\n  the catalog work parameterizes. The catalog + profiles are also intended as a standalone\n  community reference resource in their own right.\n\nThese halves are on track to become **two repos** (working names: `city-directory-data` and\n`city-directory-model`). The [Layout](#layout--organized-for-the-future-split) below is grouped by\nthat seam, and [The future split](#the-future-split) documents the interface between them.\n\n\u003e This project spun out of the sibling [`directory-pipeline`](../directory-pipeline) repo, whose\n\u003e Gemini-based NER step (`pipeline/extract_entries.py`) the model aims to replace for the\n\u003e city-directory *persons* shape. The pipeline repo also hosts the page-sampler\n\u003e (`sources/sample_directories.py`) and column/spread detectors that workstream (B) relies on.\n\u003e\n\u003e **Current working state lives in the handoff docs, not in the original plan:**\n\u003e [docs/HANDOFF.md](docs/HANDOFF.md) (the model: results, gotchas, next steps),\n\u003e [docs/GROUND_TRUTH_HANDOFF.md](docs/GROUND_TRUTH_HANDOFF.md) (gold panel, labeling contract),\n\u003e [docs/VISUAL_SAMPLING_HANDOFF.md](docs/VISUAL_SAMPLING_HANDOFF.md) (catalog backfill), and\n\u003e [docs/FRONTMATTER_KEYPAGE_HANDOFF.md](docs/FRONTMATTER_KEYPAGE_HANDOFF.md) (key-page /\n\u003e listing-start / page-offset sampling). [docs/plan.md](docs/plan.md) is the original rationale\n\u003e and data landscape — read it for background, the handoffs for truth.\n\n## Where things stand (2026-07)\n\n**Model (A):** first full fine-tune trained, eval-loader bug fixed, and scored twice — on NYU gold\nand across the whole 18-volume gold panel.\n\n`hadro/city-dir-08b-yaml` (0.8B, 100k synthetic, 3 epochs, YAML) on NYU gold (500 rows):\n\n| model | macro-F1 | micro-F1 | whole-row EM |\n|---|---|---|---|\n| GLiNER zero-shot (floor) | 0.381 | 0.594 | 8% |\n| **qwen-0.8b-yaml (ours)** | **0.760** | 0.755 | 26% |\n| Gemini 3.1-flash-lite (bar) | 0.672 | **0.910** | **70%** |\n\n**Honest read:** we edge Gemini on macro-F1 (rare fields like spouse/race), but Gemini leads\nmicro-F1 and whole-row EM — the high-volume `name`/`address` fields and complete rows, which are\nwhat matter for replacing it in the pipeline.\n\nThe **first full-panel run** (all 18 gold volumes, 1786–1933) sharpened that into four systematic\ngaps, all *training-coverage* problems, not capacity problems (panel-wide field F1: name 0.52,\naddress 0.44, home_address 0.07, race 0.00):\n\n1. **Ditto marks** — the generator never emits surname-repeat dittos (`\"`/`-`), so the model strips\n   them; every ditto row scores 0 on `name` → EM=0% on all dense Polk/Trow volumes. Biggest lever.\n2. **`home_address`** — the two-address pattern is under-produced; the model collapses `h.` into\n   `address`.\n3. **Address styles** — 1930s hyphenated outer-borough house numbers (`24-12`, `LIC`/`JH`) and\n   1786-era formats are out-of-distribution.\n4. **`race_designation`** — never emitted (small n; clean synthetic fix).\n\n**Conclusion: stop adding breadth; fix synthetic coverage, retrain, re-score.** The 18-volume\npanel is now the regression harness. Full numbers and diagnosis in\n[docs/HANDOFF.md](docs/HANDOFF.md); table in [results/eval_table.md](results/eval_table.md).\n\n**Data (B):** `master_directories.csv` at **449 rows** (NYPL / IA / LoC; NYPL API responses\narchived before the 2026-08-01 deprecation). `column_count` backfilled for **332/449** — every\nin-scope residential volume. **17 publisher×era style cards** written. The gold panel stands at\n**18 volumes / 1,169 hand-labeled lines** (continuous 1786–1933, layout columns 1–6, all five\nboroughs, all 8 fields exercised) from a 41-volume worklist, all validator-clean. The\nfront-matter/key-page pass (listing `start_page`, abbreviations `key_page`, `page_offset`) is\n**25/41 done**; the remaining 16 need deep scans.\n\n**Next (in order):**\n1. Inject the four missing features into `synth_persons.py` (dittos first), regenerate, retrain\n   (see [Training options](#train--evaluate) — the measured cheap path is ~$6/run), re-score the\n   panel; confirm the `name`/EM lift.\n2. Finish the remaining 16 front-matter volumes; fold key-page legends into style profiles.\n3. Parameterize per-publisher styles in the generator (Wave 1), broaden the panel, scale the\n   family (0.8B/2B/4B, 500k), publish with cards.\n\n## Scope\n\nThe target is **one NYC-comprehensive model** that parses NYC directories ~**1786–1925** across\nall boroughs and publishers (Trow, Lain, Polk, Doggett, Upington, Spooner, Hearne, Longworth, …),\nwith **cross-city transfer measured as a stretch goal** (Tulsa 1921 and Minneapolis 1900 held\nout). Tulsa stays in the mix as a second trained dialect. Telephone directories (112 cataloged)\nand business/copartnership directories are out of scope — cataloged as future separate tracks.\n\n## The schema — the contract between the halves\n\nEverything flows through one **union schema (8 fields)**:\n\n`name · is_business · spouse_name · race_designation · occupation_role · employer · address · home_address`\n\nplus a serialization rule — **YAML, not pipe** (pipe is positional; a dropped field column-shifts\nthe rest; measured ~54% row breakage) — and a fixed **gold/synth/model labeling contract**\n(`raw_line` = verbatim page including OCR quirks; the 8 fields = canonical values; ~17 conventions\ncovering dittos, widows, race marks, parenthetical firms…). The conventions live in\n[docs/GROUND_TRUTH_HANDOFF.md](docs/GROUND_TRUTH_HANDOFF.md) and are what will keep two repos\nhonest with each other after the split.\n\n## Layout — organized for the future split\n\n```\n# ──────────────── future repo 1: city-directory-data ────────────────\n# gathering, profiling, gold labeling, training-data generation\ndata_prep/\n  # catalog (the 449-volume master list)\n  master_directories.csv        # multi-source (nypl|ia|loc|iiif) catalog; schema in its README\n  master_directories.README.md  # catalog schema + provenance log + leads\n  ingest_collection.py          # collection link -\u003e staged rows (review-then-append); nypl/ia/loc/iiif\n  nypl_api_archive/             # 155+ MODS JSONs (NYPL API deprecates 2026-08-01) — committed\n  # visual sampling + style profiles\n  inspect_frontmatter.py        # IIIF -\u003e cached front-matter pages + contact sheets (key-page pass)\n  trow_fanout_prep.py / trow_fanout.workflow.js  # gated cheap-tier fan-out for metadata backfill\n  style_profiles/               # 17 per-publisher×era cards (.md) + style_profiles.json\n  # real-OCR gold eval panel (hand-labeled; see GROUND_TRUTH_HANDOFF.md)\n  sample_volumes.py             # stratified selector -\u003e gold_sample/{worklist.csv,WORKLIST.md}\n  run_surya_on_samples.py       # batch Surya OCR over worklist dirs (listing-only; resumable)\n  make_gold_tool.py             # self-contained HTML labeling editor from Surya JSON\n  validate_gold.py              # QA: ERRORS (break evaluate.py) + WARNINGS (convention slips)\n  gold_sample/                  # 41-volume worklist + labeling checklist\n  # training data: synthetic generator + name pools\n  synth_persons.py              # (line -\u003e record) generator; --profile {tulsa,nyc,mix}\n  fetch_names.py                # build names/surnames.tsv (40k era-skewed census surnames)\n  harvest_names.py              # pipeline entries CSVs -\u003e harvested real-surname pool\n  names/surnames.tsv            # committed census pool (harvested pool is generated, gitignored)\n  # eval-set builders (external gold -\u003e union schema)\n  nyu_to_eval.py                # NYU NDJSON -\u003e held-out NYC gold (EVAL ONLY, CC-BY-SA-NC)\n  ftd_to_eval.py                # French Trade Directories -\u003e cross-lingual transfer eval\n  harvest_own.py                # pipeline output -\u003e in-domain eval (Tulsa + Lain; real OCR)\n  harvest_minneapolis.py        # Minneapolis 1900 (MIT) -\u003e union-schema SILVER eval\n\n# ──────────────── future repo 2: city-directory-model ────────────────\n# training, evaluation, publishing\ntrain/\n  sft_qwen.py             # TRL SFT; --target pipe|yaml, LoRA/--qlora/--full, --packing, --dry-run\n  sft_unsloth_smoke.py    # Unsloth speed probe (kept for the record; not worth it at 0.8B)\neval/\n  evaluate.py             # field-level P/R/F1/EM; --save/--label; --self-test\n  gliner_baseline.py      # zero-shot GLiNER extractive baseline (the floor)\n  gemini_baseline.py      # Gemini baseline (the bar); defaults --target yaml\n  qwen_predict.py         # fine-tuned Qwen -\u003e preds; loads the MULTIMODAL class (see gotchas)\n  results_table.py        # results/scores.jsonl -\u003e model × eval-set Markdown table\nnotebooks/colab_finetune.ipynb  # free-Colab T4 fine-tune -\u003e push to Hub\ncards/                    # MODEL_CARD.md + DATASET_CARD.md templates\nresults/                  # eval_table.md (tracked); scores.jsonl log (gitignored, regenerable)\n\n# ──────────────── shared ────────────────\ndata/                     # gitignored: downloads + generated sets (the interface artifacts)\ndocs/                     # handoffs (per-workstream), plan.md, BLOG_NOTES.md\n```\n\n## The future split\n\nThe seam is **datasets**: repo 1 *produces* JSONL datasets (synthetic train, gold eval panel,\nexternal eval sets), repo 2 *consumes* them and produces models + scores. They already exchange\nnothing else — locally via `data/*.jsonl`, remotely via the HF datasets\n(`hadro/city-directory-synth`, `hadro/cde-evals`).\n\nWhat has to stay in sync across the split (flag these in both READMEs when it happens):\n\n- **The 8-field schema + YAML serialization** — baked into `synth_persons.py` and\n  `validate_gold.py` (repo 1) and `evaluate.py` / the train-prompt in `sft_qwen.py` /\n  `gemini_baseline.py` (repo 2). Extract the field list + (de)serializers into one tiny shared\n  module (or versioned schema file) at split time.\n- **The labeling contract** (GROUND_TRUTH_HANDOFF conventions) — authored on the data side,\n  binding on the model side.\n- **Eval held-outs** — NYU Trow 1850/51 and Lain Brooklyn 1897 must stay out of repo 1's\n  sampling/harvesting (they're `REVIEW:`-flagged in the catalog).\n- The sibling `directory-pipeline` repo remains a dependency of repo 1 only (page sampler, OCR).\n\n## Quickstart (synthetic data)\n\n```bash\n# eyeball a sample (default profile = mix of both dialects)\npython3 data_prep/synth_persons.py --n 8 --preview\npython3 data_prep/synth_persons.py --n 8 --preview --profile nyc    # or --profile tulsa\n\n# generate a mixed training set\npython3 data_prep/synth_persons.py --n 100000 --out data/synth_train.jsonl --seed 13\n```\n\nEach JSONL row is `{raw_line, context:{dialect, alphabetical_range, directory_year},\nrecord:{…8 fields…}}`. `raw_line` carries optional OCR noise (the model input); `record` is the\nclean target; `context` is page-level metadata fed in the prompt rather than predicted. Names\ndraw from 40k census surnames + harvested real-name pools (the original inline ~54-surname list\nwas the documented root cause of the model regularizing unseen surnames).\n\n## Train \u0026 evaluate\n\nAll scripts are PEP-723 self-contained; each has `--self-test`, `--preview`, or `--dry-run`.\nTrain AND eval with the **same `--target`** (yaml) and the **same model class** (see gotchas in\n[docs/HANDOFF.md](docs/HANDOFF.md) — the eval-loader bug cost us several runs).\n\n```bash\n# 0) inspect the exact SFT examples (--preview-prompts is stdlib-only; --dry-run needs the\n#    ML deps, so run it under uv, and --train-file is required either way)\npython3 train/sft_qwen.py --train-file data/synth_train.jsonl --preview-prompts 4\nuv run train/sft_qwen.py --train-file data/synth_train.jsonl --dry-run   # verify: 0 visual adapters\n\n# 1) build eval sets (all map into the same 8-field schema)\npython3 data_prep/nyu_to_eval.py  --in data/1850.ndjson --out data/nyu_eval.jsonl --limit 3000\npython3 data_prep/ftd_to_eval.py  --in data/ftd.json    --out data/ftd_eval.jsonl\npython3 data_prep/harvest_own.py  --dir ../directory-pipeline/output/tulsa_1921 --out data/tulsa_eval.jsonl\npython3 data_prep/harvest_minneapolis.py --dir data/minneapolis/ground_truth --out data/minneapolis_eval.jsonl\n\n# 2) fine-tune. Cheapest measured cloud path: HF Jobs rtx-pro-6000, batch 64, --packing (~$6 for\n#    0.8B/100k×3). Free/cheap alternatives (Colab T4 notebook, Kaggle, local MLX): docs/TRAINING_OPTIONS.md\npython3 train/sft_qwen.py --train-file data/synth_train.jsonl --target yaml --packing \\\n    --model Qwen/Qwen3.5-0.8B --hub-model-id \u003cyou\u003e/city-directory-extractor-0.8b --push-to-hub\n\n# 3) baselines and the fine-tuned model -\u003e preds; score everything into one table\nuv run eval/gliner_baseline.py --gold data/nyu_eval.jsonl                       # the floor\nuv run eval/gemini_baseline.py --gold data/nyu_eval.jsonl --limit 500           # the bar (GEMINI_API_KEY)\nuv run eval/qwen_predict.py --base-model Qwen/Qwen3.5-0.8B --model \u003cyou\u003e/city-directory-extractor-0.8b \\\n    --gold data/nyu_eval.jsonl --target yaml                                    # ours (check: NO \"missing adapter keys\")\npython3 eval/evaluate.py --gold data/nyu_eval.jsonl --pred data/preds_qwen.txt --target yaml \\\n    --save results/scores.jsonl --label qwen-0.8b\npython3 eval/results_table.py --out results/eval_table.md\n```\n\n## The catalog \u0026 style profiles (workstream B)\n\n`data_prep/master_directories.csv` is a 449-row, multi-institution catalog of digitized NYC city\ndirectories, built by throwing collection **links** at `ingest_collection.py` (detects\n`nypl`/`ia`/`loc`/`iiif` sources, stages rows for review, appends on `--merge`). NYPL API\nresponses are archived under `nypl_api_archive/` because that API deprecates **2026-08-01**. The\nsibling `directory-pipeline/sources/sample_directories.py` resolves each row to a IIIF manifest\nand downloads **only** a few sampled pages per volume (never whole volumes).\n\nFrom those samples we build **style profiles** (`data_prep/style_profiles/`): 17 per-publisher×era\ncards capturing column count, the abbreviations legend (ground truth), entry format, and\npage-offset behavior. They backfill structural metadata in the catalog (`column_count` 332/449;\n`start_page`/`key_page`/`page_offset` 25/41 on the gold panel) **and** are the lever to\nparameterize `synth_persons.py` so synthetic lines match real layout and abbreviations.\n\nWorkflow, per-cohort logs, and gotchas (microfilm spreads, page-offset drift, phonebook-vs-city\ngenre, cheap-tier sub-agent delegation):\n[docs/VISUAL_SAMPLING_HANDOFF.md](docs/VISUAL_SAMPLING_HANDOFF.md) and\n[docs/FRONTMATTER_KEYPAGE_HANDOFF.md](docs/FRONTMATTER_KEYPAGE_HANDOFF.md).\n\n### Real-OCR gold eval panel\n\nA hand-labeled **gold eval panel** built from the cataloged volumes via a dedicated toolchain\n(`sample_volumes.py` → `run_surya_on_samples.py` → `make_gold_tool.py` → `validate_gold.py`):\nSurya-OCR a few listing pages per volume, label each entry into the 8-field schema in a browser\neditor, validate, drop the result into `data/\u003cslug\u003e_eval.jsonl` for `eval/evaluate.py`. It is\n**eval-only** and governed by the labeling contract. Status: **18 volumes / 1,169 lines**, all\nvalidator-clean. Conventions + per-volume log:\n[docs/GROUND_TRUTH_HANDOFF.md](docs/GROUND_TRUTH_HANDOFF.md).\n\n## Data sources\n\n| Source | Role | License |\n|---|---|---|\n| Synthetic (`synth_persons.py`) | **Training** | ours → permissive |\n| Real-OCR gold panel (18 vols, ours) | Eval / regression harness | ours |\n| [NYU NYC directories 1850–1890](https://archive.nyu.edu/handle/2451/61521) | Eval / benchmark | CC-BY-SA-**NC** ⚠ |\n| [French Trade Directories](https://zenodo.org/records/8167628) | Transfer eval | open (CC) |\n| [Minneapolis 1900 (DirCity)](https://github.com/adamrangwala/DirCity_Directory_Crop-out-with-Key-Lines) | In-domain US eval (silver) | MIT |\n| `../directory-pipeline/output` (Tulsa 1921, Lain Brooklyn 1897) | In-domain eval | ours |\n| `master_directories.csv` + `nypl_api_archive/` + `style_profiles/` | **Catalog (B)**; sampling source + standalone resource | NYPL/IA/LoC metadata; cards ours |\n\n⚠ NYU is **non-commercial**; it is used for *evaluation only* so the released, synthetic-trained\nmodel stays permissively reusable. Eval-held-out volumes (NYU Trow Manhattan 1850/51, Lain\nBrooklyn 1897) are kept OUT of the sampling/harvest set and `REVIEW:`-flagged in the catalog.\n\n## Hugging Face resources (namespace `hadro`)\n\n- `hadro/city-directory-synth` — synthetic train (100k) + smoke (3k). PUBLIC.\n- `hadro/cde-evals` — NYU + synth-dev eval sets + preds. PRIVATE (respects NYU CC-BY-SA-NC).\n- `hadro/city-dir-08b-yaml` — the good 0.8B run (see Status). Earlier runs documented in the handoff.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhadro%2Fcity-directory-extraction","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhadro%2Fcity-directory-extraction","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhadro%2Fcity-directory-extraction/lists"}