{"id":28576861,"url":"https://github.com/billy-enrizky/yelpfusion","last_synced_at":"2026-04-28T17:05:03.818Z","repository":{"id":296922449,"uuid":"995023544","full_name":"billy-enrizky/YelpFusion","owner":"billy-enrizky","description":"Finding All restaurants in the Maryland area using YelpFusion API","archived":false,"fork":false,"pushed_at":"2025-06-02T22:21:12.000Z","size":10633,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-06-11T00:07:36.184Z","etag":null,"topics":["data-analysis","pandas","yelp-api","yelpfusion"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/billy-enrizky.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-06-02T21:08:45.000Z","updated_at":"2025-06-03T00:34:24.000Z","dependencies_parsed_at":"2025-06-03T11:41:21.047Z","dependency_job_id":"f71eb950-fe34-454d-97af-0fcf28b08f89","html_url":"https://github.com/billy-enrizky/YelpFusion","commit_stats":null,"previous_names":["billy-enrizky/yelpfusion"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/billy-enrizky/YelpFusion","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billy-enrizky%2FYelpFusion","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billy-enrizky%2FYelpFusion/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billy-enrizky%2FYelpFusion/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billy-enrizky%2FYelpFusion/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/billy-enrizky","download_url":"https://codeload.github.com/billy-enrizky/YelpFusion/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/billy-enrizky%2FYelpFusion/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32390125,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-28T14:34:11.604Z","status":"ssl_error","status_checked_at":"2026-04-28T14:32:37.009Z","response_time":56,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analysis","pandas","yelp-api","yelpfusion"],"created_at":"2025-06-11T00:07:41.061Z","updated_at":"2026-04-28T17:05:03.787Z","avatar_url":"https://github.com/billy-enrizky.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# YelpFusion — Maryland Restaurant Data Collector\n\n`yelpfusion.py` divides the State of Maryland into a 10 × 10 geographic grid and calls the **Yelp Fusion API** to catalogue every restaurant found in each cell.\nThe script writes duplicate-free CSVs, checkpoints its own progress so you can stop and resume at any time, and generates a high-resolution PNG/JPG map of the grid with search radii overlaid.\n\n---\n\n## Table of Contents\n\n1. [Key Features](#key-features)\n2. [Architecture \u0026 Data Flow](#architecture--data-flow)\n3. [Quick Start](#quick-start)\n4. [How It Works](#how-it-works)\n5. [Repository Layout](#repository-layout)\n6. [Configuration](#configuration)\n7. [Troubleshooting](#troubleshooting)\n8. [Contributing](#contributing)\n9. [License](#license)\n\n---\n\n## Key Features\n\n| Capability                 | Details                                                                                                                   |\n| -------------------------- | ------------------------------------------------------------------------------------------------------------------------- |\n| **10 × 10 grid**           | Covers 100 % of Maryland with cells ≈ 20 km × 40 km each.                                                                 |\n| **Adaptive search radius** | Calculates the minimal radius (≤ 40 000 m, Yelp hard-limit) that encloses the cell’s diagonal plus a 20 % overlap margin. |\n| **Stateful scraping**      | `maryland_grid_cells.csv` stores a `done` flag per cell, so an interrupted run resumes where it left off.                 |\n| **Duplicate-free CSV**     | Each Yelp `business_id` is written exactly once, even if it appears in multiple cells.                                    |\n| **Rate-limit defence**     | Detects HTTP 429, persists everything gathered so far, marks the current cell **not** done, and exits cleanly.            |\n| **Cartographic output**    | Produces `maryland_grid_visualization.{png,jpg}` with OpenStreetMap tiles (via `contextily`).                             |\n| **Automatic backups**      | Writes JSON snapshots every five cells and again when the last cell finishes.                                             |\n\n---\n\n## Architecture \u0026 Data Flow\n\n```mermaid\nflowchart TD\n    A[Start] --\u003e B[Load .env\u003cbr/\u003eYelp API key]\n    B --\u003e C{maryland_grid_cells.csv exists?}\n    C -- No --\u003e D[Generate 10×10 grid\u003cbr/\u003esave CSV]\n    C -- Yes --\u003e E[Load grid\u003cbr/\u003ewith done flags]\n    D --\u003e F\n    E --\u003e F\n    F[Render grid\u003cbr/\u003ePNG/JPG] --\u003e G[Load / init\u003cbr/\u003emaryland_restaurants.csv]\n    G --\u003e H[Iterate over grid cells]\n    H --\u003e I{Cell marked done?}\n    I -- Yes --\u003e H\n    I -- No --\u003e J[Compute center \u0026 radius]\n    J --\u003e K[\"Call /businesses/search\u003cbr/\u003e(paginated)\"]\n    K --\u003e L[\"Call /businesses/:id\u003cbr/\u003efull details\"]\n    L --\u003e M[Append NEW rows\u003cbr/\u003eto CSV]\n    M --\u003e N[Mark cell done\u003cbr/\u003eupdate CSV]\n    N --\u003e H\n    H --\u003e|every 5 cells| P[Write progress JSON]\n    H --\u003e|after last cell| Q[Write final JSON backup]\n    Q --\u003e R[Finish]\n```\n\n---\n\n## Quick Start\n\n```bash\n# 1 — Clone the repo\ngit clone https://github.com/billy-enrizky/YelpFusion.git\ncd YelpFusion\n\n# 2 — (Recommended) create a virtual environment\npython -m venv .venv\nsource .venv/bin/activate          # Windows: .venv\\Scripts\\activate\n\n# 3 — Install dependencies\npip install -r requirements.txt\n\n# 4 — Provide your Yelp key (https://developer.yelp.com/)\necho \"YELP_API_KEY=YOUR_KEY_HERE\" \u003e .env\n\n# 5 — Run the scraper\npython yelpfusion.py\n```\n\n\u003e **Tip:** The scraper is idempotent.\n\u003e If the process stops (power loss, 429, etc.), just run it again—completed cells are skipped automatically.\n\n---\n\n## How It Works\n\n| Stage                   | Technical Notes                                                                                                                             |\n| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |\n| **Grid generation**     | Bounding box: lat 37.9171 → 39.72284, lon −79.4870 → −75.0506. Cell metadata (corners, centre, search radius) stored to CSV.                |\n| **Radius calculation**  | Half-height \u0026 half-width in metres (`Δφ · 111 000`, `Δλ · 111 000 · cos φ`); diagonal × 1.2; capped at 40 000 m.                            |\n| **API pagination**      | `/businesses/search` returns ≤ 50 rows; script increments `offset` (0, 50, 100, …) up to 1 000 rows per cell.                               |\n| **Deduplication**       | A `set` of known Yelp IDs is loaded at startup and updated in-memory before each CSV append.                                                |\n| **Rate-limit handling** | On HTTP 429: writes progress JSON, marks cell “no”, exits (`sys.exit(1)`).                                                                  |\n| **Mapping**             | If `contextily` + `geopandas` are present, EPSG:4326 tiles from OpenStreetMap Mapnik are added; otherwise a plain matplotlib grid is drawn. |\n| **Checkpoint cadence**  | `maryland_restaurants_progress.json` every 5 cells; `maryland_restaurants_json_backup.json` at completion.                                  |\n\n---\n\n## Repository Layout\n\n| Path                                   | Purpose                                                 |\n| -------------------------------------- | ------------------------------------------------------- |\n| `yelpfusion.py`                        | Main orchestration script (grid, scrape, visualise).    |\n| `requirements.txt`                     | Version-pinned Python dependencies.                     |\n| `maryland_grid_cells.csv`              | Grid definition + `done` status (generated at runtime). |\n| `maryland_restaurants.csv`             | Master list of deduplicated restaurants (generated).    |\n| `maryland_grid_visualization.png/.jpg` | High-resolution map of the search grid (generated).     |\n| `*_progress.json`, `*_backup.json`     | Incremental / final Yelp snapshots (generated).         |\n\n---\n\n## Configuration\n\n### Mandatory environment variable\n\n| Variable       | Description                                          |\n| -------------- | ---------------------------------------------------- |\n| `YELP_API_KEY` | Your personal Yelp Fusion API key (store in `.env`). |\n\n### Optional constants (edit near the top of **`yelpfusion.py`**)\n\n| Constant                         | Default                    | Meaning                                |\n| -------------------------------- | -------------------------- | -------------------------------------- |\n| `GRID_ROWS / GRID_COLS`          | `10 / 10`                  | Grid granularity.                      |\n| `MD_NORTH / SOUTH / EAST / WEST` | Hard-coded Maryland bounds | Change these to scrape another region. |\n| `timeout_s`                      | `5.0`                      | Yelp client timeout (seconds).         |\n\n---\n\n## Troubleshooting\n\n| Symptom                   | Likely Cause                                | Fix                                                             |\n| ------------------------- | ------------------------------------------- | --------------------------------------------------------------- |\n| **401 Unauthorized**      | `YELP_API_KEY` missing or invalid.          | Verify key in `.env`.                                           |\n| **429 Too Many Requests** | Daily quota or QPS exceeded.                | Wait for quota reset (midnight UTC) and rerun; scraper resumes. |\n| **No map tiles**          | `contextily`/`geopandas` not installed.     | `pip install contextily geopandas`.                             |\n| **Duplicate rows**        | Manual edits to `maryland_restaurants.csv`. | Remove duplicates or regenerate the CSV.                        |\n\n---\n\n## Contributing\n\n1. Open an **Issue** describing the bug or feature.\n2. Fork \u0026 create a branch: `git checkout -b feature/your-idea`.\n3. Format code with **black** and run **flake8**.\n4. Submit a concise, well-scoped **Pull Request**.\n\n---\n\n## License\n\nReleased under the **MIT License**.\nUse of the Yelp Fusion API is governed by Yelp’s Developer Terms.\n\n---\n\n### Acknowledgements\n\n* **Yelp Fusion API** for public restaurant data.\n* **OpenStreetMap** contributors for map tiles served via `contextily`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbilly-enrizky%2Fyelpfusion","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbilly-enrizky%2Fyelpfusion","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbilly-enrizky%2Fyelpfusion/lists"}