{"id":23483343,"url":"https://github.com/amirespahbodi/google-maps-scraper","last_synced_at":"2026-05-02T06:33:50.634Z","repository":{"id":264107108,"uuid":"889386556","full_name":"AmirEspahbodi/google-maps-scraper","owner":"AmirEspahbodi","description":"google map scraper. extract title, phone, address, latitude and longitude, category, website URL, rating, reviews number, email, active_hours, reviews and first picture of listing","archived":false,"fork":false,"pushed_at":"2024-12-23T02:36:12.000Z","size":245,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-23T03:24:38.629Z","etag":null,"topics":["dynamic-website","google-map-scraper","google-map-scraping","google-maps-scraper","google-maps-scraper-python","google-maps-scraping","playwright","playwright-python","python3","web-crawling","web-scraping","webcrawling","webscraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/AmirEspahbodi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-16T08:39:46.000Z","updated_at":"2024-12-23T02:36:16.000Z","dependencies_parsed_at":null,"dependency_job_id":"698ec490-3d79-4004-97c1-73e3083c6b6e","html_url":"https://github.com/AmirEspahbodi/google-maps-scraper","commit_stats":null,"previous_names":["amirespahbodi/google-map-scraper","amirespahbodi/google-maps-scraper"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirEspahbodi%2Fgoogle-maps-scraper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirEspahbodi%2Fgoogle-maps-scraper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirEspahbodi%2Fgoogle-maps-scraper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/AmirEspahbodi%2Fgoogle-maps-scraper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/AmirEspahbodi","download_url":"https://codeload.github.com/AmirEspahbodi/google-maps-scraper/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239109050,"owners_count":19583112,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dynamic-website","google-map-scraper","google-map-scraping","google-maps-scraper","google-maps-scraper-python","google-maps-scraping","playwright","playwright-python","python3","web-crawling","web-scraping","webcrawling","webscraping"],"created_at":"2024-12-24T21:11:21.660Z","updated_at":"2026-05-02T06:33:50.628Z","avatar_url":"https://github.com/AmirEspahbodi.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Google Maps Scraper\n\n## Overview\nThis repository contains a Google Maps scraping system composed of two cooperating services:\n\n- **Scraper worker**: Uses Playwright to open multiple browser contexts/tabs, search Google Maps, scroll listings, extract details, and persist results to Excel (and optionally download listing images). The worker pulls work items from a Redis queue. \n- **API server**: A FastAPI service that accepts scrape requests, validates duplicates against Redis queues and stored output files, and exposes a status endpoint to inspect current queue and processing state.\n\nA helper script (`run_app.py`) runs both services concurrently using a thread pool.\n\n## Key Features\n- Asynchronous Playwright-driven scraping with multiple browser tabs/contexts.\n- Redis-backed queue for search requests and in-process tracking.\n- Excel output via Pandas + OpenPyXL.\n- Optional image download and storage.\n- FastAPI endpoints for requesting scrapes and observing system status.\n\n## Tech Stack\n- **Python 3.12** (Poetry-managed dependencies). \n- **Playwright** for browser automation. \n- **FastAPI** + **Uvicorn** for the API server. \n- **Redis** for queueing and coordination. \n- **Pydantic Settings** for environment configuration. \n- **lxml** for HTML parsing. \n- **Pandas / OpenPyXL** for Excel output. \n- **aiofiles / httpx** for async file and HTTP operations.\n\n## Architecture \u0026 Data Flow\n1. The API server receives a scrape request (listing category + query details) and enqueues it in Redis. \n2. The scraper worker reads queued requests, launches Playwright browsers/tabs, navigates to Google Maps, and performs search + scrolling. \n3. Listings are extracted, deduplicated across tabs, and enriched with details (title, category, address, phone, website, hours, coordinates, images). \n4. Results are written to Excel files in the configured storage directory.\n\n## Design Patterns \u0026 Structure\n- **BO/DAO/DTO layering**:\n  - `data/bo`: Business logic orchestration for search, scrolling, extraction, and multi-tab coordination.\n  - `data/dao`: Infrastructure wrappers (Redis access, Playwright page wrapper).\n  - `data/dto`: Request/response schemas for the API.\n- **Singleton RuntimeResource**: Centralized Playwright lifecycle management for browsers and tabs.\n- **Async concurrency**: Heavy use of `asyncio.gather` for multi-tab scraping and page interactions.\n\n## Project Structure\n```\n.\n├── run_app.py                # Runs scraper + API server together\n├── scraper/\n│   ├── run_scraper.py         # Entry point for the scraper worker\n│   ├── processes/             # Scraper processes\n│   ├── data/                  # bo/dao/dto + static data (user agents, referers)\n│   ├── config/                # Pydantic settings for scraper + Redis\n│   └── utils/                 # Helper utilities (Excel, image storage, singleton)\n└── server/\n    ├── main.py                # API server entry point\n    ├── apis/                  # FastAPI routes + schemas\n    ├── core/                  # Server configuration + Redis connection\n    └── utils/                 # File-based request status helpers\n```\n\n## Requirements\n- Python 3.12\n- Redis (local or remote)\n- Playwright browsers installed (Chromium/WebKit)\n\n## Setup\n1. **Install dependencies**\n   ```bash\n   poetry install\n   ```\n2. **Install Playwright browsers**\n   ```bash\n   poetry run playwright install\n   ```\n3. **Configure environment**\n   ```bash\n   cp .env.sample .env\n   ```\n   Update values as needed (Redis host/port, storage directories, separators, etc.).\n\n## Running the System\n### Run both services together\n```bash\npoetry run python run_app.py\n```\n\n### Run scraper only\n```bash\ncd scraper\npoetry run python run_scraper.py\n```\n\n### Run API server only\n```bash\ncd server\npoetry run python main.py\n```\n\n## API Endpoints\n### `POST /request`\nQueues a new scrape request.\n\n**Request body**\n```json\n{\n  \"listing_category\": \"restaurant\",\n  \"listing_type\": \"restaurant\",\n  \"verb\": \"near\",\n  \"city\": \"Berlin\",\n  \"province\": \"Berlin\"\n}\n```\n\n**Response**\nReturns lists of imported, not-imported, in-process, and waiting requests.\n\n### `GET /status`\nReturns current status:\n- `imported_requests` and `not_imported_requests` (based on storage files)\n- `in_process` (current Redis processing item)\n- `waiting_to_scrape` (queue contents)\n\n## Output \u0026 Storage\n- **Excel files**: Written to `NOT_IMPORTED_SHEETS_DIRECTORY` with a filename derived from the query fields and separators.\n- **Images**: Saved under `PICTURES_DIRECTORY` (if image URLs are found).\n\n## Configuration Reference\nEnvironment variables are defined in `.env.sample` and loaded via Pydantic Settings. Key variables include:\n- `REDIS_HOST`, `REDIS_PORT`\n- `REDIS_REQUESTED_SEARCH_QUERY_QUEUE_NAME`, `REDIS_IN_PROCESSING_SEARCH_QUERY`\n- `SEARCH_QUERY_ITEMS_SEPARATOR`, `LISTING_TYPE_ITEMS_SEPARATOR`\n- `PICTURES_DIRECTORY`, `NOT_IMPORTED_SHEETS_DIRECTORY`, `IMPORTED_SHEETS_DIRECTORY`\n- API server settings: `APP_NAME`, `HOST`, `PORT`, `ENVIRONMENT`, `WORKER`\n\n## Notes \u0026 Considerations\n- Scraping Google Maps may violate Google’s terms of service. Use responsibly and ensure compliance with local laws and platform policies.\n- Playwright is configured to run in non-headless mode by default in `RuntimeResource`.\n- Redis queues use simple list operations; scaling or fault tolerance might require additional coordination logic.\n\n## License\nThis project is licensed under the terms of the MIT license. See [LICENSE](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famirespahbodi%2Fgoogle-maps-scraper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Famirespahbodi%2Fgoogle-maps-scraper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Famirespahbodi%2Fgoogle-maps-scraper/lists"}