{"id":26873626,"url":"https://github.com/noanchovies/base-scraper-py---deprecated-use-scraper-engine-instead--","last_synced_at":"2025-03-31T09:19:41.740Z","repository":{"id":285305471,"uuid":"957650126","full_name":"noanchovies/base-scraper-py---deprecated-use-scraper-engine-instead--","owner":"noanchovies","description":"(AI Coder Ready) Boilerplate scraper for projects using selenium, beautifulsoup, pandas","archived":false,"fork":false,"pushed_at":"2025-03-31T03:06:19.000Z","size":29,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-31T04:19:41.976Z","etag":null,"topics":["aicodeassistant","aicodehelper","chatgpt","copilot","deepseek","gemini","gemini-ai","python","scraper","scraping","scraping-python","template-project","templates-and-utilities"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/noanchovies.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-03-30T21:37:58.000Z","updated_at":"2025-03-31T03:06:22.000Z","dependencies_parsed_at":"2025-03-31T00:29:02.011Z","dependency_job_id":null,"html_url":"https://github.com/noanchovies/base-scraper-py---deprecated-use-scraper-engine-instead--","commit_stats":null,"previous_names":["noanchovies/base-scraper-py","noanchovies/base-scraper-py---deprecated-use-scraper-engine-instead--"],"tags_count":0,"template":true,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/noanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/noanchovies","download_url":"https://codeload.github.com/noanchovies/base-scraper-py---deprecated-use-scraper-engine-instead--/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246443534,"owners_count":20778252,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["aicodeassistant","aicodehelper","chatgpt","copilot","deepseek","gemini","gemini-ai","python","scraper","scraping","scraping-python","template-project","templates-and-utilities"],"created_at":"2025-03-31T09:19:41.187Z","updated_at":"2025-03-31T09:19:41.730Z","avatar_url":"https://github.com/noanchovies.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"##---Use https://github.com/noanchovies/scraper-engine instead ---\n\n---This version was cool but not good enough. scraper-engine is faster, plug\u0026play, \"more better\" --- \n\n_Keeping old version for log and learning purposes_ \n\n# Base Scraper Template (Python/Selenium)\n\nA generic base template project for building web scrapers using Python, Selenium, BeautifulSoup, and Typer. Designed to be easily copied and adapted for various scraping targets.\n\n**Note:** For quickly briefing AI assistants (like Google Gemini) on this template's structure and how to adapt it, refer to the `AI_CONTEXT.txt` file in the project root. It includes a summary and an adaptation checklist for next steps.\n\n## Key Features\n\n* **Selenium WebDriver:** Uses Selenium with `webdriver-manager` for automated browser control, capable of handling dynamic JavaScript-heavy websites. Configurable headless mode.\n* **HTML Parsing:** Integrates BeautifulSoup for parsing HTML structure obtained via Selenium.\n* **Configurable:** Easily configure target URLs, output filenames, wait times, and headless mode via `src/basescraper/config.py`, `.env` files, or command-line arguments.\n* **CLI Interface:** Uses Typer to provide a clean command-line interface for running the scraper.\n* **Modular Structure:** Separates concerns into configuration (`config.py`), core scraping logic (`scraper.py`), and CLI (`cli.py`) within a standard `src` layout.\n* **Placeholder Implementation:** Core data extraction (`extract_data`) and data handling (`handle_data`) functions are provided as clear placeholders (`NotImplementedError`) that **must** be implemented for each specific scraping project.\n* **Structured Output (Example):** Includes an optional pattern for saving data to CSV using Pandas (`save_to_csv` function commented out within `handle_data` placeholder).\n\n## Technology Stack\n\n* **Language:** Python 3.8+\n* **Browser Automation:** Selenium\n* **Driver Management:** webdriver-manager\n* **HTML Parsing:** BeautifulSoup4\n* **Data Handling (Example):** Pandas (for CSV saving pattern)\n* **CLI:** Typer, Rich\n* **Configuration:** python-dotenv\n* **Packaging:** setuptools, pyproject.toml\n\n## Setup (For Using the Template)\n\n1.  **Copy Template:** Create a new project by copying this entire `base-scraper-py` directory.\n2.  **Navigate:** `cd` into your new project directory.\n3.  **Create Virtual Environment:** `python -m venv venv`\n4.  **Activate Environment:**\n    * Windows: `.\\venv\\Scripts\\activate`\n    * macOS/Linux: `source venv/bin/activate`\n5.  **Install Dependencies:** `pip install -r requirements.txt`\n6.  **(Optional) Git Init:** If desired, delete the copied `.git` folder, run `git init`, create a new remote repository, and link it (`git remote add origin \u003curl\u003e`).\n\n## Usage\n\n1.  **Implement Logic:** Follow the detailed steps in `HOW_TO_USE_BASE.txt` to implement the required `extract_data` and `handle_data` functions within `src/basescraper/scraper.py` for your specific target website.\n2.  **Configure:** Set your target URL and other parameters in `.env` or `src/basescraper/config.py`.\n3.  **Run from CLI:**\n    * Option A (Run as module): `python -m src.basescraper.cli run [OPTIONS]`\n    * Option B (If installed editable `pip install -e .`): `basescraper run [OPTIONS]`\n4.  **CLI Options:** Use `--help` to see available options:\n    ```bash\n    python -m src.basescraper.cli run --help\n    # or\n    basescraper run --help\n    ```\n    Example: `basescraper run --url \"your-target-url.com\" -o \"my_output.csv\" --no-headless`\n\n## Adapting the Base\n\nThe core adaptation steps are detailed in `HOW_TO_USE_BASE.txt`. Primarily involves implementing:\n\n* `extract_data(page_source)`: Add logic using BeautifulSoup selectors to parse the HTML (`page_source`) from your target site and return a list of dictionaries.\n* `handle_data(data, output_target)`: Add logic to process the list of dictionaries returned by `extract_data` (e.g., save to CSV, database, API).\n\n## License\n\nMIT License (Update `LICENSE` file and `pyproject.toml` if using a different license).\n\n## Contributing / Contact\n\n(Add details here if you plan for others to contribute or how to contact you).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnoanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnoanchovies%2Fbase-scraper-py---deprecated-use-scraper-engine-instead--/lists"}