{"id":29823909,"url":"https://github.com/victorcezeh/understanding-web-scraping","last_synced_at":"2025-11-09T11:03:14.152Z","repository":{"id":307008216,"uuid":"1027955291","full_name":"victorcezeh/understanding-web-scraping","owner":"victorcezeh","description":"Python web scraping projects using requests and BeautifulSoup.","archived":false,"fork":false,"pushed_at":"2025-07-28T22:48:19.000Z","size":43,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"main","last_synced_at":"2025-07-29T00:25:25.174Z","etag":null,"topics":["beautifulsoup","git","github","python","requests"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/victorcezeh.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-07-28T19:41:47.000Z","updated_at":"2025-07-28T22:48:22.000Z","dependencies_parsed_at":"2025-07-29T00:25:29.379Z","dependency_job_id":"08687575-7660-4321-99af-0cd248591958","html_url":"https://github.com/victorcezeh/understanding-web-scraping","commit_stats":null,"previous_names":["victorcezeh/understanding-web-scraping"],"tags_count":null,"template":false,"template_full_name":null,"purl":"pkg:github/victorcezeh/understanding-web-scraping","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Funderstanding-web-scraping","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Funderstanding-web-scraping/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Funderstanding-web-scraping/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Funderstanding-web-scraping/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/victorcezeh","download_url":"https://codeload.github.com/victorcezeh/understanding-web-scraping/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/victorcezeh%2Funderstanding-web-scraping/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":283496071,"owners_count":26845317,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-11-09T02:00:05.828Z","response_time":62,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["beautifulsoup","git","github","python","requests"],"created_at":"2025-07-29T02:09:40.086Z","updated_at":"2025-11-09T11:03:14.135Z","avatar_url":"https://github.com/victorcezeh.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Web Scraping Projects\n\nA collection of Python web scraping projects demonstrating various techniques using `requests`, `BeautifulSoup`, and `lxml`. These projects showcase fundamental web scraping concepts from basic single-page extraction to multi-page data collection.\n\n📖 Table of Contents\n\n- [Technologies Used](#️-technologies-used)\n- [Project Structure](#-project-structure)\n- [Projects Overview](#-projects-overview)\n- [Key Learning Concepts](#-key-learning-concepts)\n- [Getting Started](#-getting-started)\n- [Usage Examples](#-usage-examples)\n- [Skills Demonstrated](#-skills-demonstrated)\n- [Target Websites](#-target-websites)\n- [Notes](#-notes)\n- [Ethical Considerations](#-ethical-considerations)\n\n\n## 🛠️ Technologies Used\n\n- **Python 3.x**\n- **requests** - For making HTTP requests\n- **BeautifulSoup4** - For parsing HTML content\n- **lxml** - As the XML/HTML parser engine\n\n## 📁 Project Structure\n\n```\nunderstanding-web-scraping/\n├── practice-src-code/\n│   ├── authors_scraper.py\n│   ├── basic_page_scraper.py\n│   ├── book_ratings_scraper.py\n│   ├── multi_page_authors_scraper.py\n│   ├── my_computer_image.jpg\n│   ├── quotes_scraper.py\n│   ├── top_tags_scraper.py\n│   ├── wikipedia_elements_scraper.py\n│   └── wikipedia_image_downloader.py\n├── .gitignore\n├── README.md\n└── requirements.txt\n```\n\n## 🚀 Projects Overview\n\n### 1. Book Ratings Scraper (`book_ratings_scraper.py`)\n**Target Site:** [books.toscrape.com](http://books.toscrape.com)\n**Objective:** Extract titles of all books with 2-star ratings across multiple pages (1-50)\n- Demonstrates multi-page scraping\n- CSS class-based element selection\n- List data collection\n\n### 2. Wikipedia Elements Scraper (`wikipedia_elements_scraper.py`)\n**Target Site:** [Wikipedia - Grace Hopper](https://en.wikipedia.org/wiki/Grace_Hopper)  \n**Objective:** Extract specific elements from table of contents\n- Shows targeted element selection\n- Working with Wikipedia's CSS structure\n- Text extraction techniques\n\n### 3. Basic Page Scraper (`basic_page_scraper.py`)\n**Target Site:** [example.com](https://example.com)\n**Objective:** Extract page title and paragraph content\n- Fundamental scraping concepts\n- Basic HTML element selection\n- Data type exploration\n\n### 4. Wikipedia Image Downloader (`wikipedia_image_downloader.py`)\n**Target Site:** [Wikipedia - Deep Blue (chess computer)](https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)) \n**Objective:** Extract and download images from Wikipedia pages\n- Image URL extraction\n- Binary file downloading\n- File I/O operations\n\n### 5. Quotes Scraper (`quotes_scraper.py`)\n**Target Site:** [quotes.toscrape.com](http://quotes.toscrape.com)\n**Objective:** Collect all quotes from the first page\n- CSS class selection\n- Text content extraction\n- List building\n\n### 6. Authors Scraper (`authors_scraper.py`)\n**Target Site:** [quotes.toscrape.com](http://quotes.toscrape.com)  \n**Objective:** Extract unique author names from the first page\n- Set usage for duplicate removal\n- Author name collection\n- Unique data handling\n\n### 7. Top Tags Scraper (`top_tags_scraper.py`)\n**Target Site:** [quotes.toscrape.com](http://quotes.toscrape.com)  \n**Objective:** Extract the top ten tags from the homepage sidebar\n- Sidebar content scraping\n- Text cleaning with `.strip()`\n- Popular content extraction\n\n### 8. Multi-Page Authors Scraper (`multi_page_authors_scraper.py`)\n**Target Site:** [quotes.toscrape.com](http://quotes.toscrape.com)  \n**Objective:** Extract all unique authors from all pages using two different approaches\n- **Method 1:** Fixed range (when you know the total pages)\n- **Method 2:** Dynamic detection (robust approach for unknown page count)\n- Advanced pagination handling\n- End-of-content detection\n\n## 🎯 Key Learning Concepts\n\n### Web Scraping Fundamentals\n- Making HTTP requests with `requests.get()`\n- Parsing HTML with `BeautifulSoup`\n- CSS selector usage for element targeting\n- Text extraction and cleaning\n\n### Data Collection Strategies\n- **Lists** - For ordered data collection\n- **Sets** - For unique data collection (automatic duplicate removal)\n- **File I/O** - For saving downloaded content\n\n### Pagination Techniques\n- **URL templating** with `.format()` for dynamic page URLs\n- **Fixed range pagination** for known page counts\n- **Dynamic pagination** with condition checking for unknown page counts\n- **End detection** using content-based signals\n\n## 🚀 Getting Started\n\n### Prerequisites\n```bash\npip install requests beautifulsoup4 lxml\n```\n\n### Running the Scripts\nEach script can be run independently:\n```bash\npython book_ratings_scraper.py\npython quotes_scraper.py\npython multi_page_authors_scraper.py\n# ... etc\n```\n\n## 📋 Usage Examples\n\n### Basic Single Page Scraping\n```python\nimport requests\nimport bs4\n\nurl = \"https://example.com\"\nresponse = requests.get(url)\nsoup = bs4.BeautifulSoup(response.text, \"lxml\")\ntitle = soup.select(\"title\")[0].get_text()\n```\n\n### Multi-Page Scraping with Known Page Count\n```python\nbase_url = \"https://site.com/page/{}/\"\ndata = []\n\nfor page in range(1, 11):  # Pages 1-10\n    url = base_url.format(page)\n    response = requests.get(url)\n    soup = bs4.BeautifulSoup(response.text, \"lxml\")\n    # Extract data...\n```\n\n### Dynamic Multi-Page Scraping\n```python\npage = 1\nwhile True:\n    url = base_url.format(page)\n    response = requests.get(url)\n    \n    if \"No content found!\" in response.text:\n        break\n    \n    # Process page...\n    page += 1\n```\n\n## 🎓 Skills Demonstrated\n\n- **HTTP Requests** - Fetching web page content\n- **HTML Parsing** - Extracting structured data from web pages\n- **CSS Selectors** - Targeting specific elements\n- **Data Structures** - Using lists and sets effectively\n- **File Operations** - Downloading and saving binary content\n- **Control Flow** - Implementing various looping strategies\n- **String Manipulation** - Cleaning and formatting extracted text\n- **Pagination Logic** - Handling multi-page data sources\n\n## 🔍 Target Websites\n\n- **books.toscrape.com** - Demo bookstore for scraping practice\n- **quotes.toscrape.com** - Demo quotes site for scraping practice\n- **Wikipedia** - Real-world content extraction\n- **example.com** - Basic HTML structure demonstration\n\n## 📝 Notes\n\n- All scripts include comprehensive comments and docstrings\n- Code follows clean, readable Python practices\n- Each project demonstrates different aspects of web scraping\n- Projects progress from basic to more advanced techniques\n- Suitable for learning and portfolio demonstration\n\n## 🚨 Ethical Considerations\n\n- Respect rate limits and server resources\n- Use scraped data responsibly\n- Consider the website's terms of service\n- These projects use demo/educational websites designed for scraping practice\n\n---\n\n*This repository showcases fundamental web scraping techniques and serves as a learning resource for Python web scraping with requests and BeautifulSoup.*\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorcezeh%2Funderstanding-web-scraping","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvictorcezeh%2Funderstanding-web-scraping","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvictorcezeh%2Funderstanding-web-scraping/lists"}