{"id":25030193,"url":"https://github.com/spike1236/aiofetch","last_synced_at":"2026-02-17T13:03:44.129Z","repository":{"id":274620074,"uuid":"923504283","full_name":"spike1236/aiofetch","owner":"spike1236","description":"Collection of different tools for async web scraping","archived":false,"fork":false,"pushed_at":"2025-02-07T14:15:38.000Z","size":56,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-12-22T01:27:24.980Z","etag":null,"topics":["python","scraping","web"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/spike1236.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.md","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2025-01-28T11:15:56.000Z","updated_at":"2025-07-08T13:01:15.000Z","dependencies_parsed_at":"2025-01-28T12:30:43.172Z","dependency_job_id":"e9ffeeee-222e-41ce-a0cd-50b2859cb411","html_url":"https://github.com/spike1236/aiofetch","commit_stats":null,"previous_names":["spike1236/asyncscrap","spike1236/aiofetch"],"tags_count":6,"template":false,"template_full_name":null,"purl":"pkg:github/spike1236/aiofetch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spike1236%2Faiofetch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spike1236%2Faiofetch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spike1236%2Faiofetch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spike1236%2Faiofetch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/spike1236","download_url":"https://codeload.github.com/spike1236/aiofetch/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/spike1236%2Faiofetch/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29545295,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-17T13:00:00.370Z","status":"ssl_error","status_checked_at":"2026-02-17T12:57:14.072Z","response_time":100,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["python","scraping","web"],"created_at":"2025-02-05T21:57:07.506Z","updated_at":"2026-02-17T13:03:44.112Z","avatar_url":"https://github.com/spike1236.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# aiofetch\n\nA Python toolkit for asynchronous web scraping with built-in error tracking and metadata management.\n\n## Features\n\n### Web Processing\n- Asynchronous file downloading with progress tracking\n- Rate limiting with configurable delays\n- Smart retry logic with timeout handling\n- Domain-aware crawling with URL validation\n\n### Content Processing\n- Flexible HTML content parsing\n- Custom selector-based metadata extraction\n- Automated link and image extraction\n- URL normalization and path handling\n\n### File \u0026 Data Management\n- Asynchronous file operations\n- Concurrent chunk-based downloads\n- Smart path handling and file naming\n- JSON data management with validation\n\n### Error Handling \u0026 Progress Tracking\n- Comprehensive error tracking and reporting\n- Progress monitoring for long operations\n- Detailed logging with configurable outputs\n- Operation statistics and summaries\n\n### Metadata Management\n- Efficient in-memory caching\n- Field-based search functionality\n- Automatic metadata indexing\n- Structured data validation\n\n## Installation\n\n```bash\npip install aiofetch\n```\n\n## Key Components\n\n- **AsyncDownloader**: Parallel file downloading with progress tracking\n- **BatchProcessor**: Process items in configurable batches\n- **RateLimiter**: Control request frequency\n- **MetadataExtractor**: HTML metadata extraction with custom selectors\n- **PathHandler**: Path and filename utilities\n- **FileIO**: Async/sync file operations\n- **BaseCrawler**: Extensible crawler base class with domain validation\n- **LoggerFactory**: Enhanced logging with file and console outputs\n\n## Requirements\n\n- Python 3.9+\n- aiofiles\n- aiohttp\n- BeautifulSoup4\n\n## Quick start\n\n- We are going to scrape images and books data from the example website - books.toscrape.com.\n\n```python3\nimport os\nimport asyncio\nfrom urllib.parse import urljoin\nfrom aiofetch.crawler import BaseCrawler, RateLimiter\nfrom aiofetch.utils import MetadataExtractor, FileIO\nfrom aiofetch.downloader import AsyncDownloader\n\n\nclass BookScraper(BaseCrawler):\n    def __init__(self, base_url: str):\n        super().__init__(base_url)\n        self.extractor = MetadataExtractor()\n        self.rate_limiter = RateLimiter()\n\n    async def scrape_page(self, url: str) -\u003e list:\n        async with self.rate_limiter:\n            content = await self.fetch_page(url)\n            if not content:\n                return []\n            self.logger.debug(f\"Parsing HTML content from {url}\")\n            soup = await self.parse_html(content)\n            books = []\n            selectors = {\n                'title': ('h3 a', 'title'),\n                'relative_link': ('h3 a', 'href'),\n                'price': 'p.price_color',\n                'availability': 'p.instock.availability',\n                'rating': ('p.star-rating', 'class', 1),\n                'image': ('div.image_container img', 'src')\n            }\n            for article in soup.select('article.product_pod'):\n                data = self.extractor.extract_from_html(article, selectors)\n                if rel := data.pop('relative_link', None):\n                    data['url'] = urljoin(url, rel)\n                else:\n                    data['url'] = url\n                if img := data.get('image'):\n                    data['image'] = urljoin(url, img)\n                books.append(data)\n        return books\n\n    async def scrape(self, start_url: str) -\u003e list:\n        return await self.scrape_page(start_url)\n\n\nasync def main():\n    # Scrape book data\n    async with BookScraper(\"http://books.toscrape.com\") as scraper:\n        books = await scraper.scrape(\"http://books.toscrape.com/catalogue/page-1.html\")\n    \n    # Save scraped data as JSON\n    file_io = FileIO()\n    json_path = \"data/books.json\"\n    await file_io.write_json(books, json_path)\n    print(f\"Saved {len(books)} books to {json_path}\")\n    \n    # Prepare and download images\n    download_tasks = []\n    for book in books:\n        if image_url := book.get('image'):\n            filename = os.path.basename(image_url)\n            local_path = os.path.join(\"images\", filename)\n            download_tasks.append((image_url, local_path))\n    \n    if download_tasks:\n        downloader = AsyncDownloader(concurrent_limit=10)\n        results = await downloader.download_batch(download_tasks)\n        print(f\"Downloaded {sum(results)} images out of {len(download_tasks)}\")\n        downloader.save_failed_downloads()\n\n\nif __name__ == \"__main__\":\n    asyncio.run(main())\n```\n\n## License\n\nMIT License - see LICENSE file for details\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request or Issue.\n\n## Author\n\nAkram Rakhmetulla ([akram042006@gmail.com](mailto:akram042006@gmail.com))","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspike1236%2Faiofetch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspike1236%2Faiofetch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspike1236%2Faiofetch/lists"}