{"id":22970496,"url":"https://github.com/afsh7n/crawly-automation","last_synced_at":"2026-02-25T11:07:00.107Z","repository":{"id":263506996,"uuid":"890626474","full_name":"afsh7n/crawly-automation","owner":"afsh7n","description":"Crawly Automation is a lightweight, modular, and extensible web crawling framework built on top of Puppeteer. Whether you need to scrape data, automate browser interactions, manage CAPTCHAs, or handle advanced data extraction, Crawly Automation simplifies the process.","archived":false,"fork":false,"pushed_at":"2024-11-19T17:48:43.000Z","size":68,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-10-28T10:01:59.857Z","etag":null,"topics":["automation","crawler","nodejs","puppeteer","webscraping"],"latest_commit_sha":null,"homepage":"https://www.npmjs.com/package/crawly-automation","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/afsh7n.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2024-11-18T22:39:28.000Z","updated_at":"2024-11-21T21:08:43.000Z","dependencies_parsed_at":"2024-11-18T23:43:05.719Z","dependency_job_id":"734663a7-c4c2-4429-81f0-c0acfc5d5849","html_url":"https://github.com/afsh7n/crawly-automation","commit_stats":null,"previous_names":["afsh7n/crawly"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/afsh7n/crawly-automation","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsh7n%2Fcrawly-automation","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsh7n%2Fcrawly-automation/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsh7n%2Fcrawly-automation/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsh7n%2Fcrawly-automation/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/afsh7n","download_url":"https://codeload.github.com/afsh7n/crawly-automation/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/afsh7n%2Fcrawly-automation/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":29819265,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-02-25T05:36:42.804Z","status":"ssl_error","status_checked_at":"2026-02-25T05:36:31.934Z","response_time":61,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","crawler","nodejs","puppeteer","webscraping"],"created_at":"2024-12-14T22:13:41.662Z","updated_at":"2026-02-25T11:07:00.090Z","avatar_url":"https://github.com/afsh7n.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crawly Automation\n\n**Crawly Automation** is a lightweight, modular, and extensible web crawling framework built on top of Puppeteer. Whether you need to scrape data, automate browser interactions, manage CAPTCHAs, or handle advanced data extraction, Crawly Automation simplifies the process.\n\n---\n\n## 🚀 Features\n\n- **Browser Management**: Launch, manage, and close browsers and pages with ease.\n- **Request Handling**: Intercept, filter, and modify network requests dynamically.\n- **Data Extraction**: Extract structured or unstructured data with customizable selectors.\n- **Data Formatting**: Convert extracted data to **JSON** or **CSV** with built-in utilities.\n- **CAPTCHA Handling**: Solve CAPTCHAs (reCAPTCHA) automatically using third-party services like 2Captcha.\n- **User Actions**: Simulate complex user interactions like single-page and multi-page logins, dropdown selection, file uploads, and more.\n- **Utilities**: Handy tools for delays, logging, and helper functions.\n\n---\n\n## 📦 Installation\n\nInstall Crawly Automation via NPM:\n\n```bash\nnpm install crawly-automation\n```\n\n---\n\n## 🔧 Usage\n\nHere’s a quick guide to get you started with Crawly Automation.\n\n---\n\n### **1. Browser Management**\n\nLaunch a browser, create a page, and navigate to a URL:\n\n```typescript\nimport { BrowserManager } from 'crawly-automation';\n\n(async () =\u003e {\n  const browserManager = new BrowserManager();\n\n  const browser = await browserManager.launchBrowser({ headless: false });\n  const page = await browserManager.createPage();\n\n  await page.goto('https://example.com');\n  console.log('Page title:', await page.title());\n\n  await browserManager.closeBrowser();\n})();\n```\n\n---\n\n### **2. Data Extraction**\n\nExtract data from a webpage using selectors:\n\n```typescript\nimport { DataExtractor } from 'crawly-automation';\n\n(async () =\u003e {\n  const browserManager = new BrowserManager();\n  const browser = await browserManager.launchBrowser();\n  const page = await browserManager.createPage();\n\n  await page.goto('https://example.com');\n\n  const extractor = new DataExtractor();\n  extractor.addSelector('title', 'h1').addSelector('items', '.item', true);\n\n  const data = await extractor.extract(page);\n  console.log('Extracted Data:', data);\n\n  await browserManager.closeBrowser();\n})();\n```\n\nOutput:\n```json\n{\n  \"title\": \"Example Title\",\n  \"items\": [\"Item 1\", \"Item 2\", \"Item 3\"]\n}\n```\n\n---\n\n### **3. Data Formatting (JSON and CSV)**\n\nConvert extracted data to **JSON** or **CSV** using built-in utilities:\n\n```typescript\nimport { DataExtractor } from 'crawly-automation';\n\n(async () =\u003e {\n  const browserManager = new BrowserManager();\n  const browser = await browserManager.launchBrowser();\n  const page = await browserManager.createPage();\n\n  await page.goto('https://example.com');\n\n  const extractor = new DataExtractor();\n  extractor.addSelector('items', '.item', true);\n\n  const jsonData = await extractor.extractFormatted(page, 'json');\n  console.log('JSON Data:', jsonData);\n\n  const csvData = await extractor.extractFormatted(page, 'csv');\n  console.log('CSV Data:\\n', csvData);\n\n  await browserManager.closeBrowser();\n})();\n```\n\n---\n\n### **4. CAPTCHA Handling**\n\nSolve CAPTCHAs (e.g., reCAPTCHA) automatically with 2Captcha or similar services:\n\n#### **Step 1: Configure CAPTCHA Solver**\nSet up the CAPTCHA solver with your API key:\n\n```typescript\nimport { BrowserManager } from 'crawly-automation';\n\nconst browserManager = new BrowserManager();\n\n// Configure RecaptchaPlugin\nbrowserManager.setRecaptchaConfig({\n    id: '2captcha',\n    token: 'YOUR_2CAPTCHA_API_KEY',\n});\n```\n\n#### **Step 2: Handle CAPTCHA**\nHandle CAPTCHAs during your crawling process:\n\n```typescript\n(async () =\u003e {\n  const browser = await browserManager.launchBrowser();\n  const page = await browser.newPage();\n\n  await page.goto('https://example.com');\n\n  try {\n      await browserManager.handleCaptcha(page);\n  } catch (error) {\n      console.error('CAPTCHA Handling Error:', error.message);\n  }\n\n  console.log('CAPTCHA solved, proceeding...');\n  await browserManager.closeBrowser();\n})();\n```\n\n---\n\n### **5. User Actions**\n\nSimulate user interactions like logging in or uploading a file:\n\n#### **Single-Page Login**\n```typescript\nimport { UserActions } from 'crawly-automation';\n\n(async () =\u003e {\n  const browserManager = new BrowserManager();\n  const browser = await browserManager.launchBrowser();\n  const page = await browserManager.createPage();\n\n  const userActions = new UserActions(page);\n\n  await userActions.login('#username', '#password', '#submit', {\n    username: 'user@example.com',\n    password: 'securepassword',\n  });\n\n  await browserManager.closeBrowser();\n})();\n```\n\n---\n\n### **6. Logging and Helpers**\n\n#### **Logging**\nUse the built-in `Logger` utility for structured logs:\n\n```typescript\nimport { Logger } from 'crawly-automation';\n\nLogger.info('This is an informational message.');\nLogger.error('An error occurred.');\nLogger.debug('Debugging message (only visible if DEBUG=true).');\n```\n\n#### **Delay**\nAdd custom delays between actions:\n\n```typescript\nimport { delay } from 'crawly-automation';\n\nawait delay(3000); // Wait for 3 seconds\n```\n\n---\n\n## 🛠️ API Reference\n\n### **BrowserManager**\n- `launchBrowser(options: PuppeteerLaunchOptions): Promise\u003cBrowser\u003e`\n- `setRecaptchaConfig(config: { id: string; token: string }): void`\n- `handleCaptcha(page: Page): Promise\u003cboolean\u003e`\n- `closeBrowser(): Promise\u003cvoid\u003e`\n\n### **DataExtractor**\n- `addSelector(name: string, selector: string, multiple?: boolean): this`\n- `extract(page: Page): Promise\u003cRecord\u003cstring, any\u003e\u003e`\n- `extractFormatted(page: Page, format: 'json' | 'csv'): Promise\u003cstring\u003e`\n\n### **Utilities**\n- `Logger`: Structured logging (`info`, `warn`, `error`, `debug`)\n- `delay(ms: number): Promise\u003cvoid\u003e`\n\n---\n\n## 📝 License\n\nThis project is licensed under the MIT License. See the [LICENSE](./LICENSE) file for details.\n\n---\n\n## 👤 Author\n\n**Afshin Tavakolian**  \nEmail: [afsh7n@gmail.com](mailto:afsh7n@gmail.com)  \nGitHub: [AfshinTavakolian](https://github.com/afsh7n)\n\n---\n\n## ❤️ Contributions\n\nContributions, issues, and feature requests are welcome!  \nFeel free to check the [issues page](https://github.com/afsh7n/crawly-automation/issues).\n\n---\n\n## 🌟 Support\n\nIf you find this project helpful, please give it a ⭐ on GitHub!\n\n---\n\n## 📥 Installation Command\n\nFor quick installation, just run:\n\n```bash\nnpm install crawly-automation\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafsh7n%2Fcrawly-automation","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fafsh7n%2Fcrawly-automation","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fafsh7n%2Fcrawly-automation/lists"}