{"id":13401754,"url":"https://github.com/zhuyingda/webster","last_synced_at":"2025-05-15T03:06:04.035Z","repository":{"id":37405420,"uuid":"109503867","full_name":"zhuyingda/webster","owner":"zhuyingda","description":"a reliable high-level web crawling \u0026 scraping framework for Node.js.","archived":false,"fork":false,"pushed_at":"2025-03-12T13:14:06.000Z","size":185,"stargazers_count":525,"open_issues_count":1,"forks_count":57,"subscribers_count":30,"default_branch":"master","last_synced_at":"2025-05-09T19:21:55.339Z","etag":null,"topics":["automation-test","automation-ui","chromium","crawler","crawling","headless-chrome","javascript","javascript-framework","nodejs","nodejs-framework","puppeteer","scraping-framework","spider"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zhuyingda.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null},"funding":{"github":["zhuyingda"],"open_collective":"webster"}},"created_at":"2017-11-04T14:56:37.000Z","updated_at":"2025-03-23T22:28:28.000Z","dependencies_parsed_at":"2024-01-18T18:25:39.013Z","dependency_job_id":"6a553a4e-593f-4cf8-a2ba-da5ee58edf6e","html_url":"https://github.com/zhuyingda/webster","commit_stats":{"total_commits":128,"total_committers":7,"mean_commits":"18.285714285714285","dds":0.5390625,"last_synced_commit":"6a260eec8281e28c87cbee9a84f967214e313772"},"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuyingda%2Fwebster","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuyingda%2Fwebster/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuyingda%2Fwebster/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zhuyingda%2Fwebster/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zhuyingda","download_url":"https://codeload.github.com/zhuyingda/webster/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254071507,"owners_count":22009802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation-test","automation-ui","chromium","crawler","crawling","headless-chrome","javascript","javascript-framework","nodejs","nodejs-framework","puppeteer","scraping-framework","spider"],"created_at":"2024-07-30T19:01:06.700Z","updated_at":"2025-05-15T03:06:04.007Z","avatar_url":"https://github.com/zhuyingda.png","language":"JavaScript","funding_links":["https://github.com/sponsors/zhuyingda","https://opencollective.com/webster","https://opencollective.com/webster/contribute","https://opencollective.com/webster/organization/0/website","https://opencollective.com/webster/organization/1/website","https://opencollective.com/webster/organization/2/website","https://opencollective.com/webster/organization/3/website","https://opencollective.com/webster/organization/4/website","https://opencollective.com/webster/organization/5/website","https://opencollective.com/webster/organization/6/website","https://opencollective.com/webster/organization/7/website","https://opencollective.com/webster/organization/8/website","https://opencollective.com/webster/organization/9/website"],"categories":["JavaScript","All","Repository","📦 Legacy \u0026 Inactive Projects"],"sub_categories":["Crawler"],"readme":"# Webster\n[![Financial Contributors on Open Collective](https://opencollective.com/webster/all/badge.svg?label=financial+contributors)](https://opencollective.com/webster) [![npm version](https://badge.fury.io/js/webster.svg)](https://www.npmjs.com/package/webster)\n[![Build Status](https://travis-ci.org/zhuyingda/webster.svg?branch=master)](https://travis-ci.org/zhuyingda/webster)\n\n## Overview\n\nWebster is a reliable web crawling and scraping framework written with Node.js, used to crawl websites and extract structured data from their pages.\n\nWhich is different from other crawling framework is that Webster can scrape the content which rendered by browser client side javascript and ajax request\n\n## Quick Start\nLet's start a simple crawler request to google website:\n```\ndocker pull zhuyingda/webster-playground\n\ndocker run --tty -e URL=\"https://www.google.com/robots.txt\" zhuyingda/webster-playground node crawler.js\n\n# add cookie with sign-in session\ndocker run --tty -e MOD=debug -e URL=\"https://www.google.com/robots.txt\" -e Cookie=\"foo=1234; bar=abcd\" zhuyingda/webster-playground node crawler.js\n\n# set user-agent\ndocker run --tty -e URL=\"https://www.google.com/robots.txt\" -e Cookie=\"foo=1234; bar=abcd\" -e UA=\"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36\" zhuyingda/webster-playground node crawler.js\n\n# see crawling log\ndocker run --tty -e MOD=debug -e URL=\"https://www.google.com/robots.txt\" -e Cookie=\"foo=1234; bar=abcd\" -e UA=\"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36\" zhuyingda/webster-playground node crawler.js\n```\n\n## Requirements\n- Node.js 10.x+\n- Works on Linux, Mac OSX\n\nOr you can deploy on [Docker](https://hub.docker.com/r/zhuyingda/webster-runtime/).\n\n## Install\n```bash\nnpm install webster\n```\n\n## Single spider example\n\n```javascript\nconst { spider } = require('webster');\n\nclass MySpider extends spider {\n    get defUserAgent() {\n        return 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36';\n    }\n    get defDeviceType() {\n        return 'pc';\n    }\n    async parseHtml(html) {\n        return true;\n    }\n}\n\n(async () =\u003e {\n    const spider = new MySpider({\n        actions: [\n            {\n                type: 'waitForSelector',\n                selector: 'div.js-details-container',\n            }\n        ],\n        targets: [\n            {\n                selector: 'div.Box-row[role=row]',\n                type: 'text',\n                field: 'sugs'\n            }\n        ],\n    });\n    const url = `https://github.com/zhuyingda/webster`;\n    let crawlResult = await spider.startRequest(url);\n    console.log(crawlResult);\n})();\n```\n\n## Docker cluster example\n\nPull the example docker image:\n```bash\ndocker pull zhuyingda/webster-demo\ndocker run -it zhuyingda/webster-demo\n```\n\nIn this docker image, there is a simple cluster-able example:\n\n```javascript\n// producer\nconst Webster = require('webster');\nconst Producer = Webster.producer;\nconst Task = Webster.task;\n\nlet task = new Task({\n    spiderType: 'browser',\n    engineType: 'playwright',\n    browserType: 'chromium',\n    url: 'http://quotes.toscrape.com/tag/humor/',\n    targets: [\n        {\n            selector: 'span.text',\n            type: 'text',\n            field: 'quote'\n        },\n        {\n            selector: 'li.next \u003e a',\n            type: 'attr',\n            attrName: 'href',\n            field: 'link'\n        }\n    ],\n    actions: [\n        {\n            type: 'waitAfterPageLoading',\n            value: 500\n        }\n    ],\n    referInfo: {\n        para1: 'this is a refer field 1',\n        para2: 'this is a refer field 2'\n    }\n});\n\nlet myProducer = new Producer({\n    channel: 'demo_channel1',\n    dbConf: {\n        redis: {\n            host: 'redis-12419.c44.us-east-1-2.ec2.cloud.redislabs.com',\n            port: 12419,\n            password: 'X2AcjziaOOYPppWFOPiP4rmzZ9RFLViv'\n        }\n    }\n});\nmyProducer.generateTask(task).then(() =\u003e {\n    console.log('done');\n    process.exit();\n});\n```\n\n```javascript\n// consumer\nconst Webster = require('webster');\nconst Consumer = Webster.consumer;\n\nclass MyConsumer extends Consumer {\n    constructor(option) {\n        super(option);\n    }\n    afterCrawlRequest(result) {\n        console.log('your scrape result:', result);\n    }\n}\n\nlet myConsumer = new MyConsumer({\n    channel: 'demo_channel1',\n    sleepTime: 5000,\n    deviceType: 'pc',\n    dbConf: {\n        redis: {\n            host: 'redis-12419.c44.us-east-1-2.ec2.cloud.redislabs.com',\n            port: 12419,\n            password: 'X2AcjziaOOYPppWFOPiP4rmzZ9RFLViv'\n        }\n    }\n});\nmyConsumer.startConsume();\n```\n\n```bash\nnode demo_producer.js\nenv MOD=debug node demo_consumer.js\n```\n\nYou can organize your crawler cluster by Consumer and Producer like this:\n![](https://raw.githubusercontent.com/zhuyingda/webster/master/doc/webster-workflow.svg)\n\n## Usage on Raspbian Platform\n```bash\nsudo apt install chromium-browser chromium-codecs-ffmpeg\nenv MOD=debug EXE_PATH=/usr/bin/chromium-browser node demo_consumer.js\n```\n\n## Documentation\nYou can see more details from [here](http://webster.zhuyingda.com/).\n\n### Code Contributors\n\nThis project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].\n\u003ca href=\"https://github.com/zhuyingda/webster/graphs/contributors\"\u003e\u003cimg src=\"https://opencollective.com/webster/contributors.svg?width=890\u0026button=false\" /\u003e\u003c/a\u003e\n\n### Financial Contributors\n\nBecome a financial contributor and help us sustain our community. [[Contribute](https://opencollective.com/webster/contribute)]\n\n#### Individuals\n\n\u003ca href=\"https://opencollective.com/webster\"\u003e\u003cimg src=\"https://opencollective.com/webster/individuals.svg?width=890\"\u003e\u003c/a\u003e\n\n#### Organizations\n\nSupport this project with your organization. Your logo will show up here with a link to your website. [[Contribute](https://opencollective.com/webster/contribute)]\n\n\u003ca href=\"https://opencollective.com/webster/organization/0/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/0/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/1/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/1/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/2/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/2/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/3/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/3/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/4/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/4/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/5/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/5/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/6/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/6/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/7/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/7/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/8/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/8/avatar.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://opencollective.com/webster/organization/9/website\"\u003e\u003cimg src=\"https://opencollective.com/webster/organization/9/avatar.svg\"\u003e\u003c/a\u003e\n\n## License\n\n[GPL-V3](http://www.gnu.org/licenses/)\n\nCopyright (c) 2017-present, Yingda (Sugar) Zhu\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhuyingda%2Fwebster","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzhuyingda%2Fwebster","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzhuyingda%2Fwebster/lists"}