{"id":20618037,"url":"https://github.com/applitools/js-virtual-grid-crawler","last_synced_at":"2025-04-15T11:37:30.414Z","repository":{"id":40749389,"uuid":"176810864","full_name":"applitools/JS-Virtual-Grid-Crawler","owner":"applitools","description":"Quickly collect screenshots of all your website pages and responsive viewports.","archived":false,"fork":false,"pushed_at":"2023-03-04T03:23:26.000Z","size":1066,"stargazers_count":7,"open_issues_count":10,"forks_count":3,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-28T19:38:10.482Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/applitools.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-03-20T20:21:24.000Z","updated_at":"2022-04-29T12:47:26.000Z","dependencies_parsed_at":"2023-01-28T23:30:40.033Z","dependency_job_id":null,"html_url":"https://github.com/applitools/JS-Virtual-Grid-Crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/applitools%2FJS-Virtual-Grid-Crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/applitools%2FJS-Virtual-Grid-Crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/applitools%2FJS-Virtual-Grid-Crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/applitools%2FJS-Virtual-Grid-Crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/applitools","download_url":"https://codeload.github.com/applitools/JS-Virtual-Grid-Crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":249061811,"owners_count":21206568,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-16T12:06:54.069Z","updated_at":"2025-04-15T11:37:30.381Z","avatar_url":"https://github.com/applitools.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# JavaScript Ultra-Fast Visual Grid Sitemap Crawler\n\n### Quickly collect screenshots of all your website pages and responsive viewports.\n\n# Disclaimer\n* This is a POC of using the Applitools Visual Grid and sitemap crawling a website. This is free for use (with VG Subscription), change/modify, and do whatever you please with it. This is **NOT SUPPORTED by Applitools** Support or Dev teams. It's use at your own discretion and is not guaranteed to work in the future. \n\n### To Install:\n\n* ```$ git clone git@github.com:applitools/JS-Virtual-Grid-Crawler.git```\n* ```$ npm install```\n\n### To Run:\n\n```\n$ node crawler.js --help\n\nUsage: crawler [options]\n\nOptions:\n  -V, --version                 output the version number\n  -u --url [url]                Add the site URL you want to generate a sitemap for. e.g. -u https://www.seleniumconf.com\n  -s --sitemap [sitemap]        Use an already existing sitemap file. e.g. -s \"/path/to/sitemap.xml\" Note: This overrides the -u arg\n  -m, --sitemapUrl [sitemapUrl  Specify a sitemap URL. e.g. -m https://www.example.com/sitemap.xml\n  -b, --browsers [browsers]     Add the MAX number of browsers to run concurrently. e.g. -b 10. Note: Be careful with this!\n  -k --key [key]                Set your Applitools API Key. e.g. -k yourLongAPIKeyyyyy\n  -v --serverUrl [serverUrl]    Set your Applitools server URL. (Default: https://eyesapi.applitools.com). e.g. -v  https://YourEyesapi.applitools.com\n  --no-grid                     Disable the Visual Grid and run locally only (Default: false). e.g. --no-grid\n  --logs                        Enable Applitools Debug Logs (Default: false). e.g. --logs\n  --headless                    Run Chrome headless (Default: false). e.g. --headless\n  --no-fullPage                 Disable Full Page Screenshot (Default: full page). e.g. --no-fullPage\n  -U --URL [URL]                Add a single web URL you want to capture images for. e.g. -U https://www.google.com\n  -a --appName [appName]        Override the appName. e.g. -a MyApp\n  -t --testName [testName]      Override the testName. e.g. -t MyTest\n  -l --level [level]            Set your Match Level \"Layout2, Content, Strict, Exact\" (Default: Strict). e.g. -l Layout2\n  -p --proxy [proxy]            Set your Proxy URL\" (Default: None). e.g. -p http://proxyhost:port,username,password\n  -B --batch [batch]            Set your Batch Name\" (Default: sitemap filename or url). e.g. -B MyBatch\n  -h, --help                    output usage information\n```\n\n### Examples:\n\n* Set an environment letiable for your Applitools API Key. e.g. export APPLITOOLS_API_KEY=\"Your_API_KEY\"\n\n* Generate Sitemap and Run: `$ node crawler.js -u https://www.seleniumconf.com`\n* Use a sitemap.xml URL and Run: `$ node crawler.js -m https://slack.com/sitemap.xml -b 20 --headless`\n* Use existing sitemap.xml and Run: `$ node crawler.js -s ./sitemaps/www.seleniumconf.com.xml`\n* Use a self made sitemap and Run: `$ node crawler.js -s ./sitemaps/random-sitemap.xml --appName random-urls`\n* Open 20 browsers concurrently (default: 10): `$ node crawler.js -s ./sitemaps/www.primerica.com.xml -b 20`\n   * The max browsers by default is 10. However, if the sitemap.xml only has 5 links, then only 5 browsers will open.\n   * ***Be careful with this value***. Opening too many browsers might kill your machine. Leave it at the default (10) and tweak this value slightly until you know the ideal number your machine can handle.\n* Disable Visual Grid and Run locally: `$ node crawler.js -s ./sitemaps/www.seleniumconf.com.xml --no-grid`\n* Enable Applitools Debug logs: `$ node crawler.js -s ./sitemaps/www.seleniumconf.com.xml --log`\n* Run Chrome Headless: `$ node crawler.js -s ./sitemaps/www.seleniumconf.com.xml --headless`\n* Overides: Set API Key and On-Prem/Private Cloud Server URL and Run: `$ node crawler.js -u https://seleniumconf.com -k YourApiKey -v https://youreyesapi.applitools.com`\n* Crawl a single URL: `$ node crawler.js -U https://www.google.com`\n* Crawl a single URL and set a App and Test Name: `$ node crawler.js -U https://www.google.com -a Google -t HomePage`\n* Disable Full Page Screenshot: `$ node crawler.js -s ./sitemaps/www.seleniumconf.com.xml --no-fullPage`\n\n### Notes:\n\n* Quit during mid-execution:\n   * ctrl-c only once and wait! This should put you in the FINALLY block to kill the execution and close all browsers. Repeated ctrl-c might break out of the his block and leave zombie browsers running on your pc which you'll have to manually kill. \n\n### Config Options:\n   * This can be modified in the applitools.config.js file.\n\n```\n    serverUrl: \"https://eyesapi.applitools.com\",\n    apiKey: process.env.APPLITOOLS_API_KEY,\n    fullPage: true,\n    logs: false,\n    sendDom: false,\n    lazyLoad: true,\n    proxy: null, //'http://localhost:8888,yourUser,yourPassword',\n    browsersInfo: [\n        { width: 1200, height: 800, name: 'chrome'  },\n        { width: 1200, height: 800, name: 'firefox' },\n        { width: 1200, height: 800, name: 'ie10'    },\n        { width: 1200, height: 800, name: 'ie11'    },\n        { width: 1200, height: 800, name: 'edge'    },\n        { deviceName: 'iPhone X', screenOrientation: 'portrait' },\n        { deviceName: 'iPad',     screenOrientation: 'portrait' },\n        { deviceName: 'Nexus 7',  screenOrientation: 'portrait' },\n        { deviceName: 'Pixel 2',  screenOrientation: 'portrait' }\n     ],\n\n     //An Array of raw Selenium steps to take after the page loads... clicks, sendKeys, scroll etc...\n    afterPageLoad: [\n      \"driver.findElement(By.css('span.cta-link.primary.link-text-yes')).click()\",\n      \"driver.findElement(By.css('div.cc-compliance')).click()\"\n   ],\n```\n\n### ToDos:\n\n* Multithread/process the sitemap creation to speed it up.\n* Clean/Dry the code. Split methods into classes.\n* Add additional checks/actions to a sitemap. e.g: \n```\n   \u003curl\u003e\n      \u003cloc\u003ehttps://www.seleniumconf.com/\u003c/loc\u003e\n      \u003caction\u003edriver.findElement(By.tagName('button')).click();\u003c/action\u003e\n      \u003ccheck\u003eeyes.checkElementBy(By.css(\"div.section\"), null, \"Example\")\u003c/check\u003e\n   \u003c/url\u003e\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapplitools%2Fjs-virtual-grid-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fapplitools%2Fjs-virtual-grid-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fapplitools%2Fjs-virtual-grid-crawler/lists"}