{"id":16465934,"url":"https://github.com/yusuftaufiq/cli-website-crawler","last_synced_at":"2026-05-06T00:03:54.688Z","repository":{"id":182769017,"uuid":"668918023","full_name":"yusuftaufiq/cli-website-crawler","owner":"yusuftaufiq","description":"Non-blocking CLI based application to recursively crawl data from whole pages on websites in parallel and save the results to HTML output. Built with Node.js, TypeScript, NestJs, and Playwright.","archived":false,"fork":false,"pushed_at":"2023-07-21T08:07:36.000Z","size":2756,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"main","last_synced_at":"2025-01-10T13:51:43.426Z","etag":null,"topics":["cli","crawling","nestjs","nodejs","playwraight","scraping","typescript"],"latest_commit_sha":null,"homepage":"","language":"HTML","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/yusuftaufiq.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-20T23:04:57.000Z","updated_at":"2023-09-07T00:56:52.000Z","dependencies_parsed_at":"2023-07-21T10:52:59.713Z","dependency_job_id":null,"html_url":"https://github.com/yusuftaufiq/cli-website-crawler","commit_stats":null,"previous_names":["yusuftaufiq/cmlabs-backend-crawler-freelance-test","yusuftaufiq/cli-website-crawler"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuftaufiq%2Fcli-website-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuftaufiq%2Fcli-website-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuftaufiq%2Fcli-website-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/yusuftaufiq%2Fcli-website-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/yusuftaufiq","download_url":"https://codeload.github.com/yusuftaufiq/cli-website-crawler/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241026727,"owners_count":19896692,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cli","crawling","nestjs","nodejs","playwraight","scraping","typescript"],"created_at":"2024-10-11T11:35:44.672Z","updated_at":"2026-05-06T00:03:54.675Z","avatar_url":"https://github.com/yusuftaufiq.png","language":"HTML","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Table of Contents\n\n- [Description](#description)\n- [How it works](#how-it-works)\n- [Technical details](#technical-details)\n- [Installation](#installation)\n- [Usage](#usage)\n- [TODO](#todo)\n\n## Description\n\nNon-blocking CLI based application to recursively crawl data from whole pages on websites in parallel and save the results to HTML output.\n\n![Overview](./assets/overview.png)\n\n## How it works\n\n- Open pages using [Playwright](https://playwright.dev/).\n- On pages, find new link elements that have an HTML `a` tag on the page.\n- Filter only links that point to the same domain and are allowed in `robots.txt`.\n- Add links to the request queue.\n- Skips duplicate URLs.\n- Visit recently queued links.\n- Repeat the process.\n\n## Technical details\n\n- Tech stack: [Node.js](https://nodejs.org/en), [TypeScript](https://www.typescriptlang.org/), [NestJs](https://nestjs.com/), [Playwright](https://playwright.dev/)\n- Data structures: [Hash Map](./src/robots/robots.service.ts#L29), [Hash Set](./src/crawl/handlers/default.handler.ts#L40) for high performance O(1) constant insertion and retrieval.\n- Architectures: modules, services, and commands separated by feature purpose.\n\n## Installation\n\n- Requirements\n\n  - Node.js \u003e= 18\n\n- Clone this repository\n\n  ```bash\n  $ https://github.com/yusuftaufiq/cmlabs-backend-crawler-freelance-test.git\n  ```\n\n- Change to the cloned directory and install all required dependencies (may take a while)\n\n  ```bash\n  $ npm install\n  ```\n\n- Build the application\n\n  ```bash\n  $ npm run build\n  ```\n\n- Start the CLI application, all the features can be seen in the following [section](#usage)\n  ```bash\n  $ npm run start:prod -- crawl\n  ```\n  If the command is launched successfully, all results will be available in [./storage/key_value_stores](./storage/key_value_stores)\n\n## Usage\n\n- Show all available commands\n  ```bash\n  $ npm run start:prod -- --help\n  $ npm run start:prod -- crawl --help\n  ```\n- Customize targets to be crawled. (default: https://cmlabs.co/ https://www.sequence.day/ https://yusuftaufiq.com)\n  ```bash\n  $ npm run start:prod -- crawl https://books.toscrape.com/ https://quotes.toscrape.com/\n  ```\n- Control the verbosity of log messages (choices: \"off\", \"error\", \"soft_fail\", \"warning\", \"info\", \"debug\", \"perf\", default: \"info\")\n  ```bash\n  $ npm run start:prod -- crawl --log-level warning\n  ```\n- Sets the maximum concurrency (parallelism) for the crawl (default: 15)\n  ```bash\n  $ npm run start:prod -- crawl --max-concurrency 100\n  ```\n- Maximum number of pages that the crawler will open. The crawl will stop when this limit is reached. (default: 50)\n  ```bash\n  $ npm run start:prod -- crawl --max-requests 1000\n  ```\n- Timeout by which the function must complete, in seconds. (default: 30)\n  ```bash\n  $ npm run start:prod -- crawl --timeout 10\n  ```\n- Whether to run the browser in headful mode. (default: false)\n  ```bash\n  $ npm run start:prod -- crawl --headful\n  ```\n\n## TODO\n- Prioritize sitemap.xml\n- Add proxies\n- Watch out for honeypots\n- Adopt CAPTCHA solving service","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyusuftaufiq%2Fcli-website-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fyusuftaufiq%2Fcli-website-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fyusuftaufiq%2Fcli-website-crawler/lists"}