Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/sssshefer/web-crawler-http
Basic web crawler which represents the linking structure of the website
https://github.com/sssshefer/web-crawler-http
crawler jest jest-tests js
Last synced: 4 days ago
JSON representation
Basic web crawler which represents the linking structure of the website
- Host: GitHub
- URL: https://github.com/sssshefer/web-crawler-http
- Owner: sssshefer
- Created: 2024-05-05T11:32:12.000Z (7 months ago)
- Default Branch: main
- Last Pushed: 2024-05-10T15:57:38.000Z (6 months ago)
- Last Synced: 2024-05-11T09:25:07.987Z (6 months ago)
- Topics: crawler, jest, jest-tests, js
- Language: JavaScript
- Homepage:
- Size: 50.8 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Web Crawler HTTP
Basic web crawler which represents the linking structure of the website
The purpose of the application is to scan a website and generate a report detailing its internal linking structure.
Typically, such tools are utilized by web marketers and experts in search engine optimization (SEO)
## How to run the application
Git clone the repo
```bash
$ git clone https://github.com/sssshefer/web-crawler-http.git
```
Check all tests for the application
```bash
$ npm run test
```
Run the application
```bash
$ npm run start https://linkToYouWebsite.com
```