https://github.com/piedweb/seopocketcrawler
[PHP] Web Crawler to check few SEO basics. Keep it in your Pocket !
https://github.com/piedweb/seopocketcrawler
Last synced: about 1 year ago
JSON representation
[PHP] Web Crawler to check few SEO basics. Keep it in your Pocket !
- Host: GitHub
- URL: https://github.com/piedweb/seopocketcrawler
- Owner: PiedWeb
- License: mit
- Created: 2019-01-15T17:24:28.000Z (over 7 years ago)
- Default Branch: master
- Last Pushed: 2021-01-20T16:33:00.000Z (over 5 years ago)
- Last Synced: 2024-07-22T05:26:57.329Z (almost 2 years ago)
- Language: PHP
- Homepage: https://dev.piedweb.com/
- Size: 102 KB
- Stars: 2
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# CLI Seo Pocket Crawler
[](https://github.com/PiedWeb/SeoPocketCrawler/tags)
[](LICENSE)
[](https://github.com/PiedWeb/SeoPocketCrawler/actions)
[](https://scrutinizer-ci.com/g/PiedWeb/SeoPocketCrawler)
[](https://codecov.io/gh/PiedWeb/SeoPocketCrawler/branch/main)
[](https://shepherd.dev/github/PiedWeb/SeoPocketCrawler)
[](https://packagist.org/packages/piedweb/seo-pocket-crawler)
Web Crawler to check few SEO basics.
Use the collected data in your favorite spreadsheet software or retrieve them via your favorite language.
French documentation available :
https://piedweb.com/seo/crawler
## Install
Via [Packagist](https://img.shields.io/packagist/dt/piedweb/seo-pocket-crawler.svg?style=flat)
``` bash
$ composer create-project piedweb/seo-pocket-crawler
```
## Usage
### Crawler CLI
``` bash
$ bin/console crawler:go $start
```
#### Arguments:
```
start Define where the crawl start. Eg: https://piedweb.com
You can specify an id from a previous crawl. Other options will not be listen.
You can use `last` to continue the last crawl (just stopped)
```
#### Options:
```
-l, --limit=LIMIT Define where a depth limit [default: 5]
-i, --ignore=IGNORE Virtual Robots.txt to respect (could be a string or an URL).
-u, --user-agent=USER-AGENT Define the user-agent used during the crawl. [default: "SEO Pocket Crawler - PiedWeb.com/seo/crawler"]
-w, --wait=WAIT In Microseconds, the time to wait between 2 requests. Default 0,1s. [default: 100000]
-c, --cache-method=CACHE-METHOD In Microseconds, the time to wait between two request. Default : 100000 (0,1s). [default: 2]
-r, --restart=RESTART Permit to restart a previous crawl. Values 1 = fresh restart, 2 = restart from cache
-h, --help Display this help message
-q, --quiet Do not output any message
-V, --version Display this application version
--ansi Force ANSI output
--no-ansi Disable ANSI output
-n, --no-interaction Do not ask any interactive question
-v|vv|vvv, --verbose Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
```
### Extract All External Links in 1s from a previous crawl
``` bash
$ bin/console crawler:external $id [--host]
```
```
--id
id from a previous crawl
You can use `last` too show external links from the last crawl.
--host -ho
flag permitting to get only host
```
### Calcul Page Rank
Will update the previous `data.csv` generated. Then you can explore your website with the PoC `pagerank.html`
(in a server `npx http-server -c-1 --port 3000`).
``` bash
$ bin/console crawler:pagerank $id
```
```
--id
id from a previous crawl
You can use `last` too calcul page rank from the last crawl.
```
## Testing
``` bash
$ composer test
```
## Todo
- [ ] Better Links Harvesting and Recording (record context (list, nav, sentence...))
- [ ] Transform the PoC (Page Rank Visualizer)
- [ ] Complex Page Rank Calculator (with 301, canonical, nofollow, etc.)
## Contributing
Please see [contributing](https://dev.piedweb.com/contributing)
## Credits
- [PiedWeb](https://piedweb.com) ak [Robind4](https://twitter.com/Robind4)
- [All Contributors](https://github.com/PiedWeb/:package_skake/graphs/contributors)
## License
The MIT License (MIT). Please see [License File](LICENSE) for more information.
[](https://github.com/PiedWeb/SeoPocketCrawler/tags)
[](https://github.com/PiedWeb/SeoPocketCrawler/blob/master/LICENSE)
[](https://travis-ci.org/PiedWeb/SeoPocketCrawler)
[](https://scrutinizer-ci.com/g/PiedWeb/SeoPocketCrawler)
[](https://scrutinizer-ci.com/g/PiedWeb/SeoPocketCrawler/code-structure)
[](https://packagist.org/packages/piedweb/seo-pocket-crawler)