https://github.com/insign/spatie-crawler-queue-with-laravel-model
Spatie's Crawler with Laravel Model as Queue
https://github.com/insign/spatie-crawler-queue-with-laravel-model
cache crawler eloquent laravel queues spatie spatie-crawler
Last synced: 9 days ago
JSON representation
Spatie's Crawler with Laravel Model as Queue
- Host: GitHub
- URL: https://github.com/insign/spatie-crawler-queue-with-laravel-model
- Owner: insign
- License: mit
- Created: 2021-05-10T18:46:23.000Z (almost 4 years ago)
- Default Branch: main
- Last Pushed: 2021-05-10T19:58:47.000Z (almost 4 years ago)
- Last Synced: 2025-04-15T15:05:17.762Z (9 days ago)
- Topics: cache, crawler, eloquent, laravel, queues, spatie, spatie-crawler
- Language: PHP
- Homepage:
- Size: 89.8 KB
- Stars: 7
- Watchers: 2
- Forks: 1
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# spatie-crawler-queue-with-laravel-model
[Spatie's Crawler](https://github.com/spatie/crawler) with Laravel Model as Queue
This is just a laravel application 8.x with a model class, a queue class, a migration class and a command class to use Spatie's Crawler package.
### Why this is better than others spatie/crawler queues packages?
> The main reason is the others queues packages store all items in one single array, which can be a RAM problem for big sites.
> Furthermore, you can preserve and use crawled links as you want
> To expire items we use [mvdnbrk/laravel-model-expires](https://github.com/mvdnbrk/laravel-model-expires)
> Processed items are marked as [soft-deleted](https://laravel.com/docs/8.x/eloquent#soft-deleting)
# Steps
1. Clone the repo
2. Run `composer install`
3. Run `php artisan migration` (after configure `database.php`)
4. Adjust `app/Console/Commands/CrawlerRun.php`
5. Run `php artisan craw https://site_or_blog.com`
Main files to take a look:
* [`app/Console/Commands/CrawlerRun.php`](app/Console/Commands/CrawlerRun.php)
* [`app/Models/CrawlerQueue.php`](app/Models/CrawlerQueue.php)
* [`app/Observers/Crawler/ConsoleObserver.php`](app/Observers/Crawler/ConsoleObserver.php)
* [`database/migrations/2021_05_07_041816_crawler_queues.php`](database/migrations/2021_05_07_041816_crawler_queues.php)