```php
$spider->addDiscoverer(new XPathExpressionDiscoverer("//div[@id='catalogs']//a"));
```
Set some sane options for this example. In this case, we only get the first 10 items from the start page.

```php
$spider->setMaxDepth(1);
$spider->setMaxQueueSize(10);
```

Add a listener to collect stats from the Spider and the QueueManager.
There are more components that dispatch events you can use.

```php
$statsHandler = new StatsHandler();
$spider->getQueueManager()->getDispatcher()->addSubscriber($statsHandler);
$spider->getDispatcher()->addSubscriber($statsHandler);
```
Execute the crawl

```php
$spider->crawl();
```
When crawling is done, we could get some info about the crawl
```php
echo "\n ENQUEUED: " . count($statsHandler->getQueued());
echo "\n SKIPPED: " . count($statsHandler->getFiltered());
echo "\n FAILED: " . count($statsHandler->getFailed());
echo "\n PERSISTED: " . count($statsHandler->getPersisted());
```
Finally we could do some processing on the downloaded resources. In this example, we will echo the title of all resources
```php
echo "\n\nDOWNLOADED RESOURCES: ";
foreach ($spider->getDownloader()->getPersistenceHandler() as $resource) {
echo "\n - " . $resource->getCrawler()->filterXpath('//title')->text();
}

```

### Fluent Configuration

For most common settings, you can configure the spider fluently via convenience methods on `Spider` and keep related configuration in one place.

```php
use VDB\Spider\Spider;
use VDB\Spider\Discoverer\XPathExpressionDiscoverer;
use VDB\Spider\Filter\Prefetch\AllowedHostsFilter;
use VDB\Spider\PersistenceHandler\FileSerializedResourcePersistenceHandler;
use VDB\Spider\QueueManager\QueueManagerInterface;

$spider = new Spider('https://example.com');

// Configure limits and traversal in one place
$spider
->setDownloadLimit(50) // Max resources to download
->setTraversalAlgorithm(QueueManagerInterface::ALGORITHM_BREADTH_FIRST)
->setMaxDepth(2) // Max discovery depth
->setMaxQueueSize(500) // Max URIs in queue
->setPersistenceHandler(new FileSerializedResourcePersistenceHandler(__DIR__.'/results'))
->addDiscoverer(new XPathExpressionDiscoverer('//a')) // Add discoverers
->addFilter(new AllowedHostsFilter(['example.com'])); // Add prefetch filters

// Optional: enable politeness policy (delay between requests to same domain)
$spider->enablePolitenessPolicy(100);

$spider->crawl();
```

### Using Cache to Skip Already Downloaded Resources

To avoid re-downloading resources that are already cached (useful for incremental crawls):

```php
use VDB\Spider\Filter\Prefetch\CachedResourceFilter;
use VDB\Spider\PersistenceHandler\FileSerializedResourcePersistenceHandler;

// Use a fixed spider ID to share cache across runs
$spiderId = 'my-spider-cache';
$spider = new Spider('http://example.com', null, null, null, $spiderId);

// Set up file persistence
$resultsPath = __DIR__ . '/cache';
$spider->getDownloader()->setPersistenceHandler(
new FileSerializedResourcePersistenceHandler($resultsPath)
);

// Add cache filter - skip resources downloaded within the last hour
$maxAgeSeconds = 3600; // 1 hour (set to 0 to always use cache)
$cacheFilter = new CachedResourceFilter($resultsPath, $spiderId, $maxAgeSeconds);
$spider->getDiscovererSet()->addFilter($cacheFilter);

$spider->crawl();
```

For more details, see the [CachedResourceFilter documentation](docs/filters/CachedResourceFilter.md) and [example](example/example_cache.php).

Contributing
------------
Contributing to PHP-Spider is as easy as Forking the repository on Github and submitting a Pull Request.
The Symfony documentation contains an excellent guide for how to do that properly here: [Submitting a Patch](http://symfony.com/doc/current/contributing/code/patches.html#step-1-setup-your-environment).

There a few requirements for a Pull Request to be accepted:
- Follow the coding standards: PHP-Spider follows the coding standards defined in the [PSR-0](https://github.com/php-fig/fig-standards/blob/master/accepted/PSR-0.md), [PSR-1](https://github.com/php-fig/fig-standards/blob/master/accepted/PSR-1-basic-coding-standard.md) and [PSR-2](https://github.com/php-fig/fig-standards/blob/master/accepted/PSR-2-coding-style-guide.md) Coding Style Guides;
- Prove that the code works with unit tests and that coverage remains 100%;

> Note: An easy way to check if your code conforms to PHP-Spider is by running the script `bin/static-analysis`, which is part of this repo. This will run the following tools, configured for PHP-Spider: PHP CodeSniffer, PHP Mess Detector and PHP Copy/Paste Detector.

> Note: To run PHPUnit with coverage, and to check that coverage == 100%, you can run `bin/coverage-enforce`.

### Local Testing with GitHub Actions

You can run the full CI pipeline locally using [nektos/act](https://nektosact.com/):

```bash
# Fast path: run the full workflow with PHP 8.0 (recommended)
./bin/check
```

Or use the underlying act wrapper directly:

```bash
# Run all tests locally
./bin/act

# Run specific PHP version locally
./bin/act --matrix php-versions:8.0

# Run specific job or view available workflows
./bin/act -l
```

For more details, see [.github/LOCAL_TESTING.md](.github/LOCAL_TESTING.md).

Support
-------
For things like reporting bugs and requesting features it is best to create an [issue](https://github.com/mvdbos/php-spider/issues) here on GitHub. It is even better to accompany it with a Pull Request. ;-)

License
-------
PHP-Spider is licensed under the MIT license.