https://github.com/gplumb/netcrawlerdetect
A .net standard port of JayBizzle's CrawlerDetect project (https://github.com/JayBizzle/Crawler-Detect).
https://github.com/gplumb/netcrawlerdetect
bots c-sharp crawler detect dotnet-core dotnet-standard spider user-agent
Last synced: about 1 year ago
JSON representation
A .net standard port of JayBizzle's CrawlerDetect project (https://github.com/JayBizzle/Crawler-Detect).
- Host: GitHub
- URL: https://github.com/gplumb/netcrawlerdetect
- Owner: gplumb
- License: mit
- Created: 2018-01-30T16:23:28.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2023-02-04T16:48:52.000Z (over 3 years ago)
- Last Synced: 2025-03-25T14:39:04.593Z (about 1 year ago)
- Topics: bots, c-sharp, crawler, detect, dotnet-core, dotnet-standard, spider, user-agent
- Language: C#
- Size: 2.67 MB
- Stars: 20
- Watchers: 3
- Forks: 1
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
Awesome Lists containing this project
README
# NetCrawlerDetect [](https://travis-ci.org/gplumb/NetCrawlerDetect)
A .net standard port of JayBizzle's CrawlerDetect project: [https://github.com/JayBizzle/Crawler-Detect](https://github.com/JayBizzle/Crawler-Detect).
## About NetCrawlerDetect
NetCrawlerDetect is a .net standard class for detecting bots/crawlers/spiders via the user agent and/or http "from" header. Currently able to detect 1,000s of bots/spiders/crawlers.
### Installation
The easiest way to get started is to add the nuget package `NetCrawlerDetect` (see [here](https://www.nuget.org/packages/NetCrawlerDetect)).
For those who don't want to use nuget, feel free to clone this repo and copy `CrawlerDetect.cs` and the `Fixtures` folder into your project directly.
### Usage
Like its' originator, you can either pass in a collection of web headers (from which the user agent will be extracted) or pass in the user agent string directly.
The simplest use of this object is as follows:
```csharp
// Pass in the user agent directly
var detector = new CrawlerDetect();
var result = detector.IsCrawler("Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)");
// Do we have a bot/crawler/spider?
if(result == true)
{
// Yes. Fetch the name of the bot (optional)
var bot = detector.Matches[0].Value;
}
```
### Contributing
If you find a bot/spider/crawler user agent that NetCrawlerDetect fails to detect, please submit a pull request with the regex pattern added to the `_data` List in `Fixtures/Crawlers.cs` and add the failing user agent to `NetCrawlerDetect.Tests/crawlers.txt`.
Please also consider submitting a pull request to our parent project :)