https://github.com/binarykitchen/spider-detector

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS
https://github.com/binarykitchen/spider-detector

Last synced: over 1 year ago
JSON representation

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

• Host: GitHub
• URL: https://github.com/binarykitchen/spider-detector
• Owner: binarykitchen
• License: other
• Created: 2015-01-03T06:26:09.000Z (over 11 years ago)
• Default Branch: master
• Last Pushed: 2024-08-28T22:23:13.000Z (almost 2 years ago)
• Last Synced: 2025-02-17T16:16:01.127Z (over 1 year ago)
• Language: JavaScript
• Size: 70.3 KB
• Stars: 36
• Watchers: 4
• Forks: 11
• Open Issues: 2
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# spider-detector

[![Build Status](https://travis-ci.org/binarykitchen/spider-detector.svg?branch=master)](https://travis-ci.org/binarykitchen/spider-detector) [![npm version](https://badge.fury.io/js/spider-detector.svg)](https://badge.fury.io/js/spider-detector)

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

It might be useful when you have a single page app but want to deliver static pages for spiders.

## Install

```

npm install spider-detector // or `yarn install spider-detector`

```

## Direct Example

```js

const detector = require('spider-detector')

detector.isSpider('baiduspider') // return true

```

## ExpressJS example

```js

const detector = require('spider-detector')

const express = require('express')

const app = express()

app.use(detector.middleware())

app.get('/*', function(req, res) {

  if (req.isSpider()) {

    // do something else, i.E. send a static page

  } else {

    // send single page app

  }

})

```

## Why? There are already modules out there

Well, I wanted one which does not use `readFileSync` and comes with optional middleware. Furthermore some hackers do not classify Googlebot as a spider anymore which poses a problem sometimes, see next question.

## What about Googlebot?

Yep, Googlebot is able to deal with single page apps but this feature is pretty unstable. Especially under AngularJS when hash fragments are disabled with `$locationProvider.html5Mode(true)`. That's why - against all odds - I have classified `googlebot` as a spider in this module.