https://github.com/fantasywind/phantom-robot-proxy
Phantom JS Proxy for Robot / Crawler
https://github.com/fantasywind/phantom-robot-proxy
Last synced: 8 months ago
JSON representation
Phantom JS Proxy for Robot / Crawler
- Host: GitHub
- URL: https://github.com/fantasywind/phantom-robot-proxy
- Owner: fantasywind
- License: mit
- Created: 2013-11-09T06:45:39.000Z (over 12 years ago)
- Default Branch: master
- Last Pushed: 2013-11-09T07:26:56.000Z (over 12 years ago)
- Last Synced: 2025-03-30T01:44:54.365Z (about 1 year ago)
- Language: CoffeeScript
- Size: 121 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
Phantom-Robot-Proxy
===================
Phantom JS Proxy for Robot / Crawler
This is a Connect module for serve page for web crawler. It help you handling SEO setting with modern web application using hashtag / HTML5 history API. Phantom-Robot-Proxy will grab User-Agent from headers and rediect crawler request to phantomjs with redis cache.
## Requirement
* PhantomJS 1.6 or higher
* Redis Server 2.6 or higher
* Node Modules:
* node-phantom
* redis
## Usage
```
phantomProxy = require('phantomProxy')
// Express App
app.use(phantomProxy.listen)
```
## Options
### Customerize User Agent Matcher
```
phantomProxy.setMatcher(/(googlebot|bingbot|baiduspider|slurp|bingpreview|msnbot)/gi)
```
## Improvement
* Add module Checker