Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/zlurad/serp-parser
Nodejs lib to parse Google SERP html pages
https://github.com/zlurad/serp-parser
google-search search-engines serp
Last synced: 3 months ago
JSON representation
Nodejs lib to parse Google SERP html pages
- Host: GitHub
- URL: https://github.com/zlurad/serp-parser
- Owner: zlurad
- License: mit
- Created: 2018-12-30T00:00:39.000Z (almost 6 years ago)
- Default Branch: master
- Last Pushed: 2023-07-27T20:13:35.000Z (over 1 year ago)
- Last Synced: 2024-07-28T11:30:53.018Z (3 months ago)
- Topics: google-search, search-engines, serp
- Language: HTML
- Homepage:
- Size: 22.2 MB
- Stars: 43
- Watchers: 9
- Forks: 12
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
- web-stuff - serp-parser - parse SERPs using this library, easy and simple. (JS/ES6)
README
# SERP Parser
[](https://travis-ci.org/zlurad/serp-parser) [](https://codecov.io/gh/zlurad/serp-parser)
serp-parser is small lib writen in typescript used to extract search engine rank position from the html.
## Instalation
```
npm i serp-parser
yarn add serp-parser
```
## Usage - Google SERP extraction
GoogleSERP accepts both html that is extracted with any headless browser lib (`puppeteer`, `phantomjs`...) that have enabled javascript as well as html page structure from no-js-enabled requests from for example `request` lib. For full enabled js html we use `GoogleSERP` class, and for nojs pages `GoogleNojsSERP` class.
With html from headless browser we use full `GoogleSERP` parser
```
import { GoogleSERP } from 'serp-parser'
const parser = new GoogleSERP(html);
console.dir(parser.serp);
```
Or on es5 with request lib, we get nojs Google results, so we use `GoogleNojsSERP` parser that is separate class in the lib
```
var request = require("request")
var sp = require("serp-parser")
request('https://www.google.com/search?q=google', function (error, response, html) {
if (!error && response.statusCode == 200) {
parser = new sp.GoogleNojsSERP(html);
console.dir(parser.serp);
}
});
```
It will return serp object with array of results with domain, position, title, url, cached url, similar url, link type, sitelinks and snippet
```
{
"keyword: "google",
"totalResults": 15860000000,
"timeTaken": 0.61,
"currentPage": 1,
"pagination": [
{ page: 1,
path: "" },
{ page: 2,
path: "/search?q=google&safe=off&gl=US&pws=0&nfpr=1&ei=N1QvXKbhOLCC5wLlvLa4Dg&start=10&sa=N&ved=0ahUKEwjm2Mn2ktTfAhUwwVkKHWWeDecQ8tMDCOwB" },
...
],
"videos": [
{ title: "The Matrix YouTube Movies Science Fiction - 1999 $ From $3.99",
sitelink: "https://www.youtube.com/watch?v=3DfOTKGvtOM",
date: 2018-10-28T23:00:00.000Z,
source: "YouTube",
channel: "Warner Movies On Demand",
videoDuration: "2:23" },
...
],
"thumbnailGroups": [
{ "heading": "Organization software",
"thumbnails:": [ {
"sitelink": "/search?safe=off&gl=US&pws=0&nfpr=1&q=Microsoft&stick=H4sIAAAAAAAAAONgFuLUz9U3MDFNNk9S4gAzi8tMtGSyk630k0qLM_NSi4v1M4uLS1OLrIozU1LLEyuLVzGKp1n5F6Un5mVWJZZk5ucpFOenlZQnFqUCAMQud6xPAAAA&sa=X&ved=2ahUKEwjm2Mn2ktTfAhUwwVkKHWWeDecQxA0wHXoECAQQBQ",
"title": "Microsoft Corporation"
},
...
]
},
...
],
"organic": [
{
"domain": "www.google.com",
"position": 1,
"title": "Google",
"url": "https://www.google.com/",
"cachedUrl": "https://webcache.googleusercontent.com/search?q=cache:y14FcUQOGl4J:https://www.google.com/+&cd=1&hl=en&ct=clnk&gl=us",
"similarUrl": "/search?safe=off&gl=US&pws=0&nfpr=1&q=related:https://www.google.com/+google&tbo=1&sa=X&ved=2ahUKEwjm2Mn2ktTfAhUwwVkKHWWeDecQHzAAegQIARAG",
"linkType": "HOME",
"sitelinks": [
{ "title": "Google Docs",
"snippet": "Google Docs brings your documents to life with smart ...",
"type": "card" },
{ "title": "Google News",
"snippet": "Comprehensive up-to-date news coverage, aggregated from ...",
"type": "card" },
...
],
"snippet": "Settings Your data in Search Help Send feedback. AllImages. Account · Assistant · Search · Maps · YouTube · Play · News · Gmail · Contacts · Drive · Calendar."
},
{
"domain": "www.google.org",
"position": 2,
"title": "Google.org: Home",
"url": "https://www.google.org/",
"cachedUrl": "https://webcache.googleusercontent.com/search?q=cache:Nm9ycLj-SKoJ:https://www.google.org/+&cd=24&hl=en&ct=clnk&gl=us",
"similarUrl": "/search?safe=off&gl=US&pws=0&nfpr=1&q=related:https://www.google.org/+google&tbo=1&sa=X&ved=2ahUKEwjm2Mn2ktTfAhUwwVkKHWWeDecQHzAXegQIDBAF",
"linkType": "HOME",
"snippet": "Data-driven, human-focused philanthropy powered by Google. We bring the best of Google to innovative nonprofits that are committed to creating a world that..."
},
...
],
"relatedKeywords": [
{ keyword: google search,
path: "/search?safe=off&gl=US&pws=0&nfpr=1&q=google+search&sa=X&ved=2ahUKEwjm2Mn2ktTfAhUwwVkKHWWeDecQ1QIoAHoECA0QAQ" },
{ keyword: google account,
path: "/search?safe=off&gl=US&pws=0&nfpr=1&q=google+account&sa=X&ved=2ahUKEwjm2Mn2ktTfAhUwwVkKHWWeDecQ1QIoAXoECA0QAg" },
...
]
}
```
## Usage - Bing SERP extraction
**Note: Only BingNojsSerp is implemented so far.**
BingSERP works the same as GoogleSerp. It accepts both html that is extracted with any headless browser lib (`puppeteer`, `phantomjs`...) that have enabled javascript as well as html page structure from no-js-enabled requests from for example `request` lib. For full enabled js html we use `BingSERP` class, and for nojs pages `BingNojsSERP` class.
With html from headless browser we use full `BingSERP` parser
```
import { BingNojsSERP } from 'serp-parser'
const parser = new BingNojsSERP(html);
console.dir(parser.serp);
```
Or on es5 with request lib, we get nojs Bing results, so we use `BingNojsSERP` parser that is separate class in the lib
```
var request = require("request")
var sp = require("serp-parser")
request('https://www.bing.com/search?q=bing', function (error, response, html) {
if (!error && response.statusCode == 200) {
parser = new sp.BingNojsSERP(html);
console.dir(parser.serp);
}
});
```
## Parse options - specify what features to parse
On each module you can specify the parse options as a second parameter to only parse SERP features that you want, for `GoogleSERP` here are all the options:
```
// Options for GoogleSERP
#DEF_OPTIONS = {
organic: true,
related: true,
pagination: true,
ads: true,
hotels: true,
videos: true,
thumbnails: true,
shop: true,
stories: true,
locals: true,
};
// Options for GoogleNojsSERP and BingNojsSERP
#DEF_OPTIONS = {
organic: true,
related: true,
ads: true,
hotels: true,
};
```
Usage:
In this example only organic feature will be parsed
```
import { GoogleSERP } from 'serp-parser'
const parser = new GoogleSERP(html, {organic: true});
console.dir(parser.serp);
```
## Roadmap
We are working on enriching parsed data to grab all existing and new SERP features from Google search page results, as well as knowlege graph, ads, and any related info. Also we will add more Search engines along the way.
Anyone willing to help - please submit issues, feature requests, fork away and send PR's
## SERP Features Docs
### More Results Button
**SERP.moreResults**
Gives back the reference to the more results button which sometimes is shown instead of the regular google pagination
[More Results Button](/examples/MoreResultsButton.PNG)
### Card sitelinks
**SERP.sitelinks**
Gives back the reference to the links that are shown at the bottom of a search result card
[Card Sitelinks](/examples/Sitelinks.PNG)
### Top Stories
**SERP.topStoriesFeature**
Gives back the reference to the top stories
[Top Stories](/examples/TopStories.PNG)
### Featured Snippet
**SERP.snippet**
Gives back the reference to the featured snippet boxes
[Featured Snippet](/examples/FeaturedSnippet.PNG)
### Videos
**SERP.videos**
Gives back the reference to the videos related to the searched keyword
[Videos](/examples/Videos.PNG)
### Trailers And Clips
**SERP.trailersAndClips**
Gives back the reference to the trailers and clips related to the searched keyword - usually associated with movies
[Trailers And Clips](/examples/TrailersAndClips.PNG)