Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/indix/web-auto-extractor
Automatically extracts structured information from webpages
https://github.com/indix/web-auto-extractor
Last synced: about 2 months ago
JSON representation
Automatically extracts structured information from webpages
- Host: GitHub
- URL: https://github.com/indix/web-auto-extractor
- Owner: indix
- License: mit
- Created: 2016-05-13T15:08:13.000Z (over 8 years ago)
- Default Branch: master
- Last Pushed: 2022-06-23T15:31:36.000Z (about 2 years ago)
- Last Synced: 2024-04-28T21:08:19.217Z (5 months ago)
- Language: JavaScript
- Size: 108 KB
- Stars: 108
- Watchers: 46
- Forks: 29
- Open Issues: 10
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Web Auto Extractor
[](https://travis-ci.org/indix/web-auto-extractor)
Parse semantically structured information from any HTML webpage.
Supported formats:-
- Encodings that support [Schema.org](http://schema.org/) vocabularies:-
- Microdata
- RDFa-lite
- JSON-LD
- Random Meta tags
Popularly, many websites mark up their webpages with Schema.org vocabularies for better SEO. This library helps you parse that information to JSON.
**[Demo](https://tonicdev.com/npm/web-auto-extractor)** it on tonicdev
## Installation
`npm install web-auto-extractor`
## [Usage](#usage)
```js
// IF CommonJS
var WAE = require('web-auto-extractor').default
// IF ES6
import WAE from 'web-auto-extractor'
var parsed = WAE().parse(sampleHTML)
```
Let's use the following text as the `sampleHTML` in our example. It uses Schema.org vocabularies to structure a Product information and is encoded in `microdata` format.
#### [Input](#input)
```html
ACME
Executive Anvil
Sleeker than ACME's Classic Anvil, the
Executive Anvil is perfect for the business traveler
looking for something to drop from a height.
Product #: 925872
4.4 stars, based on 89
reviews
Regular price: $179.99
$119.99
(Sale ends )
Available from:
Executive Objects
Condition: Previously owned,
in excellent condition
In stock! Order now!
```
#### [Output](#output)
Our `parsed` object should look like -
```json
{
"microdata": {
"Product": [
{
"@context": "http://schema.org/",
"@type": "Product",
"brand": "ACME",
"name": "Executive Anvil",
"image": "anvil_executive.jpg",
"description": "Sleeker than ACME's Classic Anvil, the\n Executive Anvil is perfect for the business traveler\n looking for something to drop from a height.",
"mpn": "925872",
"aggregateRating": {
"@context": "http://schema.org/",
"@type": "AggregateRating",
"ratingValue": "4.4",
"reviewCount": "89"
},
"offers": {
"@context": "http://schema.org/",
"@type": "Offer",
"priceCurrency": "USD",
"price": "119.99",
"priceValidUntil": "5 November!",
"seller": {
"@context": "http://schema.org/",
"@type": "Organization",
"name": "Executive Objects"
},
"itemCondition": "http://schema.org/UsedCondition",
"availability": "http://schema.org/InStock"
}
}
]
},
"rdfa": {},
"jsonld": {},
"metatags": {
"priceCurrency": [
"USD",
"USD"
]
}
}
```
The `parsed` object includes four objects - `microdata`, `rdfa`, `jsonld` and `metatags`. Since the above HTML does not have any information encoded in `rdfa` and `jsonld`, those two objects are empty.
## Caveat
I wouldn't call it a caveat but rather the parser is strict by design. It might not parse like expected if the HTML isn't encoded correctly, so one might assume the parser is broken.
For example, take the following HTML snippet.
```html
Ghostbusters
Black Rhino
Country: United States
```
The problem here is the `itemprop` - `productionCompany` which is of `itemtype` - `Organization` doesn't have any `itemprop` as its children, in this case - `name`.
The parser assumes every `itemtype` contains an `itemprop`, or every `typeof` contains a `property` in case of `rdfa`. So the `"Black Rhino"` information is lost.
It'll be nice to fix this by having a `non-strict` mode for parsing this information. PRs are welcome.
## License
MIT