https://github.com/rchipka/node-osmosis

Web scraper for NodeJS
https://github.com/rchipka/node-osmosis

Last synced: 2 days ago
JSON representation

Web scraper for NodeJS

• Host: GitHub
• URL: https://github.com/rchipka/node-osmosis
• Owner: rchipka
• Created: 2015-02-07T02:17:26.000Z (about 10 years ago)
• Default Branch: master
• Last Pushed: 2023-12-13T04:18:37.000Z (about 1 year ago)
• Last Synced: 2025-02-06T01:54:29.235Z (9 days ago)
• Language: JavaScript
• Homepage:
• Size: 836 KB
• Stars: 4,114
• Watchers: 74
• Forks: 248
• Open Issues: 116
• Metadata Files:
  • Readme: Readme.md
  • Changelog: Changes.md

Awesome Lists containing this project

• awesome-nodejs - node-osmosis - HTML/XML parser and web scraper for Node.js.  (Repository / Crawler)

README

        
# Osmosis

HTML/XML parser and web scraper for NodeJS.

[![NPM](https://nodei.co/npm/osmosis.png)](https://www.npmjs.com/package/osmosis)

[![Build Status](https://travis-ci.org/rchipka/node-osmosis.svg)](https://travis-ci.org/rchipka/node-osmosis)

![Downloads](https://img.shields.io/npm/dm/osmosis.svg)

## Features

- Uses native libxml C bindings

- Clean promise-like interface

- Supports CSS 3.0 and XPath 1.0 selector hybrids

- [Sizzle selectors](https://github.com/jquery/sizzle/wiki#other-selectors-and-conventions),

  [Slick selectors](http://mootools.net/core/docs/1.6.0/Slick/Slick), and

  [more](https://github.com/rchipka/node-osmosis/blob/master/docs/Selectors.md)

- No large dependencies like jQuery, cheerio, or jsdom

- Compose deep and complex data structures

- HTML parser features

    - Fast parsing

    - Very fast searching

    - Small memory footprint

- HTML DOM features

    - Load and search ajax content

    - DOM interaction and events

    - Execute embedded and remote scripts

    - Execute code in the DOM

- HTTP request features

    - Logs urls, redirects, and errors

    - Cookie jar and custom cookies/headers/user agent

    - Login/form submission, session cookies, and basic auth

    - Single proxy or multiple proxies and handles proxy failure

    - Retries and redirect limits

## Example

```javascript

var osmosis = require('osmosis');

osmosis

.get('www.craigslist.org/about/sites')

.find('h1 + div a')

.set('location')

.follow('@href')

.find('header + div + div li > a')

.set('category')

.follow('@href')

.paginate('.totallink + a.button.next:first')

.find('p > a')

.follow('@href')

.set({

    'title':        'section > h2',

    'description':  '#postingbody',

    'subcategory':  'div.breadbox > span[4]',

    'date':         'time@datetime',

    'latitude':     '#map@data-latitude',

    'longitude':    '#map@data-longitude',

    'images':       ['img@src']

})

.data(function(listing) {

    // do something with listing data

})

.log(console.log)

.error(console.log)

.debug(console.log)

```

## Documentation

For documentation and examples check out [https://rchipka.github.io/node-osmosis/global.html](https://rchipka.github.io/node-osmosis/global.html)

## Dependencies

- [libxmljs-dom](https://github.com/rchipka/node-libxmljs-dom) - DOM wrapper for [libxmljs](https://github.com/libxmljs/libxmljs) C bindings

- [needle](https://github.com/tomas/needle) - Lightweight HTTP wrapper

## Donate

Please consider a donation if you depend on web scraping and Osmosis makes your job a bit easier.

Your contribution allows me to spend more time making this the best web scraper for Node.

[![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donate_LG.gif)](https://www.paypal.com/cgi-bin/webscr?item_name=node-osmosis&cmd=_donations&business=NAXMWBMWKUWUU)