https://github.com/HarryStevens/jquery-scrape

Use jQuery as your selection engine on the result of an HTTP/HTTPS request.
https://github.com/HarryStevens/jquery-scrape

Last synced: 18 days ago
JSON representation

Use jQuery as your selection engine on the result of an HTTP/HTTPS request.

• Host: GitHub
• URL: https://github.com/HarryStevens/jquery-scrape
• Owner: HarryStevens
• License: mit
• Created: 2019-06-05T15:43:49.000Z (over 5 years ago)
• Default Branch: master
• Last Pushed: 2022-12-09T05:19:11.000Z (almost 2 years ago)
• Last Synced: 2024-10-05T16:38:41.017Z (about 1 month ago)
• Language: JavaScript
• Homepage: https://www.npmjs.com/package/jquery-scrape
• Size: 81.1 KB
• Stars: 2
• Watchers: 2
• Forks: 0
• Open Issues: 5
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# jquery-scrape

Use jQuery as your selection engine on an HTTP request's response. [![Build Status](https://travis-ci.org/HarryStevens/jquery-scrape.svg?branch=master)](https://travis-ci.org/HarryStevens/jquery-scrape)

## Installation

```bash

npm i jquery-scrape -S

```

## Usage

jquery-scrape is a convenience wrapper for [request](https://github.com/request/request) and [cheerio](https://github.com/cheeriojs/cheerio). With almost no code, you can make an HTTP request and get back a jQuery selection engine. Here's a working example that you can run in your terminal:

```js

require("jquery-scrape")("https://www.example.com/", $ => {

  // Get the inner HTML of the DOM's body element.

  const html = $("body").html();

  console.log(html);

  // Get the href attribute of an anchor tag.

  const href = $("a").attr("href");

  console.log(href);

  // Traverse through each paragraph...

  $("p").each((i, p) => {

    // ...and get its text.

    const text = $(p).text().trim();

    console.log(text);

  });

});

```

Or suppose you want to scrape a table and save the result as JSON. Suppose the table is structured like [the one in this repo's test directory](https://github.com/HarryStevens/jquery-scrape/blob/master/test/test.html):

```html

  

    

      Number

      Value

    

  

  

    

      1

      Foo

    

    

      2

      Bar

          

  

```

Scrape it like so:

```js

require("jquery-scrape")("https://raw.githubusercontent.com/HarryStevens/jquery-scrape/master/test/test.html", $ => {

  let columns = [];

  $("th").each((i, th) => {

    columns.push($(th).text());

  });

  let data = [];

  const rows = $("tbody tr");

  console.log(`Found ${rows.length} rows. Scraping them...`);

  rows.each((rowIndex, row) => {

    let object = {};

    $(row).find("td").each((cellIndex, cell) => {

      object[columns[cellIndex]] = $(cell).text();

    });

    console.log(`${((rowIndex + 1) / rows.length * 100).toFixed(1)}%`);

    data.push(object);

  });

  console.log("Writing file: data.json");

  fs.writeFile("data.json", JSON.stringify(data), err => {

    if (err) throw err;

    console.log("File saved.");

    process.exit();

  });

});

```

## Tests

```bash

npm test

```