https://github.com/devansh-bhagania/nodejs_web_scraper_cheerio
A CHEERIO Node.js scraper designed to extract and save content from Wikipedia pages. It uses axios for making HTTP requests, cheerio for parsing HTML, and fs-extra for file operations.
https://github.com/devansh-bhagania/nodejs_web_scraper_cheerio
cheerio json nodejs scrapper
Last synced: 2 months ago
JSON representation
A CHEERIO Node.js scraper designed to extract and save content from Wikipedia pages. It uses axios for making HTTP requests, cheerio for parsing HTML, and fs-extra for file operations.
- Host: GitHub
- URL: https://github.com/devansh-bhagania/nodejs_web_scraper_cheerio
- Owner: Devansh-Bhagania
- License: mit
- Created: 2024-07-30T18:12:24.000Z (almost 2 years ago)
- Default Branch: devansh
- Last Pushed: 2024-07-30T18:20:18.000Z (almost 2 years ago)
- Last Synced: 2025-03-12T15:34:40.554Z (over 1 year ago)
- Topics: cheerio, json, nodejs, scrapper
- Language: JavaScript
- Homepage:
- Size: 60.5 KB
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# Wikipedia Scraper
This project is a simple Node.js scraper designed to extract and save content from Wikipedia pages. It uses `axios` for making HTTP requests, `cheerio` for parsing HTML, and `fs-extra` for file operations.
## Table of Contents
- [Overview](#overview)
- [How It Works](#how-it-works)
- [Setup and Installation](#setup-and-installation)
- [Usage](#usage)
- [Files and Directories](#files-and-directories)
- [Important Notes](#important-notes)
- [License](#license)
## Overview
This scraper fetches content from Wikipedia pages and saves the data in JSON format. It demonstrates how to perform web scraping, handle asynchronous operations, and manage file I/O in Node.js.
## How It Works
1. **Fetch Data**: The scraper uses `axios` to send HTTP GET requests to Wikipedia pages.
2. **Parse HTML**: The HTML content of the pages is parsed using `cheerio`, which allows easy extraction of relevant information.
3. **Extract Information**: The title of the page and its main content are extracted from the HTML.
4. **Save Data**: The extracted data is saved as JSON files in a specified directory using `fs-extra`.
## Setup and Installation
1. **Clone the Repository**
```bash
git clone https://github.com/your-username/wikipedia-scraper.git
cd wikipedia-scraper
```
2. **Installing dependencies**
```bash
npm install
```
3. **Update Scraper Configuration**
```bash
const PAGES = ['Node.js', 'JavaScript', 'Web_scraping'];
```
4. **Run the Scraper**
```bash
node scraper.js
```
After running the scraper, check the data directory for JSON files. Each file will contain the title and content of a specific Wikipedia page.