https://github.com/tuvimen/foxnews-scraper

A very simple shell script for scraping foxnews in json
https://github.com/tuvimen/foxnews-scraper

foxnews json reliq scraper shell

Last synced: 1 day ago
JSON representation

A very simple shell script for scraping foxnews in json

• Host: GitHub
• URL: https://github.com/tuvimen/foxnews-scraper
• Owner: TUVIMEN
• License: gpl-3.0
• Created: 2022-11-13T20:49:23.000Z (about 2 years ago)
• Default Branch: master
• Last Pushed: 2024-05-01T16:32:10.000Z (9 months ago)
• Last Synced: 2024-12-04T04:06:56.237Z (about 2 months ago)
• Topics: foxnews, json, reliq, scraper, shell
• Language: Shell
• Homepage:
• Size: 14.6 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# foxnews-scraper

A very simple shell script for scraping foxnews in json.

## Requirements

 - [reliq](https://github.com/TUVIMEN/reliq)

 - [jq](https://github.com/stedolan/jq)

 - parallel (not mandatory)

## Installation

    

    install -m 755 foxnews-scraper /usr/bin

## Json format

Here's [json](example.json).

## Usage

Script downloads all articles using sitemap.xml and writes it into stdout.

If you want to download it in parallel just uncomment the 3 comments and comment the for loop.