https://github.com/tuvimen/foxnews-scraper

A very simple shell script for scraping foxnews in json
https://github.com/tuvimen/foxnews-scraper

foxnews json reliq scraper shell

Last synced: 4 months ago
JSON representation

A very simple shell script for scraping foxnews in json

• Host: GitHub
• URL: https://github.com/tuvimen/foxnews-scraper
• Owner: TUVIMEN
• License: gpl-3.0
• Created: 2022-11-13T20:49:23.000Z (over 3 years ago)
• Default Branch: master
• Last Pushed: 2025-02-26T16:10:36.000Z (over 1 year ago)
• Last Synced: 2025-03-26T07:09:29.098Z (over 1 year ago)
• Topics: foxnews, json, reliq, scraper, shell
• Language: Shell
• Homepage:
• Size: 15.6 KB
• Stars: 2
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# foxnews-scraper

A very simple shell script for scraping foxnews in json.

## Requirements

 - [reliq](https://github.com/TUVIMEN/reliq)

 - [jq](https://github.com/stedolan/jq)

 - parallel (not mandatory)

## Installation

    

    install -m 755 foxnews-scraper /usr/bin

## Json format

Here's [json](example.json).

## Usage

Script downloads all articles using sitemap.xml and writes it into stdout.

If you want to download it in parallel just uncomment the 3 comments and comment the for loop.