https://github.com/tuvimen/xmbforum-scraper

A bash script for scraping xmb forums in json
https://github.com/tuvimen/xmbforum-scraper

json sciencemadness scraper xmb xmbforum

Last synced: 12 months ago
JSON representation

A bash script for scraping xmb forums in json

• Host: GitHub
• URL: https://github.com/tuvimen/xmbforum-scraper
• Owner: TUVIMEN
• License: gpl-3.0
• Created: 2024-03-09T15:08:15.000Z (over 2 years ago)
• Default Branch: master
• Last Pushed: 2024-05-01T16:34:02.000Z (about 2 years ago)
• Last Synced: 2024-05-02T11:25:51.243Z (about 2 years ago)
• Topics: json, sciencemadness, scraper, xmb, xmbforum
• Language: Shell
• Homepage:
• Size: 29.3 KB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
# Archive

Any further development has been transfered to [forumscraper](https://github.com/TUVIMEN/forumscraper).

# xmbforum-scraper

A bash script for scraping xmb forums in json.

## Requirements

 - [reliq](https://github.com/TUVIMEN/reliq)

 - [jq](https://github.com/stedolan/jq)

## Installation

    install -m 755 xmbforum-scraper /usr/bin

## Supported links formats

    http(s)?://forum.com/(.*/)?viewthread.php\?tid=([[:digit:]]+)

    http(s)?://forum.com/(.*/)?forumdisplay.php\?fid=([[:digit:]]+)

    http(s)?://forum.com/(.*/)?index.php\?gid=([[:digit:]]+)

    http(s)?://forum.com/.*

## Json format

Here's example of [thread](thread-example.json).

## Supported forums examples

    https://www.sciencemadness.org/whisper/

    https://forum.solbu.net

    https://forum.postcrossing.com

    https://forums.xmbforum2.com

    https://cieplodlawarszawy.pl

## Usage

    xmbforum-scraper [URL]...

Script downloads thread to files named by their id's.

Download forum into current directory using 4 processes

    xmbforum-scraper -p 4 'https://www.sciencemadness.org/whisper/forumdisplay.php?fid=9'

Download thread by irregular thread url into DIR

    xmbforum-scraper -d DIR -t 'https://forum.com/abcdef/loop/'

Download whole forum

    xmbforum-scraper -c 'https://forum.postcrossing.com/'

Get some help

    xmbforum-scraper -h