https://github.com/tuvimen/invision-scraper

A bash script for scraping invision forums in json
https://github.com/tuvimen/invision-scraper

forums invision json scraper

Last synced: 3 months ago
JSON representation

A bash script for scraping invision forums in json

• Host: GitHub
• URL: https://github.com/tuvimen/invision-scraper
• Owner: TUVIMEN
• License: gpl-3.0
• Created: 2023-05-16T17:43:55.000Z (almost 2 years ago)
• Default Branch: master
• Last Pushed: 2024-05-03T04:31:15.000Z (10 months ago)
• Last Synced: 2024-05-03T10:43:29.562Z (10 months ago)
• Topics: forums, invision, json, scraper
• Language: Shell
• Homepage:
• Size: 29.3 KB
• Stars: 1
• Watchers: 2
• Forks: 1
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Archive

Any further development has been transfered to [forumscraper](https://github.com/TUVIMEN/forumscraper).

# invision-scraper

A bash script for scraping invision forums in json.

## Requirements

 - [reliq](https://github.com/TUVIMEN/reliq)

 - [jq](https://github.com/stedolan/jq)

## Installation

    install -m 755 invision-scraper /usr/bin

## Supported links formats

    http(s)?://forum.com/topic/.*

    http(s)?://forum.com/forum(s)?/topic/.*

    http(s)?://forum.com/forums/.*

    http(s)?://forum.com/forum(s)?/forums/.*

    http(s)?://forum.com/.*/forums/.*

    http(s)?://forum.com.*

## Json format

Here's example of [topic](topic-example.json), and [user](user-example.json).

## Supported forums examples

    https://invisioncommunity.com/forums/

    https://linustechtips.com/

    https://www.thehuntinglife.com/forums/

    https://forum.htc.com/

## Usage

    invision-scraper [URL]...

Script downloads pages of threads and users and writes them in files. Files are named by their id's, but user files start with 'm-'.

Download forum into current directory using 4 processes

    invision-scraper -p 4 'https://forum.com/forums/19-name/'

Download thread by irregular thread url into DIR

    invision-scraper -d DIR -t 'https://forum.com/abcdef/loop/'

Download forum quicker by ignoring user profiles and reactions

    invision-scraper -N -n -c 'https://forum.com/forums/82-jus/'

Download whole forum

    invision-scraper -c 'https://forum.com/forums/'

Get some help

    invision-scraper -h