https://github.com/jofaval/album-scraper
Modularized python script for webscraping of images. Download an Album of Chapters blazingly fast
https://github.com/jofaval/album-scraper
album automation images multiprocessing multithreading python scraper scraping webscraping
Last synced: about 1 year ago
JSON representation
Modularized python script for webscraping of images. Download an Album of Chapters blazingly fast
- Host: GitHub
- URL: https://github.com/jofaval/album-scraper
- Owner: jofaval
- License: apache-2.0
- Created: 2022-11-26T15:08:35.000Z (over 3 years ago)
- Default Branch: master
- Last Pushed: 2023-07-03T21:48:43.000Z (about 3 years ago)
- Last Synced: 2025-03-29T12:15:02.848Z (over 1 year ago)
- Topics: album, automation, images, multiprocessing, multithreading, python, scraper, scraping, webscraping
- Language: Python
- Homepage: https://jofaval.github.io/album-scraper/
- Size: 462 KB
- Stars: 1
- Watchers: 2
- Forks: 0
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- License: LICENSE.md
Awesome Lists containing this project
README
# Album scraper
Modularized python script for webscraping images.
You can watch the changelog at [CHANGELOG](./CHANGELOG.md), it's an overall view, not greatly detailed.
## Contents
1. [Description](#description)
1. [Motivation](#motivation)
1. [History](#history)
1. [How does it work?](#how-does-it-work)
1. [Requirements](#requirements)
1. [Installation](#installation)
1. [Create a module](#create-a-module)
1. [Executing a module](#executing-a-module)
1. [What pages does it work on?](#what-pages-does-it-work-on)
1. [Legal notice](#legal-notice)
## Description
[Back to the contents](#contents)
Scrape the images from a website into an album formatting (file wise).
It's not perfect, it's straight to the point. It (somewhat) complies with Pylint, where useful
## Motivation
[Back to the contents](#contents)
It's really annoying to download and organize those images, for whatever that reason may be, and doing it all that work manually, too much mental overhead, what if you skipped one image? how do you check it? they're unsorted, name may be the default (filename on the host), will you manually rename them? Jow many will you miss while doing so? And many more...
Why act like a robot when a robot can do it for you?
## History
[Back to the contents](#contents)
Started more than a year ago, _at the time the project started, 2022_, and I applied little modifications here and there, iterating, and I finally decided to try and encapsulate it for a modular usage.
It started as an assignment, from which I was reminded of the existence of webscraping as a technique, and an opportunity to automate one routine I had arose. I expanded upon it until I've decided to publish it on open-source it.
## How does it work?
[Back to the contents](#contents)
### Requirements
[Back to the contents](#contents)
- Python 3.8.4 or higher
- not tested on lower, but it should work on >= 3.6.x
- Some decent internet speed, It worked nicely on 20 MB/s
Libraries:
- Pip
Or manually install the following:
Look at the [requirements.txt](./requirements.txt) file.
### Installation
[Back to the contents](#contents)
```bash
python -m pip install
# or pip3 install if you're on linux/unix systems
```
```bash
pip install -r requirements.txt # pip3 for any unix system
```
since python will be installed by default pip3
### Create a module
[Back to the contents](#contents)
Folow the [example](./modules/example/).
### Executing a module
[Back to the contents](#contents)
To execute it, simply create an `__init__.py` at the root for the [`modules`](./modules/) folder. And "export" the desired init function to execute, the `__main__.py` will handle the rest:
```python
from .example import init
```
And run the project, at the root:
```bash
python . # python3 for unix/linux
# or run the project folder
python album-scraper
```
Or watch out for the CLI, which might make it easier, and with more options
## What pages does it work on?
[Back to the contents](#contents)
Those without pagination, and with all the links you want to scrape visible at the homepage
_Page layouts, Image designed with [excalidraw](https://excalidraw.com/)_
## Legal notice
[Back to the contents](#contents)
I am not endorsing any illegal activities, images still hold their licensing and ownership by it's rightful author(s). If the robots.txt does not allow the officially legal webscraping of the website, any unrightful, mischievous or illegal act will still be illegal, and not my responsibility.
Use this scripts at your own responsibility.