Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/znitche/ao3-web-reader

ao3 scraper / reader
https://github.com/znitche/ao3-web-reader

ao3 archiveofourown beautifulsoup4 flask python3 webscraper

Last synced: 2 months ago
JSON representation

ao3 scraper / reader

Awesome Lists containing this project

README

        

## AO3 Web Reader

AO3 web reader / scraper app created using Flask, SQLite, Redis and Bootstrap, supporting:
- Accounts authentication.
- Getting works from AO3.
- Grouping works using tags.
- Adding works to favorites.
- Auto-update of added works.
- Downloading works.
- Force updating chapters, works and tags.
- Exporting works for ebook readers (html format).

Hello there, It's been almost 2 years since I started working on this project, a lot of time right?

Anyway since then I became much better programmer, and despite that this project works pretty good
I decided that it is time for major refactoring which will ensure easier maintenance further down the road.

And here we are, below you can find a list of things that have been done.

1. Complete dependencies overhaul and proper fragmentation.
2. Alembic based database migrations.
3. Database reimplementation (replaced Flask-SQLAlchemy with sqlalchemy) + custom pagination
(since sqlalchemy doesn't support page base pagination).
4. New session based authentication system (removed Flask-Login).
5. Background tasks refactoring (dev mode app auto reloading finally works).
6. Proper logging implementation.
7. Styling tweaks + UI bugfixes.
8. Internal app data flow refactoring, to improve readability.
9. Improved Docker services architecture.

### Production Setup
1. Clone this repo.
2. Create `.env` config file and change config values (`DB_PATH` and `LOGS_PATH`).
```
cp .env.template .env
```
3. Run docker container.
```
sudo docker compose up -d
```

### Dev Setup
1. Clone this repo.
2. Generate `.env` config file.
```
cp .env.template .env
```
3. Change `REDIS_SERVER_ADDRESS` in `.env` to `127.0.0.1`
4. Install development dependencies
```
pip3 install -r requirements/requirements-dev.txt
```
5. Run DEV docker-compose.
```
sudo docker compose -f docker-compose-dev.yml up
```

### Database Migrations
```
python3 migrate.py
```

### Accounts Management
1. Bash into container.
```
sudo docker container exec -it ao3_web_reader bash
```
2. Run accounts manager cli `python3 users_manager_cli.py`.

### Tests
App contains some example tests for available routes. To run them:
```
pytest -v tests/
```

### Extras
Back when I was learning Django I created this repo [reimplementation](https://github.com/zNitche/ao3-web-reader-django).

Keep in mind that it hasn't been updated since it was finished and might not work anymore.