https://github.com/soruly/anilist-crawler

Crawl data from anilist API and store in MariaDB.
https://github.com/soruly/anilist-crawler

anilist anime crawler

Last synced: 9 days ago
JSON representation

Crawl data from anilist API and store in MariaDB.

• Host: GitHub
• URL: https://github.com/soruly/anilist-crawler
• Owner: soruly
• License: mit
• Created: 2016-12-24T14:17:53.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2024-07-15T06:19:12.000Z (4 months ago)
• Last Synced: 2024-10-15T01:08:55.603Z (22 days ago)
• Topics: anilist, anime, crawler
• Language: JavaScript
• Homepage:
• Size: 527 KB
• Stars: 45
• Watchers: 3
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Awesome Lists containing this project

• awesome-acg - anilist-crawler - Crawl data from anilist API and store in elasticsearch. [English] (Downloaders)

README

        
# anilist-crawler

[![License](https://img.shields.io/github/license/soruly/anilist-crawler.svg?style=flat-square)](https://github.com/soruly/anilist-crawler/blob/master/LICENSE)

[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/soruly/anilist-crawler/node.js.yml?style=flat-square)](https://github.com/soruly/anilist-crawler/actions)

Crawl data from [AniList](https://anilist.co/home) API and store as json in file system, MariaDB, or elasticsearch.

## Requirements

- Node.js >= 20.12

- MariaDB >= 10.5 (optional)

- elasticsearch >= 7.0 (optional)

## How to use

1. Clone this repository

2. `npm install`

3. copy `.env.example` and rename to `.env`

4. Configure `.env` for your mariaDB, or elasticsearch, leave any of DB_HOST, ES_HOST, FS_DIR empty if you don't need it

## Examples

Fetch anime ID 123

`node index.js --anime 123`

Fetch all anime in page 240

`node index.js --page 240`

Fetch all anime from page 240 to 244 (inclusive)

`node index.js --page 240-244`

Fetch all anime from page 240 to the last page

`node index.js --page 240-`

Sometimes anime would be deleted from AniList, but it still exists locally in your database. You can use `--clean` to get a clean copy every time you start crawling.

`node index.js --clean --page 240-`

For details of AniList API please visit https://github.com/AniList/ApiV2-GraphQL-Docs/

You can try the interactive query tool here. https://anilist.co/graphiql

## Notes

- API request limit exceed (HTTP 429) has not been handled yet. With 60 requests/min per IP, it is unlikely to hit the limit with complex query.