https://github.com/soruly/anilist-crawler

Crawl data from anilist API and store in MariaDB.
https://github.com/soruly/anilist-crawler

anilist anime crawler

Last synced: 2 months ago
JSON representation

Crawl data from anilist API and store in MariaDB.

• Host: GitHub
• URL: https://github.com/soruly/anilist-crawler
• Owner: soruly
• License: mit
• Created: 2016-12-24T14:17:53.000Z (over 7 years ago)
• Default Branch: master
• Last Pushed: 2024-04-05T04:56:41.000Z (3 months ago)
• Last Synced: 2024-04-13T08:09:11.077Z (3 months ago)
• Topics: anilist, anime, crawler
• Language: JavaScript
• Homepage:
• Size: 561 KB
• Stars: 42
• Watchers: 3
• Forks: 4
• Open Issues: 0
• Metadata Files:
  • Readme: README.md
  • Funding: .github/FUNDING.yml
  • License: LICENSE

Lists

• awesome-acg - anilist-crawler - Crawl data from anilist API and store in elasticsearch. [English] (Downloaders)
• awesome-stars - soruly/anilist-crawler - Crawl data from anilist API and store in MariaDB. (JavaScript)

README

        
# anilist-crawler

[![License](https://img.shields.io/github/license/soruly/anilist-crawler.svg?style=flat-square)](https://github.com/soruly/anilist-crawler/blob/master/LICENSE)

[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/soruly/anilist-crawler/node.js.yml?style=flat-square)](https://github.com/soruly/anilist-crawler/actions)

Crawl data from [AniList](https://anilist.co/home) API and store as json in file system, MariaDB, or elasticsearch.

## Requirements

- Node.js 18.0+

- MariaDB 10.5+ (optional)

- elasticsearch 7.0+ (optional)

## How to use

1. Clone this repository

2. `npm install`

3. copy `.env.example` and rename to `.env`

4. Configure `.env` for your mariaDB, or elasticsearch, leave any of DB_HOST, ES_HOST, FS_DIR empty if you don't need it

## Examples

Fetch anime ID 123

`node index.js --anime 123`

Fetch all anime in page 240

`node index.js --page 240`

Fetch all anime from page 240 to 244 (inclusive)

`node index.js --page 240-244`

Fetch all anime from page 240 to the last page

`node index.js --page 240-`

Sometimes anime would be deleted from AniList, but it still exists locally in your database. You can use `--clean` to get a clean copy every time you start crawling.

`node index.js --clean --page 240-`

For details of AniList API please visit https://github.com/AniList/ApiV2-GraphQL-Docs/

You can try the interactive query tool here. https://anilist.co/graphiql

## Notes

- API request limit exceed (HTTP 429) has not been handled yet. With 60 requests/min per IP, it is unlikely to hit the limit with complex query.