{"id":21250920,"url":"https://github.com/natlee/myanimelist-comment-crawler","last_synced_at":"2025-04-14T03:30:35.632Z","repository":{"id":146803032,"uuid":"200415382","full_name":"NatLee/MyAnimeList-comment-crawler","owner":"NatLee","description":"Crawl all reviews and infomation of Anime works on MyAnimeList. ;)","archived":false,"fork":false,"pushed_at":"2023-06-21T02:23:46.000Z","size":52,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-27T17:35:17.072Z","etag":null,"topics":["anime","crawler","data-analysis","data-mining","data-science","kaggle","kaggle-dataset","myanimelist","python","requests","scrapy-crawler","sqlite"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NatLee.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-08-03T19:51:46.000Z","updated_at":"2024-06-20T13:35:38.000Z","dependencies_parsed_at":null,"dependency_job_id":"8a55829f-8106-43e2-8414-4838eafd82ad","html_url":"https://github.com/NatLee/MyAnimeList-comment-crawler","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLee%2FMyAnimeList-comment-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLee%2FMyAnimeList-comment-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLee%2FMyAnimeList-comment-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NatLee%2FMyAnimeList-comment-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NatLee","download_url":"https://codeload.github.com/NatLee/MyAnimeList-comment-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248815449,"owners_count":21165928,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["anime","crawler","data-analysis","data-mining","data-science","kaggle","kaggle-dataset","myanimelist","python","requests","scrapy-crawler","sqlite"],"created_at":"2024-11-21T03:39:36.840Z","updated_at":"2025-04-14T03:30:35.612Z","avatar_url":"https://github.com/NatLee.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cdiv align=\"center\" style=\"text-align: center\"\u003e\n\n\u003cp style=\"text-align: center\"\u003e\n  \u003cimg align=\"center\" src=\"https://camo.githubusercontent.com/40d00cefb120a829517e503658aaf6c987d5f9cc6be5e2e35fb20bd63bdbceb5/68747470733a2f2f7363726170792e6f72672f696d672f7363726170796c6f676f2e706e67\" width=\"30%\" height=\"100%\"\u003e\n  \u003cimg align=\"center\" src=\"https://cdn.myanimelist.net/img/sp/icon/apple-touch-icon-256.png\" width=\"10%\" height=\"100%\"\u003e\n\u003c/p\u003e\n\n# MyAnimeList Crawler\nThis crawler can crawl all work information and reviews from `MyAnimeList` by using `Scrapy`.\n\n\u003c/div\u003e\n\n\n## Usage\n\n1. Ensure your settings in `./setting.ini`.\n2. Install the required dependencies.\n\n  ```bash\n  pip install -r requirements.txt\n  ```\n\n## Usage\n\n### Crawl Anime Infomation\n\n```bash\nscrapy runspider info_spider.py --nolog\n```\n\n- Example of crawling data\n  ```json\n    {\n      \"workId\": \"11979\",\n      \"url\": \"https://myanimelist.net/anime/11979/Mahou_Shoujo_Madoka%E2%98%85Magica_Movie_2__Eien_no_Monogatari\",\n      \"jpName\": \"劇場版 魔法少女まどか☆マギカ 永遠の物語\",\n      \"engName\": \"Puella Magi Madoka Magica the Movie Part 2: Eternal\",\n      \"synonymsName\": \"Mahou Shoujo Madoka Magika Movie 2, Magical Girl Madoka Magica Movie 2\",\n      \"workType\": \"Movie\",\n      \"episodes\": \"1\",\n      \"status\": \"Finished Airing\",\n      \"aired\": \"Oct 13, 2012\",\n      \"premiered\": \"\",\n      \"producer\": \"Aniplex, Mainichi Broadcasting System, Movic, Nitroplus, Houbunsha\",\n      \"broadcast\": \"\",\n      \"licensors\": \"Aniplex of America\",\n      \"studios\": \"Shaft\",\n      \"genres\": \"Drama\",\n      \"source\": \"Original\",\n      \"duration\": \"1 hr. 49 min.\",\n      \"rating\": \"PG-13 - Teens 13 or older\",\n      \"score\": \"8.37\",\n      \"allRank\": \"#197\",\n      \"popularityRank\": \"#1132\",\n      \"members\": \"195,001\",\n      \"favorites\": \"1,026\",\n      \"scoredByUser\": \"97097\",\n      \"lastUpdate\": \"2023-05-31 13:39:02\"\n  }\n  ```\n\n### Crawl Reviews\n\n\u003e Need crawl information of works at first because we need the list of Anime works in `myanimelist`.\n\n```bash\nscrapy runspider review_spider.py --nolog\n```\n\n## Link\n\nThe dataset is put on Kaggle.\n\n- Version 1 (2006/11 to 2019/06)\n  \n  [MALCoD](https://www.kaggle.com/natlee/myanimelist-comment-dataset)\n\n  Contains 130K commnets.\n\nAfter many years, the site updated. So I refactored this code to fit the new version of MyAnimeList.\n\n- Version 2 (2006/11 to 2023/06)\n  \n  [MALCoDv2](https://www.kaggle.com/natlee/myanimelist-comment-dataset-v2)\n\n  Contains 220K commnets.\n\nYou can obtain the data from your SQLite database by using the following command.\n```bash\npython db_to_kaggle.py\n```\n\n## Misc\n\nI recommend using [VisiData](https://www.visidata.org/) to see the SQLite database and the CSV files.\n\n![](https://d33wubrfki0l68.cloudfront.net/a2039fda848c76b90ee0270854cd417a82bbd60e/0b350/img/woq9dm5llq-590.webp)\n\nIt can see details of the structured data in CLI.\n\nJust use `sudo apt install VisiData` to get the package.\n\nIn our case, you can use `vd anime.db` to get a view with the SQLite database.\n\n## Contributor\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section --\u003e\n\u003c!-- prettier-ignore-start --\u003e\n\u003c!-- markdownlint-disable --\u003e\n\u003ctable\u003e\n  \u003ctbody\u003e\n    \u003ctr\u003e\n      \u003ctd align=\"center\"\u003e\u003ca href=\"https://github.com/NatLee\"\u003e\u003cimg src=\"https://avatars.githubusercontent.com/u/10178964?v=3?s=100\" width=\"100px;\" alt=\"Nat Lee\"/\u003e\u003cbr /\u003e\u003csub\u003e\u003cb\u003eNat Lee\u003c/b\u003e\u003c/sub\u003e\u003c/a\u003e\u003c/td\u003e\n    \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\n\u003c!-- markdownlint-restore --\u003e\n\u003c!-- prettier-ignore-end --\u003e\n\n\u003c!-- ALL-CONTRIBUTORS-LIST:END --\u003e\n\n## LICENSE\n\n[MIT](LICENSE)\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatlee%2Fmyanimelist-comment-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnatlee%2Fmyanimelist-comment-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnatlee%2Fmyanimelist-comment-crawler/lists"}