{"id":13818834,"url":"https://github.com/thomashirtz/douban-crawler","last_synced_at":"2025-05-14T17:12:19.316Z","repository":{"id":114669425,"uuid":"298633467","full_name":"thomashirtz/douban-crawler","owner":"thomashirtz","description":"A simple crawler for retrieving information about movies or TV shows from the famous www.douban.com website.","archived":false,"fork":false,"pushed_at":"2021-02-05T10:44:06.000Z","size":128,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-14T17:12:15.368Z","etag":null,"topics":["crawler","douban"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/thomashirtz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-09-25T17:06:03.000Z","updated_at":"2024-02-01T16:15:20.000Z","dependencies_parsed_at":"2023-04-15T15:47:11.633Z","dependency_job_id":null,"html_url":"https://github.com/thomashirtz/douban-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomashirtz%2Fdouban-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomashirtz%2Fdouban-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomashirtz%2Fdouban-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/thomashirtz%2Fdouban-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/thomashirtz","download_url":"https://codeload.github.com/thomashirtz/douban-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254190418,"owners_count":22029634,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","douban"],"created_at":"2024-08-04T08:00:31.906Z","updated_at":"2025-05-14T17:12:19.288Z","avatar_url":"https://github.com/thomashirtz.png","language":"Python","funding_links":[],"categories":["Video Resources"],"sub_categories":["Novels in HTML"],"readme":"# douban-crawler\n\nThis repository contains a very simple crawler for retrieving movies from the site douban. This is a modified version of [this](https://github.com/jctian96/douban-web-crawler) web crawler. Therefore, all the credits go to [jctian96](https://github.com/jctian96). I decided to keep the core code of his crawler but simplify it to get the data crawled directly into dataframes. \n\n## Principles\n\nThe principle is to create a query on this webpage [https://movie.douban.com/j/new_search_subjects?](https://movie.douban.com/j/new_search_subjects?) with some search queries such as type, rating score, etc.  \nThe different queries can be found using this URL [https://movie.douban.com/tag/#/](https://movie.douban.com/tag/#/). By selecting a query it is possible to know how to change the URL to affine the research. To combine several tags, for example, '中国大陆' and '电视剧', a comma need to be used '中国大陆,电视剧'  \n\n### Tags\nthe different tags and tag categories are :\n* Types: 电影,电视剧,综艺,动画,纪录片,短片\n* Genres: 剧情,爱情,喜剧,科幻,动作,悬疑,犯罪,恐怖,青春,励志,战争,文艺,黑色,幽默,传记,情色,暴力,音乐,家庭\n* Region: 大陆,美国,香港,台湾,日本,韩国,英国,法国,德国,意大利,西班牙,印度,泰国,俄罗斯,伊朗,加拿大,澳大利亚,爱尔兰,瑞典,巴西,丹麦\n* Features: 经典,冷门佳片,魔幻,黑帮,女性\n\nIt does not seem possible to choose several tags per category. For example, it is not possible to choose \"电影,电视剧\", it will only search one of them (The first one I believe). However, it is possible to choose '电影,美国' since 'Movie and United States does not belong to the same category.\n\n### Sorting Style\nSeveral sorting styles exist:\n* Highest amount of rating: \"T\"\n* Highest rating: \"S\"\n* Latest release: \"R\"\n* Recently popular: \"U\"\n\n### Rating range\nCrawling a large amount of movies or crawling the most recent movies will lead to find more \"obscure\" or less known material that does not have any rating. In this case, using the query \"rating\" with a value superior to 0 will discard any material that doesn't have a rating. Therefore cleaning quite a bit the data obtained.\n\n## How to used the crawled data ?\nA [notebook](notebook.ipynb) is available to show how the data can be utilized afterward.\n\n![Dataframe](dataframe.jpg)\n\n## Notes\nMany pages on douban have some sort of mitigation to crawl. The page will not load before a certain action is taken. The use of [the previously mentioned link](https://movie.douban.com/j/new_search_subjects?) with search queries is very useful to avoid those issues.  \n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomashirtz%2Fdouban-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthomashirtz%2Fdouban-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomashirtz%2Fdouban-crawler/lists"}