{"id":15659688,"url":"https://github.com/ysh329/douban-crawler","last_synced_at":"2025-06-13T10:08:47.732Z","repository":{"id":83526840,"uuid":"65141908","full_name":"ysh329/Douban-Crawler","owner":"ysh329","description":"抓取豆瓣小组相关信息（小组、用户、帖子）。","archived":false,"fork":false,"pushed_at":"2016-10-17T02:53:45.000Z","size":29,"stargazers_count":26,"open_issues_count":1,"forks_count":8,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-06-13T10:03:35.976Z","etag":null,"topics":["crawler","douban","douban-crawler"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ysh329.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-08-07T16:02:10.000Z","updated_at":"2025-03-18T19:49:19.000Z","dependencies_parsed_at":null,"dependency_job_id":"6c1d6c0c-5f4f-4b78-ae91-d0498350f386","html_url":"https://github.com/ysh329/Douban-Crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ysh329/Douban-Crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysh329%2FDouban-Crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysh329%2FDouban-Crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysh329%2FDouban-Crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysh329%2FDouban-Crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ysh329","download_url":"https://codeload.github.com/ysh329/Douban-Crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ysh329%2FDouban-Crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":259624724,"owners_count":22886328,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","douban","douban-crawler"],"created_at":"2024-10-03T13:18:14.828Z","updated_at":"2025-06-13T10:08:47.699Z","avatar_url":"https://github.com/ysh329.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Douban-Crawler\n指定小组查询关键词，完全傻瓜式获取指定个数豆瓣小组详细信息，以及小组下指定个数帖子信息。\n\n## CRAWLER库表结构\n三个数据表：小组表，用户表，帖子表。\n\n### GROUP表\n注备：豆瓣小组、贴吧\n\n##### 小组基本信息\n\n|字段名|类型|含义|举例|\n|:---:|:---:|:---:|:---:|\n|GROUP_SOURCE|VARCHAR(10)|数据来源|\"douban\"或\"tieba\"|\n|GROUP_QUERY|VARCHAR(20)|查询query(类似GROUP_TAG)|\"北京,IT\"|\n|GROUP_NAME|VARCHAR(30)|组名、吧名|\"北京读书交友会\"|\n|GROUP_ID|VARCHAR(20)|全站唯一性ID|\"576850\"|\n|GROUP_MEMBER_NUM|INT|小组人数|300|\n|GROUP_URL|TEXT|地址|\"https://www.douban.com/group/10274/\"|\n|GROUP_INTRO|TEXT|介绍|\"小组介绍内容\"|\n|GROUP_CREATE_DATE|VARCHAR(10)|小组创建时间|2010-10-10|\n|GROUP_TAG|VARCHAR(20)|小组标签|\"北京,读书,交友\"|\n\n##### 活跃度基本信息(每天字段更新)  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|    \n|CURRENT_DAY_POST_NUM|INT|小组当天总帖数|100|  \n|CURRENT_DAY_COMMENT_NUM|INT|当天帖子累计总回复数|5000|\n|CURRENT_DAY_AVE_COMMENT_NUM|INT|当天帖子累计总回复数|50|      \n\n##### 管理员基本信息  \n\n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|   \n|ADMIN_NAME|VARCHAR(50)|管理员姓名|\"章小希\"|  \n|ADMIN_ID|VARCHAR(20)|全站唯一性ID(豆瓣唯一ID、贴吧唯一ID)|\"148647315\"|  \n|ADMIN_URL|TEXT|管理员账号页面|\"https://www.douban.com/people/148647315/\"|\n  \n##### 表更新时间(定期更新)  \n\n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|   \n|TABLE_UPDATE_DATE|VARCHAR(16)|最后一次表更新时间|\"2015-11-19 21:04:48\"|\n\n### POST表  \n\n##### 来源和所在小组基本信息  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|   \n|GROUP_SOURCE|VARCHAR(10)|小组来源|\"douban\"或\"tieba\"|\n|GROUP_URL|TEXT|小组地址链接|\"https://www.douban.com/group/551307/\"|\n|GROUP_ID|VARCHAR(20)|所在来源的全(站)局唯一性ID|\"hangzhougonglue\"|  \n|GROUP_NAME|VARCHAR(30)|小组名称|\"杭州旅游\"|  \n\n##### 帖子基本信息  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|    \n|POST_URL|TEXT|帖子链接|\"https://www.douban.com/group/topic/88272843/\"|\n|POST_TITLE|TEXT|帖子标题|\"这是标题\"|\n|POST_ID|VARCHAR(10)|帖子唯一性ID|\"850407300\"|\n|POST_CREATE_DATE|VARCHAR(19)|帖子创建时间|\"2014-08-10 16:58:21\"|  \n|POST_LAST_COMMENT_DATE|VARCHAR(16)|帖子最后回复时间|\"2015-08-13 15:22\"|  \n|POST_COMMENT_NUM|INT|帖子回复个数|10|\n|POST_LIKE_NUM|INT|喜欢人数|10|\n\n##### 帖子创建者基本信息  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|    \n|POST_AUTHOR_NAME|VARCHAR(50)|帖子创建者名称|\"章小希\"|  \n|POST_AUTHOR_ID|VARCHAR(10)|帖子创建者全站唯一性ID|\"148647315\"|  \n|POST_AUTHOR_SIGNATURE|TEXT|签名|\"目标，前进；一切只为生活\"|  \n|POST_AUTHOR_URL|TEXT|帖子创建者个人页面地址|\"https://www.douban.com/people/148647315/\"|\n\n##### 内容和评论  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|    \n|POST_CONTENT|TEXT|帖子内容|\"这是帖子内容\"|\n|POST_IMG_NUM|INT|图片张数|3|\n|POST_IMG_URL_LIST|TEXT|所有图片地址列表的字符串(用\"\\t\"拼接)|'www.1.com/1.png::www.1.com/2.jpg'|\n|POST_AUTHOR_COMMENT|TEXT|帖子创建者的所有评论|\"这是评论1+2+3拼接起来的结果\"|\n|POST_AUTHOR_COMMENT_NUM|INT|帖子创建者的评论个数|10|\n\n##### 感兴趣信息(需要提取/抽取)  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|    \n|POST_CONTENT_QQ|VARCHAR(12)|帖子内容里的QQ号|\"12345\"|  \n|POST_CONTENT_WECHAT|VARCHAR(16)|帖子内容里的微信号|\"12345\"|  \n|POST_CONTENT_TEL|VARCHAR(15)|帖子内容里的电话号|\"13312345678\"|  \n|POST_CONTENT_ADDRESS|VARCHAR(30)|帖子内容里的地址|\"北京市海淀区\"|\n\n### USER表  \n注备:主要用来记录小组(或贴吧)管理员和发帖人个人信息  \n##### 基本信息  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|  \n|USER_SOURCE|VARCHAR(10)|用户来源|\"douban\"或\"tieba\"|  \n|USER_NAME|TEXT|用户名、昵称|\"小豆芽\"|  \n|USER_ID|VARCHAR(20)|全站唯一性ID|\"yncyd\"|  \n|USER_SEX|INT|性别|0(未知)或1(男)或2(女)|    \n|USER_SIGNATURE|TEXT|签名|\"目标，前进；一切只为生活\"|  \n|USER_URL|TEXT|个人页面|\"https://www.douban.com/people/yncyd/\"|  \n  \n##### 发帖情况(定期更新)  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|  \n|POST_NUM|INT|发帖总数|32|  \n|POST_LAST_CREATE_DATE|VARCHAR(16)|用户发帖目录页第1页最后一次发帖日期|\"2015-01-01 11:11\"|  \n|POST_MIDDLE_CREATE_DATE|VARCHAR(16)|用户发帖目录页第1页中间一次发帖日期|\"2015-01-01 11:11\"|  \n|POST_FIRST_CREATE_DATE|VARCHAR(16)|用户发帖目录页第1页第一次发帖日期|\"2015-01-01 11:11\"|\n\n##### 活跃程度(定期更新)  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|  \n|USER_LAST_LOGIN|VARCHAR(16)|上次登陆时间|\"2015-01-01 11:11\"|  \n|USER_CREATE_DATE|VARCHAR(12)|用户创建日期|\"2015-01-01\"|    \n\n##### 感兴趣信息(需要提取/抽取)  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|  \n|USER_QQ|VARCHAR(12)|QQ号码|\"111111111\"|  \n|USER_WECHAT|VARCHAR(16)|微信号|\"ZhangSan0912\"|  \n|USER_TEL|VARCHAR(15)|手机号|\"13311111111\"|  \n|USER_MAIL|TEXT|邮箱|\"zhangsan0912@gmail.com\"|  \n|USER_ADDRESS|TEXT|所在位置|\"北京市海淀区XXX路XXX号\"或\"XXX区\"或\"南京\"|\n\n##### 表更新时间(定期更新)  \n|字段名|类型|含义|举例|  \n|:-:|:-:|:-:|:-:|\n|TABLE_UPDATE_DATE|VARCHAR(16)|最后一次表更新时间|\"2015-11-19 21:04:48\"|\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fysh329%2Fdouban-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fysh329%2Fdouban-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fysh329%2Fdouban-crawler/lists"}