{"id":20382882,"url":"https://github.com/zfb132/neu6_crawler","last_synced_at":"2025-07-25T15:10:05.086Z","repository":{"id":130412610,"uuid":"131674103","full_name":"zfb132/neu6_crawler","owner":"zfb132","description":"6维 统计爆楼帖","archived":false,"fork":false,"pushed_at":"2018-05-01T03:55:56.000Z","size":46,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-03-04T22:43:09.384Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":false,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zfb132.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-01T03:52:57.000Z","updated_at":"2020-10-21T03:27:55.000Z","dependencies_parsed_at":"2024-03-13T14:50:10.268Z","dependency_job_id":null,"html_url":"https://github.com/zfb132/neu6_crawler","commit_stats":{"total_commits":20,"total_committers":2,"mean_commits":10.0,"dds":"0.050000000000000044","last_synced_commit":"8eb6380fcb1b8da993f91d5a8753f5bfb5cb6152"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zfb132/neu6_crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zfb132%2Fneu6_crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zfb132%2Fneu6_crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zfb132%2Fneu6_crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zfb132%2Fneu6_crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zfb132","download_url":"https://codeload.github.com/zfb132/neu6_crawler/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zfb132%2Fneu6_crawler/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":267023220,"owners_count":24022915,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-07-25T02:00:09.625Z","response_time":70,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-15T02:19:26.736Z","updated_at":"2025-07-25T15:10:05.061Z","avatar_url":"https://github.com/zfb132.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# neu6_crawler\n6维 爆楼帖统计\n\n本项目基于[EasyLogin](https://github.com/zjuchenyuan/EasyLogin)\n\n目前可以爬取帖子每一页，进行分析，并输出回复前10名趋势图\n\n## 运行方法\n\n```\n# 依赖库安装\npip3 install -U requests[socks] bs4 -i https://pypi.doubanio.com/simple/ --trusted-host pypi.doubanio.com\n# 复制本项目\ngit clone https://github.com/zjuchenyuan/neu6_crawler\nmkdir -p cache\n# 将用户名username密码password要爬的帖子threadid写入config.py\nvim config.py\n# 执行爬取 也许你想修改爬取目标 自己改代码吧\npython3 gettopic.py\n# 输出分析结果\npython3 analysis_topic.py\n```\n\n## 代码文件结构\n\n- login.py 登录逻辑 基类NEU6\n- gettopic.py 爬取帖子 生成data_帖子id.pickle\n- analysis_topic.py 读取pickle文件 对爬取得到的数据进行分析: 回复数Top20用户，被引用Top10，自引用、神游、多次引用违规，最长引用链\n\n### 类、函数说明：\n\n```\nlogin.py\nclass NEU6(): \n    def __init__(username, password, login=True)\n        使用用户名密码进行登录，如果不需要自动登录可以login=False\n        使用EasyLogin提供的_import和export函数持久化cookie避免反复登录\n        登录前会调用islogin() 判断是否需要登录\n\ngettopic.py\nclass NEU6_TOPIC(NEU6):\n    def thread_page(self, threadid, pageid, extraid=1, usecache=True):\n        获得一个页面，返回[标题, 页面总数, 帖子的数组]\n        帖子的数组：[postid, floor, username, contenthtml, content, posttime, replytarget] 全部为str类型\n```","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzfb132%2Fneu6_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzfb132%2Fneu6_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzfb132%2Fneu6_crawler/lists"}