{"id":13603895,"url":"https://github.com/Jinnrry/getAwayBSG","last_synced_at":"2025-04-11T22:31:56.131Z","repository":{"id":57658148,"uuid":"165990457","full_name":"Jinnrry/getAwayBSG","owner":"Jinnrry","description":"逃离北上广","archived":false,"fork":false,"pushed_at":"2019-09-05T15:10:49.000Z","size":698,"stargazers_count":1151,"open_issues_count":8,"forks_count":117,"subscribers_count":31,"default_branch":"master","last_synced_at":"2025-04-03T23:11:33.144Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"https://jinnrry.github.io/getAwayBSG/","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"agpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Jinnrry.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":"docs/.github/CODEOWNERS","security":null,"support":null}},"created_at":"2019-01-16T06:56:14.000Z","updated_at":"2025-03-09T03:23:12.000Z","dependencies_parsed_at":"2022-08-25T17:00:31.860Z","dependency_job_id":null,"html_url":"https://github.com/Jinnrry/getAwayBSG","commit_stats":null,"previous_names":["jiangwei1995910/getawaybsg"],"tags_count":15,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jinnrry%2FgetAwayBSG","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jinnrry%2FgetAwayBSG/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jinnrry%2FgetAwayBSG/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Jinnrry%2FgetAwayBSG/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Jinnrry","download_url":"https://codeload.github.com/Jinnrry/getAwayBSG/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248489734,"owners_count":21112629,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T19:00:35.935Z","updated_at":"2025-04-11T22:31:55.422Z","avatar_url":"https://github.com/Jinnrry.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# 逃离北上广\n\n\n[![Home](https://img.shields.io/badge/link-项目主页-brightgreen.svg)](https://jinnrry.github.io/getAwayBSG/)\n[![Link](https://img.shields.io/badge/link-python实现-blue.svg)](https://github.com/jinnrry/getAwayBSG/tree/python)\n[![Downloads](https://img.shields.io/github/downloads/jinnrry/getAwayBSG/total)](https://img.shields.io/github/downloads/jinnrry/getAwayBSG/total)\n[![forks](https://img.shields.io/github/forks/jinnrry/getAwayBSG?style=flat)](https://img.shields.io/github/forks/jinnrry/getAwayBSG?style=flat)\n[![starts](https://img.shields.io/github/stars/jinnrry/getAwayBSG)](https://img.shields.io/github/stars/jinnrry/getAwayBSG)\n[![license](https://img.shields.io/github/license/jinnrry/getAwayBSG)](https://img.shields.io/github/license/jinnrry/getAwayBSG)\n[![issues](https://img.shields.io/github/issues/jinnrry/getAwayBSG)](https://img.shields.io/github/issues/jinnrry/getAwayBSG)\n[![version](https://img.shields.io/github/release/jinnrry/getAwayBSG)](https://img.shields.io/github/release/jinnrry/getAwayBSG)\n\n\n\n\u003e **注意！**\\\n\u003e 1.本项目仅供学习研究，禁止用于任何商业项目\\\n\u003e 2.运行的时候为被爬方考虑下！尽量不要爬全站。请在配置文件中设置你需要的城市爬取即可！\\\n\u003e 3.[项目主页](https://jinnrry.github.io/getAwayBSG/)里面有现成数据，不需要你自己动手运行爬虫 \n\n\n## 啥？\n\n如果你是一个正准备逃离北上广等一线城市却又找不到去处的IT人士，或许这个项目能给你点建议。\n\n通过爬虫，抓取了链接、智联的工作，租房，二手房一系列数据，为你提供各城市的宏观分析数据\n\n## 安装\n\n从[releases](https://github.com/jinnrry/getAwayBSG/releases)下载对应操作系统，对应平台的二进制文件和配置文件模板\n\n## 配置\n\n打开配置文件你就知道了\n\n## 运行\n\n链家二手房数据抓取\n\n```\ngetAwayBSG -config=config.yaml -lianjia_ershou\n```\n\n链家租房数据抓取\n\n```\ngetAwayBSG -config=config.yaml -lianjia_zufang\n```\n\n智联招聘数据抓取\n\n```\ngetAwayBSG -config=config.yaml -zhilian\n```\n\n其他命令\n\n1.clean\n\n清除缓存状态，抓取过程中会将抓取进度保存到mongodb，每次启动会从上次位置继续抓取，如果你需要清除缓存状态，执行\n```\ngetAwayBSG -clean \n```\n该命令支持脚本调用\n```\ngetAwayBSG -clean [option]\n```\n\noption支持：lianjia_ershou、zhilian、lianjia_zufang\n\n2.info\n\n方便定时脚本记录抓取情况，使用info命令可以输出当前抓取数据量到文件\n\n```\ngetAwayBSG -info -info_save_to=./numLog.txt\n```\n\n使用-info_save_to参数指定文件保存位置，默认为当前目录的numLog.txt文件中\n\n3.help\n\n输出支持的全部命令列表\n\n```\ngetAwayBSG -help\n```\n\n\n## 数据分析\n\n分析用的MongoDB语句在[Query.js](./Query.js)文件中，使用MongoDB执行即可\n\n## 编译\n\n编译使用xgo，需要先安装docker\n\n```\ngit clone https://github.com/jinnrry/getAwayBSG\n\ndocker pull karalabe/xgo-latest\n\ngo get github.com/karalabe/xgo\n\ncd getAwayBSG\n\nsh ./build.sh\n```\n\n## 部署\n\n如果需要分布式或者多进程抓取，在不同机器或者多个进程中指定相同的MongoDB源即可，程序已经支持分布式多进程抓取了。已抓取的链接和状态会通过MongoDB共享\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJinnrry%2FgetAwayBSG","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FJinnrry%2FgetAwayBSG","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FJinnrry%2FgetAwayBSG/lists"}