{"id":13826741,"url":"https://github.com/golang-collection/go-crawler-distributed","last_synced_at":"2025-07-09T01:31:35.539Z","repository":{"id":59075817,"uuid":"288061828","full_name":"golang-collection/go-crawler-distributed","owner":"golang-collection","description":"分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。","archived":false,"fork":false,"pushed_at":"2024-04-19T12:23:36.000Z","size":94952,"stargazers_count":44,"open_issues_count":2,"forks_count":6,"subscribers_count":6,"default_branch":"master","last_synced_at":"2024-08-05T09:15:41.069Z","etag":null,"topics":["crawler","docker","elasticsearch","go","go-micro","gocrawler","microservice","rabbitmq"],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/golang-collection.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2020-08-17T02:06:14.000Z","updated_at":"2024-07-26T16:26:19.000Z","dependencies_parsed_at":"2024-01-15T15:47:47.001Z","dependency_job_id":null,"html_url":"https://github.com/golang-collection/go-crawler-distributed","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golang-collection%2Fgo-crawler-distributed","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golang-collection%2Fgo-crawler-distributed/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golang-collection%2Fgo-crawler-distributed/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/golang-collection%2Fgo-crawler-distributed/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/golang-collection","download_url":"https://codeload.github.com/golang-collection/go-crawler-distributed/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":225476383,"owners_count":17480215,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","docker","elasticsearch","go","go-micro","gocrawler","microservice","rabbitmq"],"created_at":"2024-08-04T09:01:43.395Z","updated_at":"2024-11-20T05:31:08.145Z","avatar_url":"https://github.com/golang-collection.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# go-crawler-distributed\n\u003cdiv align=\"center\"\u003e\n\u003cimg border=\"0\" src=\"https://camo.githubusercontent.com/54fdbe8888c0a75717d7939b42f3d744b77483b0/687474703a2f2f6a617977636a6c6f76652e6769746875622e696f2f73622f69636f2f617765736f6d652e737667\" /\u003e\n\u003cimg border=\"0\" src=\"https://camo.githubusercontent.com/1ef04f27611ff643eb57eb87cc0f1204d7a6a14d/68747470733a2f2f696d672e736869656c64732e696f2f7374617469632f76313f6c6162656c3d254630253946253843253946266d6573736167653d496625323055736566756c267374796c653d7374796c653d666c617426636f6c6f723d424334453939\" /\u003e\n\u003ca href=\"https://github.com/SuperSupeng\"\u003e     \u003cimg border=\"0\" src=\"https://camo.githubusercontent.com/41e8e16b771d56dd768f7055354613254961d169/687474703a2f2f6a617977636a6c6f76652e6769746875622e696f2f73622f6769746875622f677265656e2d666f6c6c6f772e737667\" /\u003e \u003c/a\u003e \n\u003ca href=\"https://github.com/golang-collection/go-crawler-distributed/issues\"\u003e     \u003cimg border=\"0\" src=\"https://img.shields.io/github/issues/golang-collection/go-crawler-distributed\" /\u003e \u003c/a\u003e\n\u003ca href=\"https://github.com/golang-collection/go-crawler-distributed/network/members\"\u003e     \u003cimg border=\"0\" src=\"https://img.shields.io/github/forks/golang-collection/go-crawler-distributed\" /\u003e \u003c/a\u003e\n\u003ca href=\"https://github.com/golang-collection/go-crawler-distributed/stargazers\"\u003e     \u003cimg border=\"0\" src=\"https://img.shields.io/github/stars/golang-collection/go-crawler-distributed\" /\u003e \u003c/a\u003e\n\u003ca href=\"https://github.com/golang-collection/go-crawler-distributed/blob/master/LICENSE\"\u003e     \u003cimg border=\"0\" src=\"https://img.shields.io/github/license/golang-collection/go-crawler-distributed\" /\u003e \u003c/a\u003e\n\u003ca href=\"https://github.com/golang-collection/Urban-computing-papers/blob/master/wechat.md\"\u003e     \u003cimg border=\"0\" src=\"https://camo.githubusercontent.com/013c283843363c72b1463af208803bfbd5746292/687474703a2f2f6a617977636a6c6f76652e6769746875622e696f2f73622f69636f2f7765636861742e737667\" /\u003e \u003c/a\u003e\n\u003c/div\u003e\n\n---\n\nGithub：[https://github.com/golang-collection/go-crawler-distributed](https://github.com/golang-collection/go-crawler-distributed)\n\nDistributed crawler projects, the project supports personalization page parser secondary development, the whole project using micro service architecture, messages are sent asynchronously through message queues, Use the following framework: redigo, gorm, goquery, easyjson, viper, closer, zap, go-micro, and containable deployment is realized through Docker, intermediate crawler nodes support horizontal expansion.\n\n分布式爬虫项目，本项目支持个性化定制页面解析器二次开发，项目整体采用微服务架构，通过消息队列实现消息的异步发送，使用到的框架包括：redigo, gorm, goquery, easyjson, viper, amqp, zap, go-micro，并通过Docker实现容器化部署，中间爬虫节点支持水平拓展。\n\n# 目录结构\n```go\n- configs 存放配置文件\n- dependencies 项目依赖的环境相关的docker-compose文件\n- deploy 脚本\n    - buildScript 部署脚本\n    - deploy 直接启动项目的脚本\n    - dockerBuildScript 构建docker镜像\n    - service 用于存放服务的Dockerfile\n    - crawlOperation 除存储外统一爬虫处理模块\n- global 存放全局统一配置\n- init 项目初始化\n    - 使用方式 在引入包前面添加 _\n- internal 内部包\n    - crawler 处理爬虫相关逻辑\n        - crawerConfig 爬虫对应的消息队列配置\n        - douban 豆瓣网页解析\n        - fetcher 网页抓取\n        - meituan 美团网页解析\n        - persistence 定义用于保存数据的结构体\n        - worker 具体的工作逻辑，通过它实现代码的解耦\n    - dao 与数据库交互\n    - model 模型\n- pkg 通用包\n    - cache 缓存相关 redis\n    - consistentHash 一致性Hash\n    - db 数据库操作\n    - elastic ElasticSearch操作\n    - email 邮件相关\n    - errcode 全局统一错误配置\n    - file 文件相关\n    - idGenerator snowflake生成全局ID\n    - logger 日志\n    - mq 消息队列\n    - setting 对应配置文件\n    - tracer 调用链追踪\n    - upload 文件上传\n    - util 相关工具\n- service 微服务\n    - cache redis微服务通过grpc操作\n      - proto 定义grpc的proto文件\n      - server 定义grpc的服务端\n    - douban 豆瓣相关服务\n      - crawl_tags 用于爬取豆瓣的[tags](https://book.douban.com/tag/)，此页面为爬虫的起始界面\n      - crawl_list 用于爬取豆瓣的[list](https://book.douban.com/tag/%E5%B0%8F%E8%AF%B4)\n      - crawl_detail 用于爬取豆瓣的图书具体内容[detail](https://book.douban.com/subject/25955474/)\n      - storage_detail 用于存储豆瓣的图书具体内容，存储到MySQL中\n    - elastic elasticSearch微服务通过grpc操作\n      - proto 定义grpc的proto文件\n      - server 定义grpc的服务端\n    - meituan 美团相关服务\n      - crawl_tags 用于爬取美团的[list](https://tech.meituan.com/)，此页面为爬虫的起始界面\n      - crawl_list 用于爬取美团的[urllist](https://tech.meituan.com/page/2.html)\n      - crawl_detail 用于爬取美团技术文章的页面具体内容[detail](https://tech.meituan.com/2020/04/23/octo-watt.html)\n      - storage_detail 用于存储美团技术文章的具体内容，存储到ElasticSearch中\n```\n\n# 配置文件\nYou need to customize your configuration. Create the config.json file under the configs folder in the project root directory or add a config in Consul.\n\nThe sample of config.json.\n```json\n{\n  \"mysql\": {\n    \"user\": \"root\",\n    \"password\": \"example\",\n    \"host\": \"127.0.0.1:3306\",\n    \"db_name\": \"short_story\"\n  },\n  \"redis\": {\n    \"host\": \"127.0.0.1:6379\"\n  },\n  \"rabbitmq\": {\n    \"user\": \"guest\",\n    \"password\": \"guest\",\n    \"host\": \"127.0.0.1:5672\"\n  },\n  \"elastic\": {\n    \"url\": \"http://127.0.0.1:9200\",\n    \"index\": \"article\"\n  }\n}\n```\n\nViper reads the remote configuration file.\n\n![config](./img/consul_config.png)\n\n```go\nfunc init() {\n\terr := viper.AddRemoteProvider(\"consul\", config.ConsulURL, config.ConsulConfigPath)\n\tif err != nil {\n\t\tlogger.Error(\"read config\",zap.Error(err))\n\t\treturn\n\t}\n\tviper.SetConfigType(\"json\") // Need to explicitly set this to json\n\tif err := viper.ReadRemoteConfig(); err != nil {\n\t\tlogger.Error(\"read config\",zap.Error(err))\n\t\treturn\n\t}\n}\n```\n\nMore detail: [watchConfig](./service/watchConfig)\n\n# Parser\n\n## [douban](./crawer/douban)\n\nParser：[parser](./crawler/douban/parser) uses css selectors for page parsing.\n\n![douban](./img/douban.png)\n\n## [meituan](./crawer/meituan)\nParser: [parser](./crawler/meituan/parser) uses css selectors for page parsing.\n\n![meituan](./img/meituan.png)\n\n# Framework\n\n![framework](./img/framework.png)\n\nRegister the micro service in Consul.\n\n![service](./img/consul_service.png)\n\n# Architecture\n\n![flow](./img/flow.png)\n\n# Installation\n\nDeploying projects locally or in the cloud provides two ways:\n\n- Direct Deploy\n- Docker(Recommended)\n\n# Pre-requisite\n\n- Go 1.13.6\n- Redis 6.0\n- MySQL 5.7\n- RabbitMQ management\n- ElasticSearch\n- Consul\n- Others [go.mod](./go.mod)\n\n\n# Quick Start\n\nPlease open the command line prompt and execute the command below. Make sure you have installed `docker-compose` in advance.\n\n```bash\ngit clone https://github.com/Knowledge-Precipitation-Tribe/go-crawler-distributed\ncd go-crawler-distributed/deploy/buildScript\nbash linux_build\ncd ../\ndocker network create -d bridge crawler\ndocker-compose up -d\n```\n\nThe above command has started the basic services, including Consul, Redis, elasticSearch, RabbitMQ, and mysql. RPC services for Redis and elasticSearch are also included.\n\n```bash\ndocker-compose up -d\n```\n\n```go\n[GIN-debug] GET    /swagger/*any             --\u003e github.com/swaggo/gin-swagger.CustomWrapHandler.func1 (7 handlers)\n[GIN-debug] GET    /sd/health                --\u003e go-crawler-distributed/handler/sd.HealthCheck (7 handlers)\n[GIN-debug] GET    /sd/disk                  --\u003e go-crawler-distributed/handler/sd.DiskCheck (7 handlers)\n[GIN-debug] GET    /sd/cpu                   --\u003e go-crawler-distributed/handler/sd.CPUCheck (7 handlers)\n[GIN-debug] GET    /sd/ram                   --\u003e go-crawler-distributed/handler/sd.RAMCheck (7 handlers)\n[GIN-debug] POST   /crawler/douban           --\u003e go-crawler-distributed/handler/crawler.StartDoubanCrawler (7 handlers)\n[GIN-debug] POST   /crawler/meituan          --\u003e go-crawler-distributed/handler/crawler.StartMeituanCrawler (7 handlers)\n{\"level\":\"info\",\"time\":\"2020-09-04T11:29:14.954+0800\",\"linenum\":\"go-crawler-distributed/main.go:59\",\"msg\":\"Start to listening the incoming requests on http address:80\"}\n{\"level\":\"info\",\"time\":\"2020-09-04T11:29:14.956+0800\",\"linenum\":\"go-crawler-distributed/main.go:55\",\"msg\":\"The router has been deployed successfully.\"}\n```\n\nswagger ui:\n\n![swagger](./img/swagger.png)\n\n\n# Port\n\n- 6379: redis\n- 3306: mysql\n- 9200, 9300: elasticSearch\n\n![elasticSearch](./img/elasticsearch.png)\n\n- 5672, 15672: rabbitMQ\n\n![rabbitmq](./img/rabbitmq.png)\n\n- 8500: consul\n\n![consul](./img/consul.png)\n\n# Run\n\n## Basic services\n\nPlease use `docker-compose` to one-click to start up. Create a file named `docker-compose.yml` and input the code below.\n\n```yaml\nversion: \"3\"\n\nservices:\n\n  cache:\n    build:\n      context: cache\n      dockerfile: Dockerfile\n    networks:\n      - crawler\n\n  elastic:\n    build:\n      context: elastic\n      dockerfile: Dockerfile\n    depends_on:\n      - elastic_server\n    networks:\n      - crawler\n\n  redis:\n    image: redis\n    restart: always\n    ports:\n      - \"6379:6379\"\n    volumes:\n      - redis-data:/data\n    networks:\n      - crawler\n\n  mysql:\n    image: mysql\n    command: --default-authentication-plugin=mysql_native_password\n    restart: always\n    environment:\n      MYSQL_ROOT_PASSWORD: example\n    networks:\n      - crawler\n\n  elastic_server:\n    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0\n    ports:\n      - \"9200:9200\"\n      - \"9300:9300\"\n    volumes:\n       - elastic-data:/data\n    environment:\n      - discovery.type=single-node\n    networks:\n      - crawler\n\n  rabbitmq:\n    image: rabbitmq:management\n    hostname: myrabbitmq\n    ports:\n      - \"5672:5672\"\n      - \"15672:15672\"\n    volumes:\n      - rabbitmq-data:/var/lib/rabbitmq\n    networks:\n      - crawler\n\n  consul1:\n    image: consul\n    restart: always\n    ports:\n      - \"8500:8500\"\n      - \"8300:8300\"\n      - \"8301:8301\"\n      - \"8302:8302\"\n      - \"8600:8600\"\n    command: agent -server -bootstrap-expect 2 -ui -bind=0.0.0.0 -client=0.0.0.0\n    networks:\n      - crawler\n\n  consul2:\n    image: consul\n    restart: always\n    ports:\n      - \"8501:8500\"\n    command: agent -server -ui -bind=0.0.0.0 -client=0.0.0.0 -join consul1\n    networks:\n      - crawler\n\n  consul3:\n    image: consul\n    restart: always\n    ports:\n      - \"8502:8500\"\n    command: agent -server -ui -bind=0.0.0.0 -client=0.0.0.0 -join consul1\n    networks:\n      - crawler\n\n\nvolumes:\n  elastic-data:\n  rabbitmq-data:\n  redis-data:\n\nnetworks:\n  crawler:\n    external: true\n```\n\nThen execute the command below, and the basic service will start up. \n\n```bash\ndocker-compose up -d\n```\n\nThe above command has started the basic services, including Redis, elasticSearch, RabbitMQ, and mysql. RPC services for Redis and elasticSearch are also included.\n\n## crawler service\n\nCreate a docker-composite.yml file under the meituan or douban folder.\n\n```yaml\nversion: \"3\"\n\nservices:\n\n  crawl_list:\n    build:\n      context: crawl_list\n      dockerfile: Dockerfile\n    networks:\n      - crawler\n\n  crawl_tags:\n    build:\n      context: crawl_urllist\n      dockerfile: Dockerfile\n    networks:\n      - crawler\n\n  crawl_detail:\n    build:\n      context: crawl_detail\n      dockerfile: Dockerfile\n    networks:\n      - crawler\n\n  storage_detail:\n    build:\n      context: storage_detail\n      dockerfile: Dockerfile\n    networks:\n      - crawler\n\nnetworks:\n  crawler:\n    external: true\n```\n\nThen execute the command below, and the crawler service will start up. \n\n```bash\ndocker-compose up -d\n```\n\nIf you want to start multiple crawler nodes you can use the following command.\n\n```bash\ndocker-compose up --scale crawl_list=3 -d\n```\n\n## Direct \nStart the service directly, but be aware that you need to rely on redis, RabbitMQ and other services\n\n```bash\ncd deploy/deploy/\nbash start-meituan-direct.sh\n```\n\n# 注意\n本项目是通过go-micro的方式将其封装为了微服务，但是个人认为对于一个单独的redis来说不应该将其封装为微服务，微服务的划分应该是从业务的角度出发。\n\n# Appendix\n\n- docker安装：[https://docs.docker.com/](https://docs.docker.com/)\n- docker-compose安装：[https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/)\n- 爬虫合集：[https://github.com/facert/awesome-spider](https://github.com/facert/awesome-spider)\n\n# License\n\n[MIT](https://github.com/Knowledge-Precipitation-Tribe/DigitRecognitionService/blob/master/LICENSE)\n\nCopyright (c) 2020 Knowledge-Precipitation-Tribe\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgolang-collection%2Fgo-crawler-distributed","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fgolang-collection%2Fgo-crawler-distributed","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fgolang-collection%2Fgo-crawler-distributed/lists"}