{"id":13464631,"url":"https://github.com/wcong/ants-go","last_synced_at":"2026-01-10T08:21:41.311Z","repository":{"id":26718408,"uuid":"30175803","full_name":"wcong/ants-go","owner":"wcong","description":"open source, distributed, restful crawler engine in golang","archived":true,"fork":false,"pushed_at":"2016-04-05T05:03:38.000Z","size":146,"stargazers_count":363,"open_issues_count":4,"forks_count":118,"subscribers_count":45,"default_branch":"master","last_synced_at":"2024-10-29T17:51:03.313Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wcong.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-02T07:21:03.000Z","updated_at":"2024-09-21T12:53:19.000Z","dependencies_parsed_at":"2022-07-29T21:38:56.388Z","dependency_job_id":null,"html_url":"https://github.com/wcong/ants-go","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wcong%2Fants-go","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wcong%2Fants-go/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wcong%2Fants-go/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wcong%2Fants-go/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wcong","download_url":"https://codeload.github.com/wcong/ants-go/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245454064,"owners_count":20617970,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T14:00:47.683Z","updated_at":"2026-01-10T08:21:41.267Z","avatar_url":"https://github.com/wcong.png","language":"Go","funding_links":[],"categories":["All","Go"],"sub_categories":[],"readme":"### ants-go\n\nopen source, restful, distributed crawler engine\n\n### gitter\n\n[![Join the chat at https://gitter.im/wcong/ants-go](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/wcong/ants-go?utm_source=badge\u0026utm_medium=badge\u0026utm_campaign=pr-badge\u0026utm_content=badge)\n\n### comming up\n\n* Persistence\n* Dynamic Master\n\n### design of ants-go\n##### ants\nI wrote a crawler engine named [ants](https://github.com/wcong/ants) in python base on [scrapy](https://github.com/scrapy/scrapy). But sometimes, dynamic language is chaos.\nSo I start to write it in a compile language. \n##### scrapy\nI design the crawler framework  by imitating  [scrapy](https://github.com/scrapy/scrapy).\nsuch as downloader,scraper,and the way user write customize spider,\nbut in a compile way\n##### elasticsearch\nI design my distributed architecture by imitating [elasticsearch](https://github.com/elasticsearch/elasticsearch).\nit spire me to do a engine for distributed crawler\n### requirement\n``` shell\ngo get github.com/PuerkitoBio/goquery\ngo get github.com/go-sql-driver/mysql\n```\n### install\n\n``` shell\ngo get github.com/wcong/ants-go\ngo install github.com/wcong/ants-go\n```\n\n### run\n\n``` shell\ncd bin\n./ants-go\n```\n\n#### check cluster status\n\n```\ncurl 'http://localhost:8200/cluster'\n```\n\n#### get all spiders\n\n```\ncurl 'http://localhost:8200/spiders'\n```\n\n#### start a spider\n\n```\ncurl 'http://localhost:8200/crawl?spider=spiderName'\n```\n\n\n#### cluster in one computer\nto test cluster in one computer,you can run it from different port in different terminal\n\none node,use the default port tcp 8300 http 8200\n\n``` shell\ncd bin\n./ants-go\n```\n\nthe other node set tcp port and http port\n\n``` shell\ncd bin\n./ants-go -tcp 9300 -http 9200\n```\n#### flags\nthere are some flags you can set,check out the help message\n\n``` shell\n./ants-go -h\n./ants-go -help\n```\n\n### Customize spider\n1.\tgo to *spiders*\n2.\twrite your spiders follow the example *deap_loop_spider.go* or go to the [spider page](./SPIDER.md)\n3.\tadd you spider to spiderMap,follow the example in *LoadAllSpiders* in *load_all_spider.go*\n4.\tinstall again\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwcong%2Fants-go","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwcong%2Fants-go","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwcong%2Fants-go/lists"}