{"id":13597453,"url":"https://github.com/keenwon/antcolony","last_synced_at":"2025-05-16T12:12:56.733Z","repository":{"id":26441481,"uuid":"29892306","full_name":"keenwon/antcolony","owner":"keenwon","description":"Nodejs实现的一个磁力链接爬虫 https://findit.keenwon.com (原域名http://findit.so )","archived":false,"fork":false,"pushed_at":"2018-12-28T13:07:16.000Z","size":67,"stargazers_count":1419,"open_issues_count":6,"forks_count":418,"subscribers_count":68,"default_branch":"master","last_synced_at":"2025-04-12T08:29:36.185Z","etag":null,"topics":["antcolony","bencode","bittorrent","dht","javascript","nodejs","spider","torrent"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/keenwon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-27T01:58:09.000Z","updated_at":"2025-04-09T17:39:37.000Z","dependencies_parsed_at":"2022-07-17T04:00:32.567Z","dependency_job_id":null,"html_url":"https://github.com/keenwon/antcolony","commit_stats":null,"previous_names":["dhtcrawler/antcolony"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keenwon%2Fantcolony","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keenwon%2Fantcolony/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keenwon%2Fantcolony/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/keenwon%2Fantcolony/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/keenwon","download_url":"https://codeload.github.com/keenwon/antcolony/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254527099,"owners_count":22085919,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["antcolony","bencode","bittorrent","dht","javascript","nodejs","spider","torrent"],"created_at":"2024-08-01T17:00:33.226Z","updated_at":"2025-05-16T12:12:51.725Z","avatar_url":"https://github.com/keenwon.png","language":"JavaScript","funding_links":[],"categories":["tool","JavaScript"],"sub_categories":["docker"],"readme":"# AntColony\n\n### 介绍 \n  \nAntColony（Github）是findit磁力搜索引擎的核心。用来在DHT网络中，收集活跃资源的infohash，下载并解析资源的种子文件，存入数据库等。AntColony是若干功能的合集，也可以单独运行其中的部分功能，所以起“蚁群”这个名字也是很贴切的（没错，我就是爱动物世界）。主要分一下几块：  \n- worker：爬虫，收集资源infohash，可以同时启动多个进程的worker，提高效率  \n- male：根据收集来的infohash去下载种子文件\n- female：将种子文件录入数据库\n- queen：简单的入口，启动pm2运行worker，male和female\n\n![](http://img.keenwon.com/2015/03/20150305144005_68120.png)\n\n\n目前启动5个worker使用3000-3004的端口，2个male和1个female。  \n\n使用Mongodb储存数据，这没什么好说的；使用pm2维护和监控node进程，也没太多要说的，重点说下Redis。Redis里暂存的数据大概是这样的：  \n\n![](http://img.keenwon.com/2015/03/20150305144626_49722.png)\n  \n- bucket：比较大的一个K桶  \n- infohash：就是已经收集到的infohashs（worker收集来的，male会用来下载种子）  \n- remoteNodes：worker新认识的节点，会依次“拜访”的，目前只保存最新的10w个（一方面我的VPS内存小，另一方面真没必要记录太多）  \n- sysInfo：记录一些统计信息，例如发出多少次请求，累积收集多少infohashs，目前已经发出15亿次Request，这个频率是可控的，worker 太疯狂的话，VPS扛不住。  \n  \n### 安装\u0026运行方法\n  \n详细的使用方法请查看[部署文档](https://github.com/keenwon/antcolony/blob/master/doc/%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.md)  \n其他内容查看 [http://keenwon.com/1436.html](http://keenwon.com/1436.html \"http://keenwon.com/1436.html\")  \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeenwon%2Fantcolony","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkeenwon%2Fantcolony","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkeenwon%2Fantcolony/lists"}