{"id":15107298,"url":"https://github.com/lightzhu/node_crawler","last_synced_at":"2025-10-23T02:31:01.616Z","repository":{"id":42091620,"uuid":"267221038","full_name":"lightzhu/Node_Crawler","owner":"lightzhu","description":"Node.js 项目,koa cheerio爬虫小程序,爬取电影、免费科学上网节点，钉钉定时消息。","archived":false,"fork":false,"pushed_at":"2023-03-03T00:22:29.000Z","size":615,"stargazers_count":11,"open_issues_count":23,"forks_count":11,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-30T16:52:51.340Z","etag":null,"topics":["crawler","freevpn","mongoose","node","ss","ssr","v2ray","vmess","vpn"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/lightzhu.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-05-27T04:31:56.000Z","updated_at":"2024-02-08T10:21:38.000Z","dependencies_parsed_at":"2023-02-17T05:46:03.081Z","dependency_job_id":null,"html_url":"https://github.com/lightzhu/Node_Crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightzhu%2FNode_Crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightzhu%2FNode_Crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightzhu%2FNode_Crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/lightzhu%2FNode_Crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/lightzhu","download_url":"https://codeload.github.com/lightzhu/Node_Crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":237763855,"owners_count":19362310,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","freevpn","mongoose","node","ss","ssr","v2ray","vmess","vpn"],"created_at":"2024-09-25T21:22:55.075Z","updated_at":"2025-10-23T02:31:01.266Z","avatar_url":"https://github.com/lightzhu.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NodeCrawler\n\nNode.js 项目，管理一些爬虫小程序\n\n# 1、git clone\n\n# 2、npm install\n\n# 3、npm test(启动一个热重载的应用)\n\n# 4、npm start\n\n# 效果\n\n![](https://cdn.jsdelivr.net/gh/lightzhu/public_cdn@0.6.5/image/github/table1.png)\n\n![](https://cdn.jsdelivr.net/gh/lightzhu/public_cdn@0.6.5/image/github/table2.png)\n\n# 免费 VPN L2TP/IPsec 方式\n\n![dingtalk.png](https://cdn.jsdelivr.net/gh/lightzhu/public_cdn@0.6.7/image/vpn/free-vpn.png)\n\n# v2ray 、ssr 、ss 节点分享\n\n![free-ss.png](https://cdn.jsdelivr.net/gh/lightzhu/public_cdn@0.6.7/image/vpn/free-ss.png)\n\n![free-ssr.png](https://cdn.jsdelivr.net/gh/lightzhu/public_cdn@0.6.7/image/vpn/free-ssr.png)\n\n# 交流学习钉钉群号： 31971273\n\n# 本项目仅供交流学习使用，请准守国家相关法律法规！更新的科学上网主机勿用于违法犯罪等活动！！违反者后果自负！！！\n\n# 科学上网配置教程\n\n- [window 和 android 配置方式](https://www.2048888.xyz/2020/07/24/Node/vpn2/#more)\n- [mac 和 ios 配置方式](https://www.2048888.xyz/2020/07/24/Node/vpn_L2TP/#more)\n\n# 将项目部署到 openshift 上面\n\n- 简单的部署直接浏览 catalog 选择相应的语言，next 的之后填写应用名称和项目地址就可以了\n- 部署私有项目前两步同上，接着选择 advanced options 选项填写应用名称和项目地址，创建一个 New Secret（这个可以选择两种方式：密钥或者账户名加密码\n\n# 设置开发环境\n\n- 在 package.json 里设置对应的运行命令 export NODE_ENV='development'\n\n# 启用服务端 gzip\n\n- 安装并使用 koa-compress 插件\n\n# 关于 heroku 的部署\n\n- 安装 heroku 客户端\n- heroku login\n- 在已有项目的根目录执行 herku create(发现创建了一个随机 app，不想要)\n- git remote -v 查看当前项目有几个远程源地址\n- git remote rm heroku 删除 heroku 源\n- 载创建自己想要的应用名称，herku create [appname],生成应用的 heroku git 地址\n- 手动将这个源地址关联到项目上去 git remote add [源名称][源地址]\n- 将项目推送到 heroku 源中，项目会自动部署\n\n# 解决 puppeteer 库无法在 heroku 中正常启动的问题\n\n- https://github.com/jontewks/puppeteer-heroku-buildpack\n\n```Bash\n $ heroku buildpacks:add jontewks/puppeteer\n```\n\n- Or use the source code in this repository:\n\n```Bash\n  $ heroku buildpacks:add https://github.com/jontewks/puppeteer-heroku-buildpack.git\n```\n\n```javascript\nconst browser = await puppeteer.launch({\n  args: ['--no-sandbox', '--disable-setuid-sandbox']\n})\n```\n\n# 将项目部署到 gearhost（免费的小程序托管云平台）\n\n- gearhost 里面创建一个项目\n- github 上面创建一个项目\n- gitclone 到本地\n- 将 gearhost 里面的 LocalGit Deployments 的项目地址添加的 git 中\n- git remote add websites https://${你的项目名称}.scm.gear.host/${你的项目名称}.git\n- 接下来就可以往 gearhost 的地址源里 push 你的代码了\n- 在 gearhost 的项目面板里面设置 node 的版本号\n- 不要指定 app 的端口，优先使用 process.env.PORT\n\n# 设置开发环境\n\n- 在 package.json 里设置对应的运行命令 export NODE_ENV='development'\n\n# 在 window 平台安装 puppeteer 需要设置一些 C++环境\n\n- 首先 node-gyp 需要 Python, v3.6, v3.7, v3.8, or v3.9 任意版本\n- 安装 Visual C++ Build 环境: 我是通过 Visual Studio Community 安装 \"Desktop development with C++\" 这个选项\n- 启动 cmd,设置： npm config set msvs_version 2019(VS 的版本,看你安装情况)\n- 参考\u003ca href=\"https://github.com/nodejs/node-gyp#on-windows\"\u003e\u003c/a\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightzhu%2Fnode_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flightzhu%2Fnode_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flightzhu%2Fnode_crawler/lists"}