{"id":16129212,"url":"https://github.com/haxzie-xx/crode.js-node-web-crawler","last_synced_at":"2026-05-01T13:31:50.020Z","repository":{"id":121696682,"uuid":"94297828","full_name":"haxzie-xx/crode.js-Node-web-Crawler","owner":"haxzie-xx","description":"Node.js Crawler built for open FTP sites for movie link collection.","archived":false,"fork":false,"pushed_at":"2017-07-23T18:22:32.000Z","size":937,"stargazers_count":4,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-02-12T19:44:49.660Z","etag":null,"topics":["crawler","nodejs"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/haxzie-xx.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-06-14T06:34:36.000Z","updated_at":"2022-08-28T03:21:20.000Z","dependencies_parsed_at":null,"dependency_job_id":"1caa4ee6-90ab-41cb-b048-a53946c305bc","html_url":"https://github.com/haxzie-xx/crode.js-Node-web-Crawler","commit_stats":{"total_commits":5,"total_committers":1,"mean_commits":5.0,"dds":0.0,"last_synced_commit":"0cb7840202aa97176856dde49414cf0a2903b4c8"},"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haxzie-xx%2Fcrode.js-Node-web-Crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haxzie-xx%2Fcrode.js-Node-web-Crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haxzie-xx%2Fcrode.js-Node-web-Crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/haxzie-xx%2Fcrode.js-Node-web-Crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/haxzie-xx","download_url":"https://codeload.github.com/haxzie-xx/crode.js-Node-web-Crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247488008,"owners_count":20946865,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","nodejs"],"created_at":"2024-10-09T22:09:32.736Z","updated_at":"2026-05-01T13:31:44.993Z","avatar_url":"https://github.com/haxzie-xx.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# crode.js\n\nVisit [haxzie.me](http://haxzie.me) for demo.\n\nNode.js web crawler scrapping file downloadable link from open ftp servers.\ncrode.js is an express app.crawling is made possible using [npm - Crawler](https://www.npmjs.com/package/crawler).\napp scrapes the html file using npm modules:\n\n-[request](https://www.npmjs.com/package/request)\n\n-[cheerio](https://www.npmjs.com/package/cheerio)\n\n### Storage\n\ncrode.js uses Mongo db by default currenty uses only two db schemas for storing download links found and visited.\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaxzie-xx%2Fcrode.js-node-web-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fhaxzie-xx%2Fcrode.js-node-web-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fhaxzie-xx%2Fcrode.js-node-web-crawler/lists"}