{"id":21074917,"url":"https://github.com/nextlevelshit/adonis-crawler","last_synced_at":"2026-05-22T07:12:49.381Z","repository":{"id":88474890,"uuid":"84087591","full_name":"nextlevelshit/adonis-crawler","owner":"nextlevelshit","description":"A free web crawler on top of the incredibile AdonisJS Framework ","archived":false,"fork":false,"pushed_at":"2017-03-08T08:49:41.000Z","size":41,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-01-20T22:42:25.541Z","etag":null,"topics":["adonisjs","crawler","javascript","nodejs","regex","spider","websocket"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nextlevelshit.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-03-06T15:12:39.000Z","updated_at":"2017-03-08T08:50:10.000Z","dependencies_parsed_at":null,"dependency_job_id":"f04a5662-7948-46f7-b62d-fbfbdc8b6ca0","html_url":"https://github.com/nextlevelshit/adonis-crawler","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextlevelshit%2Fadonis-crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextlevelshit%2Fadonis-crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextlevelshit%2Fadonis-crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextlevelshit%2Fadonis-crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nextlevelshit","download_url":"https://codeload.github.com/nextlevelshit/adonis-crawler/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243517575,"owners_count":20303659,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["adonisjs","crawler","javascript","nodejs","regex","spider","websocket"],"created_at":"2024-11-19T19:18:40.526Z","updated_at":"2026-05-22T07:12:44.340Z","avatar_url":"https://github.com/nextlevelshit.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Adonis Web Crawler\n\nA free web crawler on top of the incredibile AdonisJS Framework\n\n## Road map\n\nThings wanting to get done.\n\n- [x] Working draft\n- [x] Recursive search\n- [x] Pause and continue button\n- [ ] Normalize links by using the server response\n- [ ] Normalize links by removing trailing hashes\n- [ ] Convert links inside `CrawlerController` to models\n- [ ] Set own regex for extracting data\n- [ ] Write tests\n\n## Author\n\nMichael Czechowski \u003cmail@dailysh.it\u003e\n\n_Dependencies and external libraries are listed in the [`package.json`](https://github.com/nextlevelshit/adonis-crawler/blob/master/package.json#L19)_\n\n## License\n```\nAdonis Crawler - A free web crawler on top of AdonisJs Framework\nCopyright (C) 2017 Michael Czechowski\n\nThis program is free software: you can redistribute it and/or modify\nit under the terms of the GNU General Public License as published by\nthe Free Software Foundation, either version 3 of the License, or\n(at your option) any later version.\n\nThis program is distributed in the hope that it will be useful,\nbut WITHOUT ANY WARRANTY; without even the implied warranty of\nMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the\nGNU General Public License for more details.\n\nYou should have received a copy of the GNU General Public License\nalong with this program.  If not, see \u003chttp://www.gnu.org/licenses/\u003e.\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextlevelshit%2Fadonis-crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnextlevelshit%2Fadonis-crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextlevelshit%2Fadonis-crawler/lists"}