{"id":18841625,"url":"https://github.com/alphasights/crawl","last_synced_at":"2025-07-07T01:06:45.926Z","repository":{"id":1785741,"uuid":"2709606","full_name":"alphasights/crawl","owner":"alphasights","description":"Crawl pages within a domain, reporting any page that returns a bad response code","archived":false,"fork":false,"pushed_at":"2018-03-26T16:13:26.000Z","size":52,"stargazers_count":5,"open_issues_count":0,"forks_count":6,"subscribers_count":55,"default_branch":"master","last_synced_at":"2024-04-25T15:22:26.764Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"fooplugins/FooTable","license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/alphasights.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2011-11-04T14:06:36.000Z","updated_at":"2021-02-09T21:33:05.000Z","dependencies_parsed_at":"2022-08-19T21:11:38.409Z","dependency_job_id":null,"html_url":"https://github.com/alphasights/crawl","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphasights%2Fcrawl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphasights%2Fcrawl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphasights%2Fcrawl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/alphasights%2Fcrawl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/alphasights","download_url":"https://codeload.github.com/alphasights/crawl/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248839317,"owners_count":21169791,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T02:52:01.863Z","updated_at":"2025-04-14T07:31:02.477Z","avatar_url":"https://github.com/alphasights.png","language":"Ruby","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Crawl\n\n[![Circle CI](https://circleci.com/gh/alphasights/crawl/tree/master.svg?style=shield\u0026circle-token=188c328ee0669c8eba8dbda7f7f9943445a50c70)](https://circleci.com/gh/alphasights/crawl/tree/master)\n\n\nCrawl pages within a domain, reporting any page that returns a bad response code\n\nUsage:\n\n    \u003e crawl [options] domain\n\n    Usage: crawl [options] domain\n    -s, --start /home,/about         Starting path(s), defaults to /\n    -u, --username username          Basic auth username\n    -p, --password password          Basic auth password\n    -c, --connections count          Max mumber of parallel connections to use. The default is 5.\n    -v, --verbose                    Give details when crawling\n    -h, --help                       Show this message\n        --version                    Print version\n\n\n\nExample:\n\n    \u003e crawl https://engineering.alphasights.com --connections=5 --start=/ --verbose\n\n      Adding /\n    Fetching / ...\n      Adding /positions/ruby-developer\n      Adding /positions/js-ember-developer\n      Adding /positions/ux-ui-designer\n      Adding /positions/support-specialist\n    Fetching /positions/ruby-developer\n    Fetching /positions/js-ember-developer ...\n    Fetching /positions/ux-ui-designer ...\n    Fetching /positions/support-specialist ...\n\n    5 pages crawled without errors.\n\n### Copyright and License\n\nCopyright AlphaSights and Contributors, 2015\n\n[MIT Licence](LICENSE.txt)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falphasights%2Fcrawl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Falphasights%2Fcrawl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Falphasights%2Fcrawl/lists"}