{"id":21650944,"url":"https://github.com/iansinnott/url-spider","last_synced_at":"2025-06-16T06:04:19.516Z","repository":{"id":66111870,"uuid":"291895168","full_name":"iansinnott/url-spider","owner":"iansinnott","description":null,"archived":false,"fork":false,"pushed_at":"2020-09-09T03:47:59.000Z","size":14,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-04T23:19:20.878Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/iansinnott.png","metadata":{"files":{"readme":"readme.org","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-01T04:25:18.000Z","updated_at":"2020-09-20T19:23:20.000Z","dependencies_parsed_at":"2023-03-10T02:16:43.326Z","dependency_job_id":null,"html_url":"https://github.com/iansinnott/url-spider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/iansinnott/url-spider","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iansinnott%2Furl-spider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iansinnott%2Furl-spider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iansinnott%2Furl-spider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iansinnott%2Furl-spider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/iansinnott","download_url":"https://codeload.github.com/iansinnott/url-spider/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/iansinnott%2Furl-spider/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260109459,"owners_count":22960025,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-25T07:46:06.187Z","updated_at":"2025-06-16T06:04:19.509Z","avatar_url":"https://github.com/iansinnott.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"* URL Spider\n\nA simple Script to spider all the URLs for a given domain _and_ its subdomains.\n\nExample:\n\n#+BEGIN_SRC shell\nyarn start:run 'https://iansinnott.com'\n#+END_SRC\n\nWill spider all the URLs at my site as well as all URLs at =blog.iansinnott.com=, =lab.iansinnott.com=, etc. URLs to external sites will be skipped.\n\nOnce the script runs it will dup all the information to a temp file. The location on your system will depend on the built-in =mktemp= util.\n\n** Usage\n\n#+BEGIN_SRC shell\nyarn start:run \u003curl\u003e\n#+END_SRC`\n\nWill spider the =\u003curl\u003e= and all its subdomains.\n\n\n* FIXME\n\nThis script does no stream processing. In other words, it will quite happily eat up all the JS heap memory if the site you're spidering has many URLs.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiansinnott%2Furl-spider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fiansinnott%2Furl-spider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fiansinnott%2Furl-spider/lists"}