{"id":20676119,"url":"https://github.com/sammwyy/craw","last_synced_at":"2026-04-29T23:33:07.475Z","repository":{"id":122454585,"uuid":"289072349","full_name":"sammwyy/craw","owner":"sammwyy","description":"a website-crawler library for nodejs","archived":false,"fork":false,"pushed_at":"2020-08-20T18:22:19.000Z","size":13,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-05-10T23:32:22.534Z","etag":null,"topics":["crawler","crawlers","html","javascript","library","node","nodejs","nodejs-module","npm","npm-module","parser","spider","website"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sammwyy.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-08-20T17:50:22.000Z","updated_at":"2020-08-20T21:18:28.000Z","dependencies_parsed_at":null,"dependency_job_id":"339feca6-b7fa-45c4-b4f5-20cf52e58504","html_url":"https://github.com/sammwyy/craw","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/sammwyy/craw","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammwyy%2Fcraw","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammwyy%2Fcraw/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammwyy%2Fcraw/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammwyy%2Fcraw/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sammwyy","download_url":"https://codeload.github.com/sammwyy/craw/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sammwyy%2Fcraw/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32448399,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T22:27:22.272Z","status":"ssl_error","status_checked_at":"2026-04-29T22:10:49.234Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","crawlers","html","javascript","library","node","nodejs","nodejs-module","npm","npm-module","parser","spider","website"],"created_at":"2024-11-16T21:11:52.755Z","updated_at":"2026-04-29T23:33:07.459Z","avatar_url":"https://github.com/sammwyy.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CRAW\na website-crawler library for nodejs\n\n## Documentation\nDocumentation of the library in a summarized and precise way.  \n\n### Usage\n```javascript\nconst craw = require('craw');\n\nasync function start () {\n  const result = await craw(\"https://2lstudios.dev/\");\n  console.log(result.toJSON());\n}\n\nstart();\n```\n\n### result.getContent()\nGet the content of the website as headers, paragraphs, paragraphs and all the text in general.  \n\u003cins\u003eOutput:\u003c/ins\u003e\n```javascript\n{\n  text: \"....\", // String\n  h1: [], // Array\n  h2: []. // Array\n  h3: [], // Array\n  h4: [], // Array\n  h5: [], // Array\n  h6: [], // Array\n  words: [] // Array\n}\n```\n\n### result.getFrames()\nGet a list with iframes from the website.  \n\u003cins\u003eOutput:\u003c/ins\u003e\n  ```javascript\n  [...]  // Array\n```\n\n### result.getImports()\nGet a list of imports from the website. (like css, favicon and js)  \n\u003cins\u003eOutput:\u003c/ins\u003e\n```javascript\n{\n  scripts: [ // Array\n    {\n      integrity: \"...\", // String\n      src: \"...\", // String\n      async: ... // Boolean\n    }\n  ],\n\n  styles: [ // Array\n    {\n      integrity: \"...\", // String\n      href: \"...\", // String\n      rel: \"...\" // String\n    }\n  ],\n  \n  favicon: {\n    type: \"...\", // String\n    href: \"...\" // String \n  }\n}\n```\n\n### result.getLinks()\nGet a list of hyperlinks from the website.  \n\u003cins\u003eOutput:\u003c/ins\u003e\n```javascript\n[ // Array\n  {\n    url: \"...\", // String\n    anchor: \"...\", // String\n    rel: [ ... ] // Array of Strings\n  }\n]\n```\n\n### result.getMedia()\nGet a list of multimedia elements from the website. (Like images, audios and videos)  \n\u003cins\u003eOutput:\u003c/ins\u003e\n```javascript\n{\n  audios: [ // Array\n    {\n      src: \"...\", // String\n      type: \"...\" // String\n    }\n  ],\n  images: [ // Array\n    {\n      src: \"...\", // String\n      alt: \"...\", // String\n      loading: \"...\" // String\n    }\n  ],\n  videos: [ ... ] // Array of strings\n}\n```\n\n### result.getMeta()\nGet a list of metadata tags from the website.  \n\u003cins\u003eOutput:\u003c/ins\u003e\n```javascript\n{\n  author: \"...\", // String\n  viewport: \"...\", // String\n  robots: \"...\", // String\n  description: \"...\", // String\n  keywords: [], // Array of strings\n  image: \"...\", // String (Favicon)\n  charset: \"...\", // String\n  ... any other metadata tag like OG or Twitter ...\n}\n```\n\n### result.getTitle()\nGet the title of the website.  \n\u003cins\u003eOutput:\u003c/ins\u003e\n```javascript\n\"...\" // String\t\n```\n\n### result.toJSON()\nRun all functions and add the results of each one in the same object.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsammwyy%2Fcraw","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsammwyy%2Fcraw","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsammwyy%2Fcraw/lists"}