{"id":21162550,"url":"https://github.com/josepedrodias/naivebot","last_synced_at":"2026-04-29T14:08:16.529Z","repository":{"id":144777374,"uuid":"97934984","full_name":"JosePedroDias/naivebot","owner":"JosePedroDias","description":"attempt to mimic googlebot behaviour in nodejs with nightmarejs","archived":false,"fork":false,"pushed_at":"2017-07-22T08:15:17.000Z","size":4,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-13T05:03:40.756Z","etag":null,"topics":["crawler","googlebot","nightmarejs","nodejs","robots"],"latest_commit_sha":null,"homepage":null,"language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/JosePedroDias.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2017-07-21T10:14:26.000Z","updated_at":"2017-07-21T10:18:54.000Z","dependencies_parsed_at":null,"dependency_job_id":"88b6ef5d-05d9-439c-ae55-bba6cbf470f6","html_url":"https://github.com/JosePedroDias/naivebot","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/JosePedroDias/naivebot","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosePedroDias%2Fnaivebot","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosePedroDias%2Fnaivebot/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosePedroDias%2Fnaivebot/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosePedroDias%2Fnaivebot/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/JosePedroDias","download_url":"https://codeload.github.com/JosePedroDias/naivebot/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/JosePedroDias%2Fnaivebot/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":32428645,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-29T13:34:34.882Z","status":"ssl_error","status_checked_at":"2026-04-29T13:34:29.830Z","response_time":110,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["crawler","googlebot","nightmarejs","nodejs","robots"],"created_at":"2024-11-20T13:30:49.933Z","updated_at":"2026-04-29T14:08:16.515Z","avatar_url":"https://github.com/JosePedroDias.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# naivebot\n\nSimulates googlebot visiting pages, kinda.\n\nThis is very experimental, naive and a possibly plain wrong approach.\n\nI'm not publishing this as a npm module because it's much easier to edit the hooks in the index.js\nitself then to create override capabilities for those.\n\n\n## config\n\nEdit `config.json` file.\n\n```js\n{\n  \"domain\": \"pixels.camp\", // domain to scrap\n  \"userAgent\": \"\", // user agent to set (TODO)\n  \"resolution\": [800, 600], // screen resolution to use\n  \"pages\": [\"https://pixels.camp/\"] // initial pages (kinda like sitemap.xml)\n}\n```\n\n\n## current crawling behaviour\n\nPages and their screenshots are persisted to `pages` directory.\n\nBootstrapped `toVisit` array of pages comes from `config.json`.\nWhile that array has elements, scrap continues.\nEach scrap consists of several promises being fulfilled:\n\n* waitPageReady - resolves once page is deemed ready. currenly waits 5 secs.\n* atPageStart - something to do once page is ready. ex: dismiss modal.\n* indexFollowCriteria - returns object with booleans for `index` and `follow`, work like the robots counterpart, i.e., index saves the scrapped page, follow adds found links to `toVisit`.\n\nNotice that most of these receive and return an object with:\n* nightmare - the nightmare instance\n* o - scrapped data from page\n* state - scrapping state.\n\nIndexed pages are stored to `\u003cpage\u003e.json` and screenshot to `\u003cpage\u003e.png`,\nwhere `\u003cpage\u003e` is a file-system friendly version of the page path.\n\n\nThis is the object persisted for every page marked for storage:\n\n```js\n{\n  url          : location.href\n  title        : document.title\n  text         : document.body.innerText\n  html         : document.documentElement.outerHTML\n  mRobots      : // meta robots\n  mTitle       : // meta title\n  mKeywords    : // meta keywords\n  mDescription : // meta description\n  h1           : // first h1's inner text\n  afterH1      : // inner text of element after first h1\n  links        : // array of a hrefs\n}\n```\n\nNotice this h1 and afterH1, which are attempts to elect alternate titles and descriptions.\n\n\n## TODO\n\n* investigate how googlebot determines page loaded or alternate clever approach\n* check if links scrapped are as naive as ours (`\u003ca\u003e`s on page body)\n* improve path processing - support #! paths\n* (less relevant) map robots.txt and sitemap.xml to config.json\n\n\n## references\n\n\nhttps://developers.google.com/search/reference/robots_meta_tag#valid-indexing--serving-directives\nhttps://github.com/segmentio/nightmare/blob/master/Readme.md\nhttps://segment.com/blog/ui-testing-with-nightmare/\nhttps://varvy.com/googlebot.html\nhttps://support.google.com/webmasters/answer/96569?hl=en\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosepedrodias%2Fnaivebot","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjosepedrodias%2Fnaivebot","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjosepedrodias%2Fnaivebot/lists"}