{"id":15435072,"url":"https://github.com/mikeal/spider","last_synced_at":"2025-04-12T19:46:39.770Z","repository":{"id":1150402,"uuid":"1034096","full_name":"mikeal/spider","owner":"mikeal","description":"Programmable spidering of web sites with node.js and jQuery","archived":false,"fork":false,"pushed_at":"2019-06-02T18:07:39.000Z","size":132,"stargazers_count":725,"open_issues_count":17,"forks_count":104,"subscribers_count":48,"default_branch":"master","last_synced_at":"2024-10-30T00:32:35.357Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mikeal.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2010-10-29T05:17:39.000Z","updated_at":"2024-08-12T19:19:33.000Z","dependencies_parsed_at":"2022-08-16T12:15:26.648Z","dependency_job_id":null,"html_url":"https://github.com/mikeal/spider","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeal%2Fspider","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeal%2Fspider/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeal%2Fspider/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mikeal%2Fspider/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mikeal","download_url":"https://codeload.github.com/mikeal/spider/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248625497,"owners_count":21135513,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-01T18:42:23.527Z","updated_at":"2025-04-12T19:46:39.744Z","avatar_url":"https://github.com/mikeal.png","language":"JavaScript","funding_links":[],"categories":["Nodejs"],"sub_categories":[],"readme":"# Spider -- Programmable spidering of web sites with node.js and jQuery\n\n## Install\n\nFrom source:\n\n\u003cpre\u003e\n  git clone git://github.com/mikeal/spider.git \n  cd spider\n  npm link ../spider\n\u003c/pre\u003e\n\n## (How to use the) API\n\n### Creating a Spider\n\u003cpre\u003e\n  var spider = require('spider');\n  var s = spider();\n\u003c/pre\u003e\n\n#### spider(options)\n\nThe `options` object can have the following fields:\n\n* `maxSockets` - Integer containing the maximum amount of sockets in the pool. Defaults to `4`.\n* `userAgent` - The User Agent String to be sent to the remote server along with our request. Defaults to `Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7` (firefox userAgent String).\n* `cache` -  The Cache object to be used as cache. Defaults to NoCache, see code for implementation details for a new Cache object.\n* `pool` - A hash object containing the agents for the requests. If omitted the requests will use the global pool which is set to maxSockets.\n\n### Adding a Route Handler\n\n#### spider.route(hosts, pattern, cb)\nWhere the params are the following : \n\n* `hosts` - A string -- or an array of string -- representing the `host` part of the targeted URL(s).\n* `pattern` - The pattern against which spider tries to match the remaining (`pathname` + `search` + `hash`) of the URL(s).\n* `cb` - A function of the form `function(window, $)` where\n  * `this` - Will be a variable referencing the `Routes.match` return object/value with some other goodies added from spider. For more info see https://github.com/aaronblohowiak/routes.js\n  * `window` - Will be a variable referencing the document's window.\n  * `$` - Will be the variable referencing the jQuery Object.\n\n### Queuing an URL for spider to fetch.\n\n`spider.get(url)` where `url` is the url to fetch.\n\n### Extending / Replacing the MemoryCache \n\nCurrently the MemoryCache must provide the following methods:\n\n* `get(url, cb)` - Returns `url`'s `body` field via the `cb` callback/continuation if it exists. Returns `null` otherwise.\n  * `cb` - Must be of the form `function(retval) {...}`\n* `getHeaders(url, cb)` - Returns `url`'s `headers` field via the `cb` callback/continuation if it exists. Returns `null` otherwise.\n  * `cb` - Must be of the form `function(retval) {...}`\n* `set(url, headers, body)` - Sets/Saves `url`'s `headers` and `body` in the cache.\n\n### Setting the verbose/log level\n`spider.log(level)` - Where `level` is a string that can be any of `\"debug\"`, `\"info\"`, `\"error\"`\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikeal%2Fspider","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmikeal%2Fspider","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmikeal%2Fspider/lists"}