{"id":27048653,"url":"https://github.com/risyasin/arachnod","last_synced_at":"2025-04-05T07:15:30.455Z","repository":{"id":27085309,"uuid":"30552148","full_name":"risyasin/arachnod","owner":"risyasin","description":"High performance crawler for Nodejs","archived":false,"fork":false,"pushed_at":"2018-10-26T15:34:55.000Z","size":513,"stargazers_count":16,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-04-24T04:11:39.231Z","etag":null,"topics":["cheerio","crawler","javascript","nodejs","redis","scraper","spider"],"latest_commit_sha":null,"homepage":"http://risyasin.github.io/arachnod/","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/risyasin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-02-09T18:50:35.000Z","updated_at":"2023-09-08T16:54:33.000Z","dependencies_parsed_at":"2022-07-18T08:17:23.795Z","dependency_job_id":null,"html_url":"https://github.com/risyasin/arachnod","commit_stats":null,"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/risyasin%2Farachnod","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/risyasin%2Farachnod/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/risyasin%2Farachnod/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/risyasin%2Farachnod/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/risyasin","download_url":"https://codeload.github.com/risyasin/arachnod/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247299851,"owners_count":20916193,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cheerio","crawler","javascript","nodejs","redis","scraper","spider"],"created_at":"2025-04-05T07:15:29.878Z","updated_at":"2025-04-05T07:15:30.441Z","avatar_url":"https://github.com/risyasin.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"### Arachnod\n#### High performance crawler for Nodejs\n\nPowerful \u0026 Easy to use web crawler for Nodejs.  [Arachnod](http://arachnod.evrima.net) has been designed for heavy and long running tasks, \nfor performance \u0026 efective resource usage. For it's goals Arachnod uses [Redis](http://www.redis.io)'s power as a backend. \nCovering all heavy \u0026 time consuming tasks such as controlling urls \u0026 their tasks to store \u0026 distribute information among the Arachnod's child tasks \n(Spiderlings). Arachnod also avoids to use any server-side DOM requiring technics \nsuch as [jQuery](http://www.jquery.com) with [JSdom](https://github.com/tmpvar/jsdom) to use resources properly. \nFrankly, I have tested JSdom for along time with no luck, always memory leaks \u0026 high memory usage. \nLibxml based XPath solutions were not actually real, Instead, Arachnod uses [Cheerio](http://cheeriojs.github.io/cheerio/) for accessing DOM elements. \nAlso uses [SuperAgent](https://github.com/visionmedia/superagent) as HTTP Client. \n\n\n### How to install \n\n`$ npm install arachnod`\n    \nOr via Git\n`$ git clone git@github.com:risyasin/arachnod.git`\n    \nThen, install required Nodejs modules with npm \n`$ npm install`\n    \n**Please** make sure you have a running [redis-server](https://redis.io) \n\n### How to use\n\n``` js\nvar bot = require('arachnod');\n    \nbot.on('hit', function (doc, $) {\n    \n    // Do whatever you want to do parsed html content.\n    \n    var desc = $('article.entry-content').text();\n    \n    console.log(doc.path, desc);\n    \n    // if you don't need to follow all links.\n    bot.pause();\n    \n    \n});\n\n\nbot.crawl({\n    'redis': '127.0.0.1',\n    'parallel': 4,\n    'start': 'https://github.com/risyasin/arachnod',\n    'resume': false\n});\n\n\nbot.on('error', function (err, task) {\n    console.log('Bot error:', err, err.stack, task);\n});\n\n\nbot.on('end', function (err, status) {\n    console.log('Bot finished:', err, status);\n});\n```\n\n\n### Documentation \n\n\n##### Parameters\nParameter Name  | Description\n------------- | -------------\n**start** | Start url for crawling (Mandatory) \n**parallel** | Number of child processes that will handle network tasks (Default: 8) Do not this more than 20. \n**redis** | Host name or IP address that Redis runs on (Default: 127.0.0.1)\n**redisPort**  | Port number for Redis (Default: 6379)\n**verbose**  | Arachnod will tell you more, **1** (silence) - **10** (everything). Default: 1.\n**resume**  | Resume support, Simply, does not resets queues if there is any. (Default: false) \n**ignorePaths**  | Ignores paths starts with. Must be multiple in array syntax such as `['/blog','/gallery']` \n**ignoreParams**  | Ignores query string parameters, Must be in array syntax. such as `['color','type']`  \n**sameDomain**  | Stays in the same hostname. (implemented as of 0.4.4)\n**useCookies**  | Using cookies (implemented at 0.4.4 as **cookie** parameter)\n**basicAuth**  | Provide basic authentication credentials. `user:pass` \n**obeyRobotsTxt**  | As it's name says. Honors the robots.txt (will be implemented at v0.5) \n \n\n\n\n##### Events\nEvent Name  | Description\n------------- | -------------\n**hit** | Emits when a url has been downloaded \u0026 processed, sends two parameters in order *doc* Parsed url info, **$** as Cheerio object. \n**error**  | Emits when an error occurs at any level including child processes. Single parameter Error or Exception.  \n**end**  | Emits when reached at the end of tasks queue. Return statistics.  \n**stats**  | Emits bot stats whenever a child changes it's states (such as downloading or querying queues). Use wisely.  \n\n\n\n\n##### Methods\nMethod Name  | Description\n------------- | -------------\n**crawl(Parameters)** | Starts a new crawling session with parameters\n**pause()**  | Stops bot but does not delete any task queue.\n**resume()**  | Starts back a paused session. Useful to control resource usage in low spec systems (single core etc.). \n**queue(url)**  | Adds given url to task queue. \n**getStats()**  | Returns various statistics such as downloaded, checked, finished url counts, memory size etc. \n \n\n \n \n##### What's Next\n* Regex support for ignore parameters\n* Cookie support\n* Robots.txt \u0026 rel=nofollow support\n* Actions for content-type or any given response headers \n* Custom headers\n* Custom POST/PUT method queues\n* ~~Free-Ride mode (will be fun)~~ Probably not useful.\n* Stats for each download/hit event\n* Plugin support\n\n\n\n#### Support \nIf you like Arachnod, Help me to improve it.\n\n\n#### License\n\nCopyright 2015-17 yasin inat\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frisyasin%2Farachnod","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frisyasin%2Farachnod","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frisyasin%2Farachnod/lists"}