{"id":18318370,"url":"https://github.com/tether/roach","last_synced_at":"2025-04-05T21:32:43.473Z","repository":{"id":14918034,"uuid":"17642190","full_name":"tether/roach","owner":"tether","description":"A very adaptable web crawler framework. Impossible to kill.","archived":false,"fork":false,"pushed_at":"2014-05-29T16:24:45.000Z","size":1783,"stargazers_count":7,"open_issues_count":20,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T02:03:53.283Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tether.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2014-03-11T18:55:54.000Z","updated_at":"2016-05-19T12:18:46.000Z","dependencies_parsed_at":"2022-09-16T22:02:26.504Z","dependency_job_id":null,"html_url":"https://github.com/tether/roach","commit_stats":null,"previous_names":["petrofeed/roach"],"tags_count":2,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tether%2Froach","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tether%2Froach/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tether%2Froach/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tether%2Froach/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tether","download_url":"https://codeload.github.com/tether/roach/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247406082,"owners_count":20933803,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-05T18:09:15.935Z","updated_at":"2025-04-05T21:32:43.135Z","avatar_url":"https://github.com/tether.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Roach\n\n\u003e A very adaptable web crawler framework. Impossible to kill.\n\n![Roach](roach_medium.png)\n\n[![Build Status](https://travis-ci.org/PetroFeed/roach.png)](https://travis-ci.org/PetroFeed/roach)\n\n## Getting Started\n\n1. Install the module via npm:\n  \n```\nnpm install roach\n```\n\n## Usage\n\n### Single Document, Roach Filter, Multiple Transports\n\n```js\nvar Roach = require('roach');\n\nvar roach = new Roach();\n\n// Use multiple transports with default options\nroach.use('rabbitmq').use('mongodb');\n\n// Add a job to get all the 'a' links from google.com\nroach.addJob( 'http://google.com', 'google' ).filter( 'links' );\n\n// run the job\nroach.run();\n```\n\n### Multiple Documents, Roach Filter, Multiple Transports\n\n```js\nvar Roach = require('roach'),\n    Proxy = Roach.Proxy;\n\nvar roach = new Roach();\n\n// Use multiple transports with default options\nroach.use('rabbitmq').use('mongodb');\n\nvar proxy = new Proxy( 'http://google.com', 'google' );\n\nproxy.filter('links')\n     .each(function(url, index){\n        roach.addJob(url, 'job:' + index)\n             .filter('filters/xml_parser.js');\n      })\n     .run(function(){\n        roach.run();\n      });\n```\n\n### Reading From File System, Custom Proxy \u0026 Parser Filters\n\n**Note:** Custom filters need to end with `.js` (crappy I know, but it was tricky to automatically resolve your filters vs. our custom ones.)\n\n```js\nvar Roach = require('roach'),\n    Proxy = Roach.Proxy;\n\nvar roach = new Roach();\n\nvar config = roach.config('config.json');\n\nroach.use('redis', config.transports.redis);\n\nvar proxy = new Proxy( '/tmp', 'awesomesauce' );\n\n// Use the Roach directory filter and a custom filter\n// to grab every .txt file the '/tmp' directory\nproxy.filter( 'directory' )\n\t .filter( 'filters/text_file.js' )\n\t .each(function(filepath, index){\n\n\t \t// Apply some custom filters to each job that \n\t \t// gets created from each fetched file\n\t    roach.addJob(filepath, 'job:' + index)\n\t         .filter('filters/special_error.js')\n\t         .filter('filters/every_third_word.js');\n\t })\n\t .run(function(){\n\t    roach.run();\n\t });\n```\n\n## Architecture\n\n![Architecture](docs/img/crawler_architecture.jpg)\n\n### Data Flow\n\n![Data Flow](docs/img/crawler_data_flow.jpg)\n\n### Job\n\nTODO\n\n### Proxy\n\nTODO\n\n### Parser\n\nTODO\n\n### Filters\n\nTODO\n\n### Transports\n\nRoach has multiple transports that can be used to save your data somewhere. They all have default options that are specific to the individual transport. The options can be overridden by passing in an options object like so:\n\n```js\nroach.use('rabbitmq', { host: '192.168.1.1', port: 8001 });\n```\n\nAll the transports should adhere to the same basic interface and they pretty simple to implement so feel free to write your own!\n\n#### Logger\n\nThis is as simple as it gets. It just spits stuff out to console. Useful for debugging or just piping results to a file.\n\n**Defaults:**\n\n```js\n{\n    name : 'logger'\n}\n\n```\n\n#### File\n\nFor saving to file system.\n\n**Defaults:**\n\n```js\n{\n    path : path.resolve('/tmp'),\n    name: 'file',\n    filename: 'data:'\n}\n\n```\n\n#### MongoDB\n\nSaving to a single collection. No replica set support at the moment. Single DB only.\n\n**Defaults:**\n\n```js\n{\n    protocol : 'mongo',\n    name: 'mongodb',\n    host : 'localhost',\n    port : '27017',\n    db: 'roach',\n    collection: 'roach',\n    safe: true\n}\n\n```\n\n\n#### RabbitMQ\n\nObviously, saving to RabbitMQ.\n\n**Defaults:**\n\n```js\n{\n    protocol : 'amqp',\n    name: 'rabbitmq',\n    host : 'localhost',\n    port : '5672',\n    exchange : {\n        name: 'roach',\n        options: {\n            durable : true,\n            confirm : true\n        }\n    },\n    login: 'guest',\n    password: 'guest',\n    vhost: '/',\n    routingKey: '#'\n}\n```\n\n#### Redis (in progress)\n\nSaving to Redis. Right now this is tightly coupled to our own use case, so it currently saves a key to a set and then saves the key + a score + the data to a sorted set. We plan on abstracting so that you can determine your save method or override the default.\n\n**Defaults:**\n\n```js\n{\n    protocol : 'redis',\n    name: 'redis',\n    host : 'localhost',\n    port : 6379\n}\n```\n\n### Utils\n\nWe provide a few helpful utility libraries wrapped up in the `Roach.Utils` namespace. At the moment they are:\n\n* `Roach.Utils._` -\u003e [underscore](http://underscorejs.org)\n* `Roach.Utils._.str` -\u003e [underscore-string](http://epeli.github.io/underscore.string/)\n* `Roach.Utils.date` -\u003e [moment](http://momentjs.com)\n* `Roach.Utils.request` -\u003e Helpful utilities for making http requests\n\n## License\n\n**Creative Commons 3.0 - Attribution Sharealike**\n\nYou can remix, copy or use for both commercial and non-commercial products and services but you need to provide attribution for the original work in the source code to *\"PetroFeed Inc.\"*. You must also share the original or any derivative under the same license. A description of the license can be found [here](http://creativecommons.org/licenses/by-sa/3.0).\n\n---\n\nProudly brought to you by [PetroFeed](http://PetroFeed.com).\n\n\n![Pedro](https://www.petrofeed.com/img/company/pedro.png)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftether%2Froach","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftether%2Froach","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftether%2Froach/lists"}