{"id":16559327,"url":"https://github.com/anders94/disruptor","last_synced_at":"2025-03-21T11:31:29.685Z","repository":{"id":7717017,"uuid":"9082505","full_name":"anders94/disruptor","owner":"anders94","description":"A distributed real-time computation platform in node.js.","archived":false,"fork":false,"pushed_at":"2014-05-31T22:44:24.000Z","size":432,"stargazers_count":29,"open_issues_count":0,"forks_count":4,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-03-13T03:13:27.236Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/anders94.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2013-03-28T17:31:04.000Z","updated_at":"2024-01-30T06:48:39.000Z","dependencies_parsed_at":"2022-09-11T07:21:25.312Z","dependency_job_id":null,"html_url":"https://github.com/anders94/disruptor","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fdisruptor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fdisruptor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fdisruptor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/anders94%2Fdisruptor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/anders94","download_url":"https://codeload.github.com/anders94/disruptor/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243933419,"owners_count":20370986,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-11T20:25:34.701Z","updated_at":"2025-03-21T11:31:29.386Z","avatar_url":"https://github.com/anders94.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"Disruptor\n=========\n\n\u003cimg src=\"http://anders.com/1offs/disruptor.png\" width=\"400\" height=\"228\" alt=\"disruptor\" align=\"right\" /\u003e\n\n**Disruptor** is a distributed realtime computation system for node.js. Disruptor makes it\neasy to process unbounded streams of data across a mesh of many machines. It has minimal \nconfiguration requirements and no single point of failure.\n\nPeers are started given the address and port of another peer. They quickly find all the other peers \nin the mesh without additional configuration. Worker programs are written in Javascript as \nindependant node.js applications which get spawned by each peer in the cluster and run continuously. \nAs work comes in, (json payloads via http requests) it is processed via a random live worker. \nResults come back to the requestor as HTTP responses via json payloads.\n\nThere is no master peer, monitoring peer or other single point of failure. The design stresses \nsimplicity wherever possible and requires minimal setup.\n\n**Note:** Some things, such as the automatic packaging and distribution of client applications, \nare not yet implemented. This is still a work in progress.\n\nInstall\n-----\n    npm install -g disruptor\n\nor for latest code:\n\n    git clone https://github.com/anders94/disruptor.git\n    cd disruptor/\n    npm install\n\nUsage\n-----\nDisruptor takes an IP and port on which to listen and the IP and port of some other peer in the \nmesh. All the peers will find each other and stay in communication as peers enter and leave \nthe mesh.\n\n    disruptor peer myHost:myPort anotherHost:itsPort\n\nExample\n-------\nYou can simulate a mesh of peers running on a single host. In a shell:\n\n    disruptor peer 127.0.0.1:1111 127.0.0.1:22222\n\nIn another shell:\n\n    disruptor peer 127.0.0.1:2222 127.0.0.1:11111\n\nThe peers should find each other. Start a few more peers and point each to one of the running peers \nin the mesh. They should all find eachother other.\n\nTo see what other peers a disruptor peer knows about, visit it with a web browser:\n\n    http://127.0.0.1:1111\n\nIn a production environment, rather than 127.0.0.1, you would use a network accessible interface\nand run one peer on each machine.\n\nThere is no setup beyond this. Peers that die or become inaccessible should be automatically \nremoved from the mesh over time. Simulate this by shutting down a peer and watch as the other\npeers eventually forget about it. Bringing it back in similarly should be seemless. Taking\npeers out of the mesh and replacing them is a common operation and shouldn't adversely effect \nthe mesh.\n\nBe careful about running directly addressible peers on the live Internet. There isn't security \nyet so, although highly improbible, if someone else's mesh were to find your mesh, the meshes \nwould attempt to merge. Usually meshes are run on private address ranges such as 192.168.0.0/16, \n172.16.0.0/12 or 10.0.0.0/8 without direct addressibility from the Internet.\n\nCreating Worker Apps\n--------------------\nWorker applications exist in directories under apps/ (for example apps/wordcount) and respond to:\n\n```javascript\nprocess.on('message', function() { ... }) \n```\n\nThey emit results with:\n\n```javascript\nprocess.send( ... );\n```\n\nFor example, here is a word counting worker that might exist in apps/wordcount/counter.js:\n\n```javascript\nvar natural = require('natural'),\n    tokenizer = new natural.WordTokenizer();\n\nprocess.on('message', function(message) {\n        var total = 0, unique = 0;\n        var hash = {};\n        var ary = tokenizer.tokenize(message);\n        for (var id in ary) { // throw stemmed word into hash\n            hash[natural.PorterStemmer.stem(ary[id])] = true;\n            total ++;\n        }\n\n        for (var key in hash) // count unique word stems\n            unique ++;\n\n        process.send({ message: message, total: total, unique: unique });\n    });\n```\n\n**Note:** For this example you will need to:\n\n    npm install natural\n\nGiven this input:\n```\nThe First World War was to be the war to end all wars.\n```\n\nyou should get this output:\n```\n{ message: 'The First World War was to be the war to end all wars.',\n    total: 13,\n   unique: 9 }\n```\n\nWorker apps, once started, run continuously and can handle requests and send responses at \nany time. Any number of differently named workers can run on the same node at the same time.\n\n**Note:** Worker apps and any npm packages used in worker apps need to be installed on every \nnode. Disruptor intends to eventually do this automatically if the modules are installed \nlocally to each app (ie: apps/wordcount/npm_modules for the above example) but this \nfunctionality is not yet implemented. Currently, you must sync the app directory with all \nthe peers. A good command to use for this is rsync for the time being:\n\n    rsync -ae ssh ~/disruptor/apps 1.2.3.4:~/disruptor\n\nIf you don't install an app's modules locally, (ie: apps/wordcount/npm_modules) you must also\ninstall any requirements on each peer as well:\n\n    npm install natural\n\nwhich will install the dependancy in disruptor's main node_modules directory. This is less than\nideal and should be avoided.\n\nStarting Workers\n----------------\nWorkers are started by telling one of the running nodes to tell all the peers it knows about \nto start a particular application. With the first example mesh still running, we might do this:\n\n    disruptor start 127.0.0.1:1111 apps/wordcount/counter\n\nStopping all the workers is done similarly.\n\n    disruptor stop 127.0.0.1:1111 apps/wordcount/counter\n\nIn the future, starting a job will first make sure it runs locally, package it up into a \ncompressed archive, distribute it and then start it on all known peers.\n\n**Note: This functionality is under active developed.**\n\nSending Compute Tasks to Workers\n--------------------------------\nYou can send json payloads to be processed to any peer in the mesh through an HTTP request. \nThe peer you choose will be considered the master for this session and will pick a single \nrandom worker for the task. Responses will flow back from the random worker to the temporary \nmaster and then back to the requestor.\n\n    disruptor send 127.0.0.1:1111 apps/wordcount/counter \\\n    \"{'the quick brown fox jumped over the lazy dog'}\"\n\nAlternatively, you can send requests directly via HTTP:\n\n    $ curl -X POST -H \"content-type: application/json\" \\\n        http://127.0.0.1:1111/apps/wordcount/counter --data @-\u003c\u003c\\EOF\n    {'the quick brown fox jumped over the lazy dog'}\n    EOF\n\nJSON results come back in the body of the HTTP response.\n\n    { message: 'the quick brown fox jumped over the lazy dog',\n        total: 9,\n       unique: 8 }\n\nA requestor only needs to know the address of one of the peers in the mesh to send jobs. The \nknowledge of active workers in the mesh and the random distribution of work is handled by the \npeer automatically. In this way, a single known entry point into the mesh is all that is \nnecessary to run jobs distributed across the mesh.\n\n**Note: This functionality is under active development.**\n\nAuthor\n------\n**Anders Brownworth**\n\n+ [http://twitter.com/anders94](@anders94)\n+ [http://github.com/anders94](github.com/anders94)\n+ [http://anders.com/](anders.com)\n\nPlease get in touch if you would like to contribute.\n\nAre You Using This? Let me know!\n--------------------------------\nI started this project to do distributed natural language processing and machine \nlearning. However, I'm sure the need to massively distribute node.js processing \nexists for many other jobs. I'm interested in solving real-world problems with \ndisruptor so it is useful to know what jobs it is or isn't solving. Please tweet \n[http://twitter.com/anders94](@anders94) or otherwise get in touch.\n\nCopyright and license\n---------------------\nCopyright 2013 Anders Brownworth\n\nLicensed under the Apache License, Version 2.0 (the \"License\"); you may not use this work except \nin compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:\n\n  [http://www.apache.org/licenses/LICENSE-2.0](apache.org/licenses/LICENSE-2.0)\n\nUnless required by applicable law or agreed to in writing, software distributed under the \nLicense is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either \nexpress or implied. See the License for the specific language governing permissions and\nlimitations under the License.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanders94%2Fdisruptor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fanders94%2Fdisruptor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fanders94%2Fdisruptor/lists"}