{"id":19083133,"url":"https://github.com/vikhyat/stormycloud","last_synced_at":"2025-04-30T08:49:47.272Z","repository":{"id":3956826,"uuid":"5050169","full_name":"vikhyat/StormyCloud","owner":"vikhyat","description":"Ridiculously simple distributed applications in Ruby.","archived":false,"fork":false,"pushed_at":"2014-03-31T13:20:15.000Z","size":264,"stargazers_count":9,"open_issues_count":1,"forks_count":2,"subscribers_count":3,"default_branch":"master","last_synced_at":"2024-09-19T23:12:22.578Z","etag":null,"topics":["distributed-systems","mapreduce","ruby"],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"pkp/ojs","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/vikhyat.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-07-14T19:02:04.000Z","updated_at":"2019-07-15T21:14:10.000Z","dependencies_parsed_at":"2022-09-14T15:10:24.747Z","dependency_job_id":null,"html_url":"https://github.com/vikhyat/StormyCloud","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikhyat%2FStormyCloud","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikhyat%2FStormyCloud/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikhyat%2FStormyCloud/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/vikhyat%2FStormyCloud/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/vikhyat","download_url":"https://codeload.github.com/vikhyat/StormyCloud/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223780076,"owners_count":17201287,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","mapreduce","ruby"],"created_at":"2024-11-09T02:46:11.746Z","updated_at":"2024-11-09T02:46:12.330Z","avatar_url":"https://github.com/vikhyat.png","language":"Ruby","readme":"StormyCloud\n-----------\n\n**Goal:** Make it _ridiculously_ easy to write simple distributed application\nin Ruby.\n\n[![Build Status](https://secure.travis-ci.org/vikhyat/StormyCloud.png?branch=master)](http://travis-ci.org/vikhyat/StormyCloud)\n\nInstallation\n------------\n\nStormyCloud can be installed using RubyGems:\n\n    $ gem install stormy-cloud\n\nUsage\n-----\n\nHere's an example that will compute the sum of the squares of the first 1000\nnumbers:\n\n```ruby\nrequire 'stormy-cloud'\n\nStormyCloud.new(\"square_summation\", \"10.6.2.213\") do |c|\n\n  c.split { (1..1000).to_a }\n\n  c.map do |t|\n    sleep 2   # do some work\n    t ** 2\n  end\n\n  c.reduce do |t, r|\n    @sum ||= 0\n    @sum += r\n  end\n\n  c.finally do\n    puts @sum\n  end\n\nend\n```\n\nYou _must_ specify the three blocks, `split`, `map` and `reduce`. The `finally`\nblock is optional, and will be called when the job is completed.\n\nThe `split` function must return an array of smaller sub-tasks which can be\ncompleted in parallel.\n\nThe `map` function must take one of these sub-tasks as input and return the\nresult of the computation.\n\nThe `reduce` block is called once for each task and its result.\n\n`split`, `reduce` and `finally` will be run on a central server, but `map` will\nbe run on worker nodes.\n\nThe values returned by `split`, `reduce` and `finally` should be serializable to JSON.\n\nConfiguration\n-------------\n\nSome configuration variables can be set inside the block, as shown below:\n\n```ruby\nStormyCloud.new(\"square_summation\", \"10.6.2.213\") do |c|\n  c.config :wait, 20\n  c.config :port, 9861\n  c.config :debug, true\n\n  [...]\nend\n```\n\nCurrently, the only supported configuration variables are:\n\n  * **wait**: Amount of time to wait for a result from the node before\nreturning a task to the node.\n  * **port**: When using the TCP transport, this is the TCP port used on the\nserver.\n  * **debug**: When this is set to true, the entire task will be run on a\nsingle machine sequentially.\n\nEmit API\n--------\n\nBy default the reduce function is passed a key-value pair (t, r) where `t` is the\noriginal task and `r` is the value returned the the `map` function when it is\ncalled with the task `t`. In some cases, we require that the `map` function emit\nan arbitrary number of key-value pairs to be reduced. For that purpose it is\npossible to call `emit` inside the map function any number of times to emit\narbitrary key-value pairs to be reduced.\n\nFor example:\n\n    c.map do |task|\n      emit \"key1\", \"value1\"\n      emit \"key2\", \"value2\"\n    end\n\nIf your `map` function calls the `emit` function then the default key-value pair of\n(task, return_value) will not be emitted.\n\nRunning a Job\n-------------\n\nRunning a job is as simple as copying a file onto the nodes and running a\ncommand.\n\nFirst, make sure that the machine that will act as the central server and the\nones which will be nodes all have Ruby installed along with the gem. Also make\nsure the script contains the correct IP address of the actual server.\n\nStart the server by running:\n\n    $ ruby job.rb server\n\nThen log in to each of the nodes and run the following commands:\n\n    $ ruby job.rb node\n\nWhen the server is run, a HTTP server will be spawned on that machine on port\n4567, which can be used to track the progress of the job. It will show the\nconnected nodes, currently assigned tasks the estimated time to completion.\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvikhyat%2Fstormycloud","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fvikhyat%2Fstormycloud","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fvikhyat%2Fstormycloud/lists"}