https://github.com/doches/clusterfuck

A subversive distributed systems tool
https://github.com/doches/clusterfuck

Last synced: 10 months ago
JSON representation

A subversive distributed systems tool

• Host: GitHub
• URL: https://github.com/doches/clusterfuck
• Owner: doches
• License: mit
• Created: 2009-10-15T22:33:32.000Z (almost 17 years ago)
• Default Branch: master
• Last Pushed: 2010-09-06T13:30:51.000Z (almost 16 years ago)
• Last Synced: 2025-10-02T23:21:07.952Z (10 months ago)
• Language: Ruby
• Size: 105 KB
• Stars: 6
• Watchers: 4
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.rdoc
  • License: LICENSE

Awesome Lists containing this project

README

          
= Clusterfuck

==== A Subversive Distributed-Systems Tool

Clusterfuck is a tool for automating the process of SSH-ing into remote machines and kickstarting a large number

of jobs. It's probably best explained by an example, so here's what I use it for:

As part of my research I need to compute the distance between each pair of objects in a set of about 70,000 items.

Computing the distance between each pair takes a few seconds; running the entire job on a single machine generally takes over a day.

However, as a member of the University I have a ssh login that works on quite a few machines, so I found myself breaking the job up into smaller, quicker chunks and running each chunk on a different machine.

Clusterfuck was born out of my frustration with that method -- "surely," I said to myself, "this can be automated."

If you have a lot of jobs to run and access to multiple machines on which to run them, Clusterfuck is for you!

== Usage

To use Clusterfuck you'll first need to create a configuration file (a "clusterfile"). An example clusterfile might look something like this:

  

  Clusterfuck::Task.new do |task|

    task.hosts = %w{clark asimov}

    task.jobs = (0..3).map { |x| Clusterfuck::Job.new("host{x}","sleep 0.5 && hostname") }

    task.temp = "fragments"

    task.username = "SSHUSERNAME"

    task.password = "SSHPASSWORD"

    task.debug = true

  end

This creates a new clusterfuck task and distributes the jobs across two hosts, +clark+ and +asimov+. 

The jobs to be run in this case are pretty trivial; we basically ssh into each machine, sleep for a little bit, then get the hostname.

Whatever each job prints to stdout is saved in +task+.+temp+ (under the current working directory); running

this clusterfile will create 4 files in ./fragments/: host0.[hostname], host1.[hostname], host2.[hostname], and host3.[hostname] (where [hostname] is the name of the machine on which the job was run). 

+task+.+username+ and +task+.+password+ are the SSH credentials used to log into the maching -- currently, Clusterfuck

can only use one global set of credentials. There's no technical reason for this, other than the fact that I don't

really need to use machine-specific logins, so it'll probably appear in future releases.

+task+.+verbose+ turns on verbose output (messages to stdout each time a job is started, skipped, or canceled).

Once you have a clusterfile you can kick off your jobs by running the command +clusterfuck+ in the same directory.

== Note on Patches/Pull Requests

 

* Fork the project.

* Add something cool or fix a nefarious bug. Documentation wins extra love.

* Add tests for it. I'd really like this, but since I haven't written any tests myself yet I can't really blame you if you skip it...

* Commit, but do not mess with rakefile, version, or history.

  (if you want to have your own version that's ok -- but

  bump the version in a separate commit that I can ignore when I pull)

* Send me a pull request. 

== Copyright

Copyright (c) 2009 Trevor Fountain. See LICENSE for details.