Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/ahoward/forkoff

brain-dead simple parallel processing for ruby
https://github.com/ahoward/forkoff

Last synced: 17 days ago
JSON representation

brain-dead simple parallel processing for ruby

Awesome Lists containing this project

README 

        
NAME



  forkoff



SYNOPSIS



  brain-dead simple parallel processing for ruby



URI



  http://rubyforge.org/projects/codeforpeople

  http://github.com/ahoward/forkoff



INSTALL



  gem install forkoff



DESCRIPTION



  forkoff works for any enumerable object, iterating a code block to run in a

  child process and collecting the results.  forkoff can limit the number of

  child processes which is, by default, 2.



SAMPLES



  

  <========< samples/a.rb >========>



  ~ > cat samples/a.rb



    # forkoff makes it trivial to do parallel processing with ruby, the following

    # prints out each word in a separate process

    #

    

      require 'forkoff'

    

      %w( hey you ).forkoff!{|word| puts "#{ word } from #{ Process.pid }"}



  ~ > ruby samples/a.rb



    hey from 7907

    you from 7908



  <========< samples/b.rb >========>



  ~ > cat samples/b.rb



    # for example, this takes only 4 seconds or so to complete (8 iterations

    # running in two processes = twice as fast)

    #

    

      require 'forkoff'

    

      a = Time.now.to_f

    

      results =

        (0..7).forkoff do |i|

          sleep 1

          i ** 2

        end

    

      b = Time.now.to_f

    

      elapsed = b - a

    

      puts "elapsed: #{ elapsed }"

      puts "results: #{ results.inspect }"



  ~ > ruby samples/b.rb



    elapsed: 4.19184589385986

    results: [0, 1, 4, 9, 16, 25, 36, 49]



  <========< samples/c.rb >========>



  ~ > cat samples/c.rb



    # forkoff does *NOT* spawn processes in batches, waiting for each batch to

    # complete.  rather, it keeps a certain number of processes busy until all

    # results have been gathered.  in otherwords the following will ensure that 3

    # processes are running at all times, until the list is complete. note that

    # the following will take about 3 seconds to run (3 sets of 3 @ 1 second).

    #

    

    require 'forkoff'

    

    pid = Process.pid

    

    a = Time.now.to_f

    

    pstrees =

      %w( a b c d e f g h i ).forkoff! :processes => 3 do |letter|

        sleep 1

        { letter => ` pstree -l 2 #{ pid } ` }

      end

    

    

    b = Time.now.to_f

    

    puts

    puts "pid: #{ pid }"

    puts "elapsed: #{ b - a }"

    puts

    

    require 'yaml'

    

    pstrees.each do |pstree|

      y pstree

    end



  ~ > ruby samples/c.rb



    

    pid: 7922

    elapsed: 3.37899208068848

    

    --- 

    a: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |-+- 07923 ahoward ruby -Ilib samples/c.rb

       |-+- 07924 ahoward (ruby)

       \-+- 07925 ahoward ruby -Ilib samples/c.rb

    

    --- 

    b: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |-+- 07923 ahoward ruby -Ilib samples/c.rb

       |-+- 07924 ahoward ruby -Ilib samples/c.rb

       \-+- 07925 ahoward ruby -Ilib samples/c.rb

    

    --- 

    c: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |-+- 07923 ahoward ruby -Ilib samples/c.rb

       |-+- 07924 ahoward (ruby)

       \-+- 07925 ahoward ruby -Ilib samples/c.rb

    

    --- 

    d: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |-+- 07932 ahoward ruby -Ilib samples/c.rb

       |--- 07933 ahoward ruby -Ilib samples/c.rb

       \--- 07934 ahoward ruby -Ilib samples/c.rb

    

    --- 

    e: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |--- 07932 ahoward (ruby)

       |-+- 07933 ahoward ruby -Ilib samples/c.rb

       \-+- 07934 ahoward (ruby)

    

    --- 

    f: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |--- 07932 ahoward (ruby)

       |-+- 07933 ahoward ruby -Ilib samples/c.rb

       \-+- 07934 ahoward ruby -Ilib samples/c.rb

    

    --- 

    g: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |-+- 07941 ahoward ruby -Ilib samples/c.rb

       |--- 07942 ahoward ruby -Ilib samples/c.rb

       \--- 07943 ahoward ruby -Ilib samples/c.rb

    

    --- 

    h: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |-+- 07941 ahoward (ruby)

       |-+- 07942 ahoward ruby -Ilib samples/c.rb

       \--- 07943 ahoward ruby -Ilib samples/c.rb

    

    --- 

    i: |

      -+- 07922 ahoward ruby -Ilib samples/c.rb

       |--- 07942 ahoward (ruby)

       \-+- 07943 ahoward ruby -Ilib samples/c.rb

    



  <========< samples/d.rb >========>



  ~ > cat samples/d.rb



    # forkoff supports two strategies of reading the result from the child: via

    # pipe (the default) or via file.  you can select which to use using the

    # :strategy option.

    #

    

      require 'forkoff'

    

      %w( hey you guys ).forkoff :strategy => :file do |word|

        puts "#{ word } from #{ Process.pid }"

      end



  ~ > ruby samples/d.rb



    hey from 7953

    you from 7954

    guys from 7955



HISTORY

  1.1.0 

    - move to a model with one work queue and signals sent from consumers to

    producer to noitify ready state.  this let's smaller jobs race through a

    single process even while a larger job may have one sub-process bound up.

    incorporates a fix from http://github.com/fredrikj/forkoff which meant

    some processes would lag behind when jobs didn't have similar execution

    times.



  1.0.0

    - move to github



  0.0.4

    - code re-org

    - add :strategy option

    - default number of processes is 2, not 8



  0.0.1



    - updated to use producer threds pushing onto a SizedQueue for each consumer

      channel.  in this way the producers do not build up a massize parllel data

      structure but provide data to the consumers only as fast as they can fork

      and proccess it.  basically for a 4 process run you'll end up with 4

      channels of size 1 between 4 produces and 4 consumers, each consumer is a

      thread popping of jobs, forking, and yielding results.



    - removed use of Queue for capturing the output.  now it's simply an array

      of arrays which removed some sync overhead.



    - you can configure the number of processes globally with



        Forkoff.default['proccess'] = 4



    - you can now pass either an options hash



        forkoff( :processes => 2 ) ...



      or plain vanilla number



        forkoff( 2 ) ...



      to the forkoff call



    - default number of processes is 8, not 2

        



  0.0.0



    initial version