{"id":13483586,"url":"https://github.com/ahoward/forkoff","last_synced_at":"2025-03-27T14:31:22.748Z","repository":{"id":598293,"uuid":"233807","full_name":"ahoward/forkoff","owner":"ahoward","description":"brain-dead simple parallel processing for ruby","archived":false,"fork":false,"pushed_at":"2014-03-14T13:50:16.000Z","size":145,"stargazers_count":73,"open_issues_count":0,"forks_count":5,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-10-30T17:48:00.613Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Ruby","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ahoward.png","metadata":{"files":{"readme":"README","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2009-06-22T22:22:36.000Z","updated_at":"2024-07-03T02:41:55.000Z","dependencies_parsed_at":"2022-07-18T10:48:46.473Z","dependency_job_id":null,"html_url":"https://github.com/ahoward/forkoff","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahoward%2Fforkoff","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahoward%2Fforkoff/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahoward%2Fforkoff/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ahoward%2Fforkoff/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ahoward","download_url":"https://codeload.github.com/ahoward/forkoff/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245863090,"owners_count":20684787,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-07-31T17:01:13.080Z","updated_at":"2025-03-27T14:31:22.457Z","avatar_url":"https://github.com/ahoward.png","language":"Ruby","readme":"NAME\n\n  forkoff\n\nSYNOPSIS\n\n  brain-dead simple parallel processing for ruby\n\nURI\n\n  http://rubyforge.org/projects/codeforpeople\n  http://github.com/ahoward/forkoff\n\nINSTALL\n\n  gem install forkoff\n\nDESCRIPTION\n\n  forkoff works for any enumerable object, iterating a code block to run in a\n  child process and collecting the results.  forkoff can limit the number of\n  child processes which is, by default, 2.\n\nSAMPLES\n\n  \n  \u003c========\u003c samples/a.rb \u003e========\u003e\n\n  ~ \u003e cat samples/a.rb\n\n    # forkoff makes it trivial to do parallel processing with ruby, the following\n    # prints out each word in a separate process\n    #\n    \n      require 'forkoff'\n    \n      %w( hey you ).forkoff!{|word| puts \"#{ word } from #{ Process.pid }\"}\n\n  ~ \u003e ruby samples/a.rb\n\n    hey from 7907\n    you from 7908\n\n\n  \u003c========\u003c samples/b.rb \u003e========\u003e\n\n  ~ \u003e cat samples/b.rb\n\n    # for example, this takes only 4 seconds or so to complete (8 iterations\n    # running in two processes = twice as fast)\n    #\n    \n      require 'forkoff'\n    \n      a = Time.now.to_f\n    \n      results =\n        (0..7).forkoff do |i|\n          sleep 1\n          i ** 2\n        end\n    \n      b = Time.now.to_f\n    \n      elapsed = b - a\n    \n      puts \"elapsed: #{ elapsed }\"\n      puts \"results: #{ results.inspect }\"\n\n  ~ \u003e ruby samples/b.rb\n\n    elapsed: 4.19184589385986\n    results: [0, 1, 4, 9, 16, 25, 36, 49]\n\n\n  \u003c========\u003c samples/c.rb \u003e========\u003e\n\n  ~ \u003e cat samples/c.rb\n\n    # forkoff does *NOT* spawn processes in batches, waiting for each batch to\n    # complete.  rather, it keeps a certain number of processes busy until all\n    # results have been gathered.  in otherwords the following will ensure that 3\n    # processes are running at all times, until the list is complete. note that\n    # the following will take about 3 seconds to run (3 sets of 3 @ 1 second).\n    #\n    \n    require 'forkoff'\n    \n    pid = Process.pid\n    \n    a = Time.now.to_f\n    \n    pstrees =\n      %w( a b c d e f g h i ).forkoff! :processes =\u003e 3 do |letter|\n        sleep 1\n        { letter =\u003e ` pstree -l 2 #{ pid } ` }\n      end\n    \n    \n    b = Time.now.to_f\n    \n    puts\n    puts \"pid: #{ pid }\"\n    puts \"elapsed: #{ b - a }\"\n    puts\n    \n    require 'yaml'\n    \n    pstrees.each do |pstree|\n      y pstree\n    end\n\n  ~ \u003e ruby samples/c.rb\n\n    \n    pid: 7922\n    elapsed: 3.37899208068848\n    \n    --- \n    a: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |-+- 07923 ahoward ruby -Ilib samples/c.rb\n       |-+- 07924 ahoward (ruby)\n       \\-+- 07925 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    b: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |-+- 07923 ahoward ruby -Ilib samples/c.rb\n       |-+- 07924 ahoward ruby -Ilib samples/c.rb\n       \\-+- 07925 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    c: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |-+- 07923 ahoward ruby -Ilib samples/c.rb\n       |-+- 07924 ahoward (ruby)\n       \\-+- 07925 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    d: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |-+- 07932 ahoward ruby -Ilib samples/c.rb\n       |--- 07933 ahoward ruby -Ilib samples/c.rb\n       \\--- 07934 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    e: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |--- 07932 ahoward (ruby)\n       |-+- 07933 ahoward ruby -Ilib samples/c.rb\n       \\-+- 07934 ahoward (ruby)\n    \n    --- \n    f: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |--- 07932 ahoward (ruby)\n       |-+- 07933 ahoward ruby -Ilib samples/c.rb\n       \\-+- 07934 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    g: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |-+- 07941 ahoward ruby -Ilib samples/c.rb\n       |--- 07942 ahoward ruby -Ilib samples/c.rb\n       \\--- 07943 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    h: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |-+- 07941 ahoward (ruby)\n       |-+- 07942 ahoward ruby -Ilib samples/c.rb\n       \\--- 07943 ahoward ruby -Ilib samples/c.rb\n    \n    --- \n    i: |\n      -+- 07922 ahoward ruby -Ilib samples/c.rb\n       |--- 07942 ahoward (ruby)\n       \\-+- 07943 ahoward ruby -Ilib samples/c.rb\n    \n\n\n  \u003c========\u003c samples/d.rb \u003e========\u003e\n\n  ~ \u003e cat samples/d.rb\n\n    # forkoff supports two strategies of reading the result from the child: via\n    # pipe (the default) or via file.  you can select which to use using the\n    # :strategy option.\n    #\n    \n      require 'forkoff'\n    \n      %w( hey you guys ).forkoff :strategy =\u003e :file do |word|\n        puts \"#{ word } from #{ Process.pid }\"\n      end\n\n  ~ \u003e ruby samples/d.rb\n\n    hey from 7953\n    you from 7954\n    guys from 7955\n\n\n\nHISTORY\n  1.1.0 \n    - move to a model with one work queue and signals sent from consumers to\n    producer to noitify ready state.  this let's smaller jobs race through a\n    single process even while a larger job may have one sub-process bound up.\n    incorporates a fix from http://github.com/fredrikj/forkoff which meant\n    some processes would lag behind when jobs didn't have similar execution\n    times.\n\n  1.0.0\n    - move to github\n\n  0.0.4\n    - code re-org\n    - add :strategy option\n    - default number of processes is 2, not 8\n\n  0.0.1\n\n    - updated to use producer threds pushing onto a SizedQueue for each consumer\n      channel.  in this way the producers do not build up a massize parllel data\n      structure but provide data to the consumers only as fast as they can fork\n      and proccess it.  basically for a 4 process run you'll end up with 4\n      channels of size 1 between 4 produces and 4 consumers, each consumer is a\n      thread popping of jobs, forking, and yielding results.\n\n    - removed use of Queue for capturing the output.  now it's simply an array\n      of arrays which removed some sync overhead.\n\n    - you can configure the number of processes globally with\n\n        Forkoff.default['proccess'] = 4\n\n    - you can now pass either an options hash\n\n        forkoff( :processes =\u003e 2 ) ...\n\n      or plain vanilla number\n\n        forkoff( 2 ) ...\n\n      to the forkoff call\n\n    - default number of processes is 8, not 2\n        \n\n  0.0.0\n\n    initial version\n","funding_links":[],"categories":["Processes and Threads","Concurrency and Parallelism"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahoward%2Fforkoff","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fahoward%2Fforkoff","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fahoward%2Fforkoff/lists"}