{"id":20935026,"url":"https://github.com/leandrosardi/pampa","last_synced_at":"2025-07-17T23:40:00.159Z","repository":{"id":103753677,"uuid":"219510919","full_name":"leandrosardi/pampa","owner":"leandrosardi","description":"Ruby library for async \u0026 distributed computing, supporting dynamic reconfiguration, distribution of the computation jobs, error handling, job-retry and fault tolerance, and fast (non-direct) communication to ensure real-time capabilities.","archived":false,"fork":false,"pushed_at":"2024-08-03T09:56:11.000Z","size":1858,"stargazers_count":1,"open_issues_count":9,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-16T07:50:34.190Z","etag":null,"topics":["distributed-computing","distributed-task-processing","distributed-task-scheduling","distributed-tasks","elastic-data-processing","supercomputing"],"latest_commit_sha":null,"homepage":"","language":"CSS","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/leandrosardi.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2019-11-04T13:42:40.000Z","updated_at":"2025-03-30T10:32:39.000Z","dependencies_parsed_at":"2024-04-23T21:33:45.276Z","dependency_job_id":"358fb8e1-9a9d-4a62-b2b8-c343b35f9981","html_url":"https://github.com/leandrosardi/pampa","commit_stats":null,"previous_names":[],"tags_count":16,"template":false,"template_full_name":null,"purl":"pkg:github/leandrosardi/pampa","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leandrosardi%2Fpampa","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leandrosardi%2Fpampa/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leandrosardi%2Fpampa/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leandrosardi%2Fpampa/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/leandrosardi","download_url":"https://codeload.github.com/leandrosardi/pampa/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/leandrosardi%2Fpampa/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":265537541,"owners_count":23784420,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-computing","distributed-task-processing","distributed-task-scheduling","distributed-tasks","elastic-data-processing","supercomputing"],"created_at":"2024-11-18T22:12:34.190Z","updated_at":"2025-07-17T23:40:00.133Z","avatar_url":"https://github.com/leandrosardi.png","language":"CSS","funding_links":[],"categories":[],"sub_categories":[],"readme":"\n![Gem version](https://img.shields.io/gem/v/pampa) ![Gem downloads](https://img.shields.io/gem/dt/pampa)\n\n# Pampa - Async \u0026 Distributed Data Processing\n\n**Pampa** is a Ruby library for async \u0026 distributed computing, providing the following features:\n\n- cluster-management with dynamic reconfiguration (joining and leaving nodes);\n- distribution of the computation jobs to the (active) nodes;\n- error handling, job-retry, and fault tolerance;\n- fast (non-direct) communication to ensure real-time capabilities.\n\nThe **Pampa** framework may be widely used for:\n\n- large scale web scraping with what we call a \"bot-farm\";\n- payments processing for large-scale eCommerce websites;\n- reports generation for highly demanded SaaS platforms;\n- heavy mathematical model computing;\n\nand any other tasks that require a virtually infinite amount of CPU computing and memory resources.\n\nAs a final words, **Pampa** supports [PostrgreSQL](https://www.postgresql.org) and [CockroachDB](https://www.cockroachlabs.com), and it has been tested on [Ubuntu 18.04](https://releases.ubuntu.com/18.04/) and [Ruby 3.1.2p20](https://www.ruby-lang.org/en/news/2022/04/12/ruby-3-1-2-released/).\n\n**Outline**\n\n1. [Installation](#1-installation)\n2. [Getting Started](#2-getting-started)\n3. [Define Your Cluster](#3-define-your-cluster)\n4. [Define a Job](#4-define-a-job)\n5. [Setup Your Database](#5-setup-your-database)\n6. [Connect To Your Database](#6-connect-to-your-database)\n7. [Running Dispatcher](#7-running-dispatcher)\n8. [Running Workers](#8-running-workers)\n9. [Selection Snippet](#9-selection-snippet)\n10. [Relaunching Snippet](#10-relaunching-snippet)\n11. [Elastic Workers Assignation](#11-elastic-workers-assignation)\n12. [Reporting](#12-reporting)\n13. [Stand Alone Processes](#13-stand-alone-processes)\n\n## 1. Installation\n\n```cmd\ngem install pampa\n```\n\n## 2. Getting Started\n\nThis example set up a **cluster** of **workers** to process all the numbers from 1 to 100,000; and build the list of odd numbers inside such a range.\n\nCreate a new file `~/config.rb` where you will define your **cluster**, **jobs**, **database connection** and **logging**.\n\n```bash\ntouch ~/config.rb\n```\n\nAdditionally, you may want to add the path `~` to your the environment varaible `RUBYLIB`, in order to require the `config.rb` file from your Ruby code.\n\n```bash\nexport RUBYLIB=~\n```\n\n## 3. Define Your Cluster\n\nAs a first step, you have to define a **cluster** of **workers**.\n\n- A **cluster** is composed by one or more **nodes** (computers).\n\n- Each **worker** is a process running on a **node**.\n\nThe code below define your computer as a **node** of 10 **workers**. \n\nAdd this code to your `config.rb` file:\n\n```ruby\n# setup one or more nodes (computers) where to launch worker processes\nBlackStack::Pampa.add_nodes(\n  [\n    {\n      :name =\u003e 'local',\n      # setup SSH connection parameters\n      :net_remote_ip =\u003e '127.0.0.1',  \n      :ssh_username =\u003e '\u003cyour ssh username\u003e', # example: root\n      :ssh_port =\u003e 22,\n      :ssh_password =\u003e '\u003cyour ssh password\u003e',\n      # setup max number of worker processes\n      :max_workers =\u003e 2,\n    },\n  ]\n)\n```\n\n## 4. Define a Job\n\nA **job** is a sequence of **tasks**. \n\nEach **task** performs the same **function** on different records of a table.\n\nThe code below is for processing all records into a table `numbers`, and update the field `is_odd` with `true` or `false`.\n\nAdd this code to your `config.rb` file:\n\n```ruby\n# setup the cluster\nBlackStack::Pampa.add_job({\n  :name =\u003e 'search_odd_numbers',\n\n  # any worker can be assigned for processing this job.\n  :filter_worker_id =\u003e /.*/,\n  \n  # no more than 5 workers can be assigned for processing this job.\n  :max_assigned_workers =\u003e 5,\n      \n  # add more workers if the number of pending tasks is over 5.\n  :max_pending_tasks =\u003e 500,\n\n  # Minimum number of tasks that a worker must have in queue.\n  # Default: 5\n  :queue_size =\u003e 5, \n  \n  # Maximum number of minutes that a task should take to process.\n  # If a tasks didn't finish X minutes after it started, it is restarted and assigned to another worker.\n  # Default: 15\n  :max_job_duration_minutes =\u003e 15,  \n  \n  # Maximum number of times that a task can be restarted.\n  # Default: 3\n  :max_try_times =\u003e 3,\n\n  # Define the tasks table: each record is a task.\n  # The tasks table must have some specific fields for handling the tasks dispatching.\n  :table =\u003e :numbers, # Note, that we are sending a class object here\n  :field_primary_key =\u003e :value,\n  :field_id =\u003e :odd_checking_reservation_id,\n  :field_time =\u003e :odd_checking_reservation_time, \n  :field_times =\u003e :odd_checking_reservation_times,\n  :field_start_time =\u003e :odd_checking_start_time,\n  :field_end_time =\u003e :odd_checking_end_time,\n  :field_success =\u003e :odd_checking_success,\n  :field_error_description =\u003e :odd_checking_error_description,\n\n  # Function to execute for each task.\n  :processing_function =\u003e Proc.new do |task, l, job, worker, *args|\n    l.logs 'Checking if '+task[:value].to_s+' is odd... '\n    if task[:value] % 2 == 0\n      task[:is_odd] = false\n      l.logf 'No.'.red\n    else\n      task[:is_odd] = true\n      l.logf 'Yes.'.green\n    end\n    DB[:numbers].where(:value=\u003etask[:value]).update(:is_odd=\u003etask[:is_odd])\n  end\n})\n```\n\n## 5. Setup Your Database\n\nObviously, you have to create the table in your database.\nAnd you have to insert some seed data too.\n\n[Here is the PostgreSQL script you have to run the example in this tutorial](https://github.com/leandrosardi/pampa/blob/master/examples/demo.sql).\n\n## 6. Connect To Your Database\n\nIn order to operate with the table `numbers`, you have to connect **Pampa** to your database.\n\nAdd this code to your `config.rb` file:\n\n```ruby\n# DB ACCESS - KEEP IT SECRET\n# Connection string to the demo database: export DATABASE_URL='postgresql://demo:\u003cENTER-SQL-USER-PASSWORD\u003e@free-tier14.aws-us-east-1.cockroachlabs.cloud:26257/mysaas?sslmode=verify-full\u0026options=--cluster%3Dmysaas-demo-6448'\nBlackStack::PostgreSQL::set_db_params({ \n  :db_url =\u003e '89.116.25.250', # n04\n  :db_port =\u003e '5432', \n  :db_name =\u003e 'micro.data', \n  :db_user =\u003e 'blackstack', \n  :db_password =\u003e '*****',\n  :db_sslmode =\u003e 'disable',\n})\n```\n\n## 7. Running Dispatcher\n\nThe **dispatcher** will run an infinite loop, assigning tasks to each **worker** at each iteration of such a loop. \n\n**Step 1:** Create your `dispatcher.rb` script-file.\n\n```\ntouch ~/dispatcher.rb\n```\n\n**Step 2:** Write this code into your `dispatcher.rb` file.\n\n```ruby\nrequire 'pampa/dispatcher'\n```\n\n**Step 3:** Run the dispatcher.\n\nRun the command below on your `local` node in order to run your worker.\n\n```\nexport RUBYLIB=~/\nruby ~/dispatcher.rb\n```\n\n**Parameters:**\n\n1. **delay:** You may set the delay in seconds between iterations:\n\n```\nruby ~/dispatcher.rb delay=30\n```\n\n2. **config:** Be default, `dispatcher.rb` will `require 'config.rb'`. That is why you have to execute `export RUBYLIB=~/` before running the dispatcher. Though, you can define a custom location of the configuration file too.\n\n```\nruby ~/dispatcher.rb config=~/foo/config.rb\n```\n\n3. **db:** Use this parameters to choose a database driver. The supported values are: `postgres`, `crdb`. By default, it is: `postgres`.\n\n```\nruby ~/dispatcher.rb db=crdb\n```\n\n4. **log:** Use this parameter to indicate the process to write the log in the file `./dispatcher.log` or not. The default value is `yes`.\n\n```\nruby ~/dispatcher.rb log=no\n```\n\n**Code Snippets:**\n\nYou can define a code to be executed right after **dispatcher** connected to the database.\n\nAdd a code like this in your `config.rb` file.\n\n```ruby\n# Pampa Code Snippets\nBlackStack::Pampa.set_snippets({\n  :dispatcher_function =\u003e Proc.new do |l, *args|\n    require 'my-project/model'\n  end,\n})\n```\n\nYou can also pass a logger to the snippet.\n\n```ruby\n# Pampa Code Snippets\nBlackStack::Pampa.set_snippets({\n  :dispatcher_function =\u003e Proc.new do |l, *args|\n    l = BlackStack::DummyLogger.new(nil) if l.nil?\n\n    l.logs 'Loading model.. '\n    require 'my-project/model'\n    l.logf 'done'.green\n  end,\n})\n```\n\n## 8. Running Workers\n\nThe **worker** will run an infineet loop, processing all assigned tasks at each iteration of such a loop. \n\n**Step 1:** Create a new file `worker.rb` file.\n\n```\ntouch ~/worker.rb\n```\n\n**Step 2:** Write this code into your `worker.rb` file.\n\n```ruby\nrequire 'pampa/worker'\n```\n\n**Step 3:** Run a worker.\n\nRun the command below on your `local` node in order to run your worker.\n\n```\nexport RUBYLIB=~/\nruby ~/worker.rb id=localhost.1\n```\n\n**Parameters:**\n\n1. **delay:** You may set the delay in seconds between iterations:\n\n```\nruby ~/worker.rb id=localhost.1 delay=30\n```\n\n2. **config:** Be default, `worker.rb` will `require 'config.rb'`. That is why you have to execute `export RUBYLIB=~/` before running the dispatcher. Though, you can define a custom location of the configuration file too.\n\n```\nruby ~/worker.rb id=localhost.1 config=~/foo/config.rb\n```\n\n3. **db:** Use this parameters to choose a database driver. The supported values are: `postgres`, `crdb`. By default, it is: `postgres`.\n\n```\nruby ~/worker.rb id=localhost.1 db=crdb\n```\n\n4. **log:** Use this parameter to indicate the process to write the log in the file `./worker.#{id}.log` or not. The default value is `yes`.\n\n```\nruby ~/worker.rb id=localhost.1 log=no\n```\n\n**Code Snippets:**\n\nYou can define a code to be executed right after **worker** connected to the database.\n\nAdd a code like this in your `config.rb` file.\n\n```ruby\n# Pampa Code Snippets\nBlackStack::Pampa.set_snippets({\n  :dispatcher_function =\u003e Proc.new do |l, *args|\n    initialize_scraper()\n  end,\n})\n```\n\nYou can also pass a logger to the snippet.\n\n```ruby\n# Pampa Code Snippets\nBlackStack::Pampa.set_snippets({\n  :dispatcher_function =\u003e Proc.new do |l, *args|\n    l = BlackStack::DummyLogger.new(nil) if l.nil?\n\n    l.logs 'Initializing scraper... '\n    initialize_scraper()\n    l.logf 'done'.green\n  end,\n})\n```\n\n## 9. Selection Snippet\n\nYou can re-write the default function used by the **dispatcher** to choose the records it will assign to the **workers**.\n\n**Example:** You want to dispatch the records in the table `numbers`, but sorted by `value` reversely.\n\n```ruby\n# define the job\nBlackStack::Pampa.add_job({\n  :name =\u003e 'search_odd_numbers',\n  :queue_size =\u003e 5, \n  :max_job_duration_minutes =\u003e 15,  \n  :max_try_times =\u003e 3,\n  :table =\u003e :numbers, # Note, that we are sending a class object here\n  :field_primary_key =\u003e :value,\n  :field_id =\u003e :odd_checking_reservation_id,\n  :field_time =\u003e :odd_checking_reservation_time, \n  :field_times =\u003e :odd_checking_reservation_times,\n  :field_start_time =\u003e :odd_checking_start_time,\n  :field_end_time =\u003e :odd_checking_end_time,\n  :field_success =\u003e :odd_checking_success,\n  :field_error_description =\u003e :odd_checking_error_description,\n  :filter_worker_id =\u003e /\\.1$/, # only worker number 1 will receive tasks of this job.\n  :max_pending_tasks =\u003e 10, \n  :max_assigned_workers =\u003e 5, \n  :processing_function =\u003e Proc.new do |task, l, job, worker, *args|\n    l.logs 'Checking if '+task[:value].to_s.blue+' is odd... '\n    if task[:value] % 2 == 0\n      task[:is_odd] = false\n      l.logf 'No.'.yellow\n    else\n      task[:is_odd] = true\n      l.logf 'Yes.'.green\n    end\n    DB[:numbers].where(:value=\u003etask[:value]).update(:is_odd=\u003etask[:is_odd])\n  end,\n\n  # write a snippet for selecting records to dispatch.\n  :selecting_function =\u003e Proc.new do |n, *args|\n    DB[\"\n      SELECT *\n      FROM numbers\n      WHERE odd_checking_reservation_id IS NULL           -- record not reserved yet\n      AND odd_checking_start_time IS NULL                 -- record is not pending to relaunch\n      AND COALESCE(odd_checking_reservation_times,0) \u003c 3  -- record didn't fail more than 3 times\n      ORDER BY value DESC                                -- I want to order by number reversely\n      LIMIT #{n}                                          -- don't dispatch more than n records at the time\n    \"].all\n  end,\n})\n```\n\n**Example:** You define another **job** to submit the odd numbers to a server.\n\n```ruby\n# define the job\nBlackStack::Pampa.add_job({\n  :name =\u003e 'submit_odd_numbers',\n  :queue_size =\u003e 5, \n  :max_job_duration_minutes =\u003e 15,  \n  :max_try_times =\u003e 3,\n  :table =\u003e :numbers, # Note, that we are sending a class object here\n\n  :field_primary_key =\u003e :value,\n  :field_id =\u003e :submit_odd_reservation_id,\n  :field_time =\u003e :submit_odd_reservation_time, \n  :field_times =\u003e :submit_odd_reservation_times,\n  :field_start_time =\u003e :submit_odd_start_time,\n  :field_end_time =\u003e :submit_odd_end_time,\n  :field_success =\u003e :submit_odd_success,\n  :field_error_description =\u003e :submit_odd_error_description,\n  \n  :filter_worker_id =\u003e /\\.2/,\n  \n  :max_pending_tasks =\u003e 10, \n  :max_assigned_workers =\u003e 5, \n  \n  :processing_function =\u003e Proc.new do |task, l, job, worker, *args|\n    # TODO: Run a post call here, to subit the record.\n  end,\n\n  # write a snippet for selecting records to dispatch.\n  :selecting_function =\u003e Proc.new do |n, *args|\n    DB[\"\n      SELECT *\n      FROM numbers\n      WHERE submit_odd_reservation_id IS NULL             -- record not reserved yet\n      AND submit_odd_start_time IS NULL                   -- record is not pending to relaunch\n      AND COALESCE(submit_odd_reservation_times,0) \u003c 3    -- record didn't fail more than 3 times\n\n      AND odd_checking_end_time IS NOT NULL               -- record that have been checked\n      AND COALESCE(odd_checking_success, FALSE) = TRUE    -- record that have been checked successfully\n      AND COALESCE(id_odd, FALSE) = TRUE                  -- only submit odd values\n\n      ORDER BY odd_checking_end_time DESC                 -- submit records in the order they have been checked\n      LIMIT #{n}                                          -- don't dispatch more than n records at the time\n    \"].all\n  end,\n})\n```\n\n**Other Examples:**\n\n- You may want to deliver emails of active email campaigns only (`active=true`).\n- You may want to process orders in the order they have been created (`order by create_time`).\n- You may want to add a delay of 1 day from the moment a new user signed up and he/she receives an email notification.\n\n## 10. Relaunching Snippet\n\nUse `:relaunching_function` to write your own snippet code that will choose the records you want to relaunch.\n\n**Example:** You may want to check if each number is still odd every 10 minutes, because you are afraid the laws of the universe have suddenly changed\n\n```ruby\n# define the job\nBlackStack::Pampa.add_job({\n  :name =\u003e 'search_odd_numbers',\n  :queue_size =\u003e 5, \n  :max_job_duration_minutes =\u003e 15,  \n  :max_try_times =\u003e 3,\n  :table =\u003e :numbers, # Note, that we are sending a class object here\n  :field_primary_key =\u003e :value,\n  :field_id =\u003e :odd_checking_reservation_id,\n  :field_time =\u003e :odd_checking_reservation_time, \n  :field_times =\u003e :odd_checking_reservation_times,\n  :field_start_time =\u003e :odd_checking_start_time,\n  :field_end_time =\u003e :odd_checking_end_time,\n  :field_success =\u003e :odd_checking_success,\n  :field_error_description =\u003e :odd_checking_error_description,\n  :filter_worker_id =\u003e /\\.1$/, # only worker number 1 will receive tasks of this job.\n  :max_pending_tasks =\u003e 10, \n  :max_assigned_workers =\u003e 5, \n  :processing_function =\u003e Proc.new do |task, l, job, worker, *args|\n    l.logs 'Checking if '+task[:value].to_s.blue+' is odd... '\n    if task[:value] % 2 == 0\n      task[:is_odd] = false\n      l.logf 'No.'.yellow\n    else\n      task[:is_odd] = true\n      l.logf 'Yes.'.green\n    end\n    DB[:numbers].where(:value=\u003etask[:value]).update(:is_odd=\u003etask[:is_odd])\n  end,\n\n  # you want to check if each number is still odd every 10 minutes, \n  # because you are afraid the laws of the universe have suddenly changed.\n  :relaunching_function =\u003e Proc.new do |n, *args|\n    DB[\"\n      SELECT *\n      FROM numbers\n      WHERE odd_checking_end_time IS NOT NULL                                           -- record that have been checked\n      AND odd_checking_end_time \u003c CAST('#{now}' AS TIMESTAMP) - INTERVAL '10 MINUTES'   -- checked 10 minutes ago\n      LIMIT #{n}                                                                        -- don't relaunch more than n records at the time\n    \"].all\n  end,\n})\n```\n\n**Other Examples:**\n\n- Every 5 minutes you want to check if a list of websites are online.\n- You want to trace the CPU usage of a pool of servers.\n- You want to keep mirroring infromation between databases.\n\n## 11. Elastic Workers Assignation\n\nThe **dispatcher** process not only assign **tasks** to **workers**, but it also **assign** and **unassign** **workers** to each **job**, depending on the number of **task** in **queue** for such a **job**.\n\nWhen you define a job, \n\n- use the parameter `:max_pending_tasks` to tell the dispatcher when it should assign more workers for this job. If the number of pending task is higer than `:max_pending_tasks`, the dispatcher will scale the number of assigned workers;\n\n- but also use the parameter `:max_assigned_workers` to prevent your job to monopolize the entire pool of workers and get in the way of other jobs if its number of pending tasks raises so high;\n\nand finally,\n\n- you can use the `:filter_worker_id` to define a regular expession to filter the workers that may be assigned for your job. E.g.: Use `/.*/` to allow any worker to be assigned, or use `/\\.1$/` to allow the first worker in each node to be assigned only.\n\n## 12. Reporting\n\nYou can get the number of \n\n- total tasks,\n- completed task,\n- pending tasks,\nand\n- failed tasks;\n\ncalling the methods shown below.\n\n```ruby\nj = BlackStack::Pampa.jobs.first\n\np j.name\n\np j.total.to_label\np j.completed.to_label\np j.pending.to_label\np j.failed.to_label\n```\n\nIf you wrote snippets for either selecting or relaunching records, you may need to write sneeppets for the reporting methods too, in order to make their numbers congruent.\n\n```ruby\n# define the job\nBlackStack::Pampa.add_job({\n  :name =\u003e 'search_odd_numbers',\n\n  # ...\n\n  :total_function =\u003e Proc.new do |*args|\n    # TODO: return a number here\n  end,\n\n  :completed_function =\u003e Proc.new do |*args|\n    # TODO: return a number here\n  end,\n\n  :pending_function =\u003e Proc.new do |*args|\n    # TODO: return a number here\n  end,\n\n  :completed_function =\u003e Proc.new do |*args|\n    # TODO: return a number here\n  end,\n\n})\n```\n\n## Inspiration\n\n- [https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox](https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox)\n\n## Disclaimer\n\nThe logo has been taken from [here](https://icons8.com/icon/ay4lYdOUt1Vd/geometric-figures).\n\nUse this library at your own risk.\n\n## Versioning\n\nWe use [SemVer](http://semver.org/) for versioning. For the versions available, see the last [ruby gem](https://rubygems.org/gems/simple_command_line_parser). \n\n## Authors\n\n* **Leandro Daniel Sardi** - *Initial work* - [LeandroSardi](https://github.com/leandrosardi)\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.\n\n## Further Work\n\n### Counting Pending Tasks: `:occupied_function`\n\nThe `:occupied_function` function returns an array with the pending **tasks** in queue for a **worker**.\n\nThe default function returns all the **tasks** with `:field_id` equal to the name of the worker, and the `:field_start_time` empty.\n\nYou can setup a custom version of this function.\n\nExample: You may want to sort ....\n\nadditional function to decide how many records are pending for processing\nit should returns an integer\nkeep it nil if you want to run the default function\n\n### Selecting of Workers: `:allowing_function`\n\n_(this feature is pending to develop)_\n\nadditional function to decide if the worker can dispatch or not\nexample: use this function when you want to decide based on the remaining credits of the client\nit should returns true or false\nkeep it nil if you want it returns always true\n\n### Scheduled Tasks\n\n_(this feature is pending to develop)_\n\n### Multi-level Dispatching\n\n_(this feature is pending to develop)_\n\n### Setup Resources for Workers\n\n_(this feature is pending to develop)_\n\n```ruby\n# setup nodes (computers) where to launch\nn = BlackStack::Pampa.add_nodes([{\n    # setup SSH connection parameters\n    :net_remote_ip =\u003e '54.160.137.218',  \n    :ssh_username =\u003e 'ubuntu',\n    :ssh_port =\u003e 22,\n    :ssh_private_key_file =\u003e './plank.pem',\n    # setup max number of worker processes\n    :max_workers =\u003e 2,\n    # setup max memory consumption per worker (MBs)\n    :max_ram =\u003e 512, \n    # setup max CPU usage per worker (%)\n    :max_cpu =\u003e 10,\n    # setup max disk I/O per worker (mbps)\n    :max_disk_io =\u003e 1000,\n    # setup max ethernet I/O per worker (mbps)\n    :max_net_io =\u003e 1000,\n\n}])\n```\n\n## 13. Stand Alone Processes\n\nPampa provides a standard framework for running stand-alone processes too.\n\nUse the function `run_stand_alone` for processing recursivelly, every some seconds.\n\n**Example:**\n\n```ruby\nrequire 'pampa'\n\nDELAY = 10 # seconds\nONCE = false # if false, execute this process every DELAY seconds\n\nBlackStack::Pampa.run_stand_alone({\n    :log_filename =\u003e 'example.log',\n    :delay =\u003e DELAY,\n    :run_once =\u003e ONCE,\n    :function =\u003e Proc.new do |l, *args|\n        begin\n            l.log 'Hello, World!'\n\n            l.log \"1. Executing this process recursively, with a timeframe of #{DELAY} seconds between the starting of each one.\"\n            l.log \"2. If an execution takes more than #{DELAY} seconds, the next execution will start immediatelly.\"\n            l.log \"3. If an error occurs, the process will stop. So be sure to catch all exceptions here.\"\n\n            # your code here\n\n        rescue =\u003e e\n            l.logf \"Error: #{e.to_console.red}\"\n        # CTRL+C will be catched here\n        rescue Interrupt =\u003e e\n            l.logf \"Interrupted\".red\n            exit(0)\n        ensure\n            l.log \"End of the process.\"\n        end\n    end, \n})\n```\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleandrosardi%2Fpampa","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fleandrosardi%2Fpampa","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fleandrosardi%2Fpampa/lists"}