{"id":20171387,"url":"https://github.com/brutusin/wava","last_synced_at":"2025-04-10T02:42:53.125Z","repository":{"id":57739025,"uuid":"69558487","full_name":"brutusin/wava","owner":"brutusin","description":"Linux batch scheduler supporting job memory usage limits","archived":false,"fork":false,"pushed_at":"2017-06-02T11:03:34.000Z","size":798,"stargazers_count":11,"open_issues_count":7,"forks_count":3,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-03-24T04:13:05.538Z","etag":null,"topics":["batch-processing","cgroups","job-queue","job-scheduler","linux","scheduler"],"latest_commit_sha":null,"homepage":"","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/brutusin.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-09-29T10:48:30.000Z","updated_at":"2022-08-30T15:25:15.000Z","dependencies_parsed_at":"2022-08-24T17:40:19.583Z","dependency_job_id":null,"html_url":"https://github.com/brutusin/wava","commit_stats":null,"previous_names":[],"tags_count":57,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brutusin%2Fwava","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brutusin%2Fwava/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brutusin%2Fwava/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/brutusin%2Fwava/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/brutusin","download_url":"https://codeload.github.com/brutusin/wava/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248145100,"owners_count":21055053,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch-processing","cgroups","job-queue","job-scheduler","linux","scheduler"],"created_at":"2024-11-14T01:24:33.735Z","updated_at":"2025-04-10T02:42:53.103Z","avatar_url":"https://github.com/brutusin.png","language":"Java","readme":"## org.brutusin:wava [![Build Status](https://api.travis-ci.org/brutusin/wava.svg?branch=master)](https://travis-ci.org/brutusin/wava) [![Maven Central Latest Version](https://maven-badges.herokuapp.com/maven-central/org.brutusin/wava-root/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.brutusin/wava-root/)\r\n\r\n`wava` is a linux command line tool wich allows for multiple users to securely run batch processes, scheduded in a timely manner under contraints of allocated (resident) memory capacities.\r\n\r\n![wava menu](https://github.com/brutusin/wava/raw/master/img/wava-menu.gif)\r\n\r\n**Table of contents**\r\n- [Top](#orgbrutusinwava--)\r\n- [Overview](#overview)\r\n- [Features](#features)\r\n  * [Capacity guarantees](#capacity-guarantees)\r\n  * [Security](#security)\r\n  * [Resource-based scheduling](#resource-based-scheduling)\r\n  * [Priority-based scheduling](#priority-based-scheduling)\r\n  * [Statistics](#statistics)\r\n- [Architecture](#architecture)\r\n- [Priority and groups](#priority-and-groups)\r\n  * [Job order](#job-order)\r\n  * [Niceness](#niceness)\r\n- [Events](#events)\r\n- [Job hierarchy](#job-hierarchy)\r\n  * [Blocked state](#blocked-state)\r\n  * [Deadlock prevention](#deadlock-prevention)\r\n- [Requirements](#requirements)\r\n- [Installation](#installation)\r\n- [Configuration](#configuration)\r\n- [Running](#running)\r\n- [Support bugs and requests](#support-bugs-and-requests)\r\n- [Authors](#authors)\r\n- [License](#license)\r\n\r\n\r\n## Overview\r\n\r\n`wava` scheduler is designed to run batch processes in a single Linux machine in an operator-friendly manner while imposing a memory limit across the overall job set (strict limits) and guaranteeing a minimum of available memory per job (soft limits).\r\n\r\nTraditionally fixed-sized queues were used for enqueueing jobs, but when used over a heterogeneus (in terms of memory demands) set of jobs, they show different weaknesses: inefficient resource utilization, system performance degradation, and job resource competition.\r\n\r\n`wava` scheduler is designed to overcome this, that is, without losing easy of use, providing guarantees for both system stability and job resource allocation and, on the other side, offering better resource utilization rates. \r\n\r\n## Features\r\n### Capacity guarantees\r\nJobs are submitted with a minimum memory claim (job size), then enqueued, and finally executed in a sandboxed environment (implemented via [cgroups](https://en.wikipedia.org/wiki/Cgroups)) with the scheduler capacity as hard limit and the jobs size as soft limit.\r\n\r\nAt runtime, the job process tree is allowed to allocate an amount of resident memory up to the scheduler-capacity (if no more jobs are scheduled), and in case of memory pressure forced to swap out.\r\n\r\nThe scheduler guareantees that each job process tree has at its disposition at least the amount of minimum memory claimed at job submission.\r\n\r\n### Security\r\nJob processes are run by the same machine user that submitted the job, so the scheduler can not be used to [escale the running privileges](https://en.wikipedia.org/wiki/Privilege_escalation) of a user.\r\n\r\n### Resource-based scheduling\r\nScheduling is based on memory. The scheduler is configured to have a certain capacity and each job has an associated minimum memory size. \r\n\r\nThe main **scheduling constraint** is the following: the sum of the running jobs minimum memory size cannot exceed the scheduler capacity. \r\n\r\n### Priority-based scheduling\r\nThis feature allows jobs to be submitted and scheduled with different priorities. \r\n\r\nAll jobs belong (implicity or explicity) to a [priority group](#priority-and-groups) that determines their global ordering, used for positioning in the queue and assigning a process niceness when running.\r\n\r\n### Statistics\r\nBoth global and per job statistics can be recorded.\r\nStatistics sensitivity can be tuned with the `statsCpuStep`,`statsRssStep`,`statsSwapStep` and `statsIOStep` configuration parameters.\r\n\r\n```\r\n#start          end             running queued  cpu(%)  rss(B)  swap(B) io(B/s)\r\n1494930723326   1494930723326   0       0       0.0     0       0       0\r\n1494930723326   1494930731348   1       0       0.0     0       0       0\r\n1494930731348   1494930733355   1       0       92.7    696320  0       0\r\n...\r\n```\r\n\r\n#### Global statistics\r\nTo enable global statistics set the `\"logStat\"=true\"` in the configuration file.\r\n\r\n#### Job statistics\r\nAt submit time (`wava -r`) specify an additional parameter `-s` with the folder to store the stats files for this only execution.\r\n\r\n\r\n## Architecture\r\n\r\nThe scheduler runs as a centralized process (`wava -s`) and all interaction with it is performed by separated client processes started when invoking the other command options.  \r\n\r\nIn particular job submissions are performed by separate peer processes (`wava -r`) that serve as lightweight placeholders of the real jobs executed by the scheduler. Peer and job processes have their lifecycle bound to each other. If one dies the other dies too.\r\n\r\n![wava menu](https://github.com/brutusin/wava/raw/master/img/process-diagram.png)\r\n\r\nThe scheduler pipes standard io-streams between the job processes and their respective peer processes. Additionally, it pipes scheduler [events](#events) to the peer `stderr` unless an event file has been specified in submission (`wava -r -e \u003cfile\u003e`).\r\n\r\n![wava example](https://github.com/brutusin/wava/raw/master/img/wava-example2.gif)\r\n*Running an example command requiring a minimum of 100MB. Observe also that this command runs untils user 'nacho' cancels it, returning a non-zero return code*\r\n\r\n## Priority and groups\r\n\r\nAll submitted jobs belong to a priority group that determines their global ordering, used for positioning in the queue and assigning a process niceness when running.\r\n\r\nBesides `priority` groups also have a `timeToIdleSeconds` property. This is the time elapsed between the last job finishes and the group is removed. If this value is set to `-1`, the group is eternal.\r\n\r\nJobs that do not specify a group at submit time are assigned to the `default` group (`priority=0`, `timeToIdleSeconds=-1`).\r\n\r\nJobs that specify a non-existing group create at submit-time a *dynamic* group (`priority=0`, `timeToIdleSeconds` specified at [configuration](#configuration-description)).\r\n\r\n![wava group listing](https://github.com/brutusin/wava/raw/master/img/wava-groups.gif)\r\n*Sample output of command `wava -g -l` for querying groups*\r\n\r\n### Job order\r\n\r\nJobs are ordered by the following rules:\r\n- First by group priority (lower value means higher priority)\r\n- Then by group id (incremental). In case of same priority, jobs of the oldest group go first.\r\n- Finally, by job id (incremental). For jobs inside the same group, FIFO ordering.\r\n\r\n![wava job listing](https://github.com/brutusin/wava/raw/master/img/wava-jobs-2.0.gif)\r\n*Sample output of command `wava -j` for querying jobs (white: running, yellow: queued). This scheduler instance has a capacity of 500 MB*\r\n\r\n### Niceness\r\nThe scheduler sets the niceness of the job processes according to their global ordering within the working niceness range. The concrete strategy is determined by the [`NicenessHandler`](wava-core/src/main/java/org/brutusin/wava/core/plug/NicenessHandler.java) implementation used (set in [configuration](#configuration-description)). \r\n\r\n## Events\r\nBesides `stderr` and `stdout`, the scheduler process maintains a dedicated channel (named pipe) for communicating events to client processes. These events are serialized in the form:\r\n```\r\n${time-millis}:${event-type}[:${event-value}]\r\n```\r\nFor peer processes these events are output to `stderr` after being formatted as `[wava] [date] [${event-type}:${event-value}]` unless a file is specified (`wava -r -e \u003cfile\u003e`) for redirecting them.\r\n\r\nEvent type ([`Events.java`](wava-client/src/main/java/org/brutusin/wava/io/Event.java)) | Valued | Description\r\n------------------   | --- | -----\r\n`id`                 | yes | Id assigned to the job.\r\n`queued`             | yes | Position in the queue, if the job is queued.\r\n`priority`           | yes | Piority of the job, given by its group. \r\n`running`            | yes | Root pId of the job process when started.  \r\n`niceness`           | yes | Niceness set to the job process.   \r\n`cancelled`          | yes | User cancelling the job. \r\n`ping`               | no  | Send periodically to detect stale peers.\r\n`exceed_tree        `| yes | Memory claim exceeds capacity\r\n`shutdown`           | yes | Scheduler is being stopped \r\n`maxrss`             | yes | Max RSS allocated to the process tree\r\n`maxswap`            | yes | Max swap allocated to the process tree\r\n`error`              | yes | To send information about an error.\r\n`retcode`            | yes | Return code for the client process to use.\r\n`starvation_relaunch`| yes | Indicates that the job has been reenqueued due to a [starvation scenario](#deadlock-prevention) (applies for idempotent jobs)\r\n`starvation_stop`    | yes | Indicates that the job has been stopped due to a starvation scenario (applies for non-idempotent jobs)\r\n\r\n## Job hierarchy\r\nRunning jobs can submit more jobs, thus a job hierarchy is established. This potentially can lead to a deadlock scenario, when all parent (running) jobs are waiting for a queued child job to finish.\r\n\r\n### Blocked state\r\n All parent jobs with no children running are considered blocked, ie waiting for their queued children to finish.\r\n\r\n### Deadlock prevention\r\nIn order to avoid deadlock, and prevent from starvation (having too much jobs blocked by a waiting children), the scheduler follows a series of rules that may force a running blocking job to be re-enqueued (if submitted as 'idempotent') or even stoped.\r\n\r\nOn each scheduler main-loop iteration:\r\n\r\n1. The scheduler choses a candidate job to be preempted, based on its idempotency (idempotent first) and priority (low priority first).\r\n2. In case that the ratio of the sum of the sizes of the blocked jobs to the scheduler capacity exceeds a configurable value, the scenario is considered as starving, and the scheduler preempts the candidate job to make room for a potentially blocking job to run.\r\n\r\n## Requirements\r\n`$JAVA_HOME` environment variable set pointing to a JRE 8+\r\n\r\n## Installation\r\n### 1. Create the `WAVA_HOME` environment variable pointing to the desired installation folder:\r\n```sh\r\nexport WAVA_HOME=/opt/wava\r\n```\r\nmake this variable persistent adding the previous line to the file: `~root/.bashrc`\r\n\r\n\u003e Multiple scheduler instances are supported in a single machine. Just change the current `WAVA_HOME`value to point to one or another\r\n\r\n\r\n### 2. Execute the installation script:\r\n```sh\r\nsudo -E bash -c \"$(curl -L https://raw.githubusercontent.com/brutusin/wava/master/wava-core/src/main/scripts/wava-update)\"\r\n```\r\n\r\n\r\n### 3. Service registration (systemd):\r\n\r\n1. Create service file `/etc/systemd/system/wava.service` with the following contents:\r\n\r\n```\r\n[Unit]\r\n#@author Ignacio del Valle Alles\r\nDescription=WAVA scheduler\r\n\r\n[Service]\r\nType=forking\r\nExecStart=/opt/wava/bin/wava -s \u0026\r\nExecStop=/opt/wava/bin/wava -x\r\n# set delegate yes so that systemd does not reset the cgroups\r\nDelegate=yes\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n```\r\n\r\n2. Reload `systemd` configuration:\r\n```\r\nsudo systemctl daemon-reload\r\n```\r\n\r\n3. Enable for running at startup:\r\n```\r\nsudo systemctl enable wava.service\r\n```\r\n\r\n4. Init service:\r\n```\r\nsudo systemctl start wava.service\r\n```\r\n\r\n#### systemd known issues:\r\nIn version 219 it is neccessary to set `JoinControllers=` at `/etc/systemd/system.conf` to have different controllers for `cpuacct` and `cpu` cgroup subsystems, otherwise the cgroup of the jobs cannot be changed at runtime. Maybe this configuration change requires to rebuild the initial ramdisk image to take effect.\r\n\r\n## Configuration\r\nConfiguration is set in file: `$WAVA_HOME/cfg/wava.json`. Environment variables can be used in this file.\r\n### Default configuration\r\n\r\n```javascript\r\n\r\n  \"tempFolder\" : \"/dev/shm\",\r\n  \"uICfg\" : {\r\n    \"ansiColors\" : true,\r\n    \"sIMemoryUnits\" : true\r\n  },\r\n  \"schedulerCfg\" : {\r\n    \"nicenessHandlerClassName\" : \"org.brutusin.wava.core.plug.impl.niceness.HomogeneusSpreadNicenessHandler\",\r\n    \"cgroupRootPath\" : \"/sys/fs/cgroup\",\r\n    \"refreshLoopSleepMillisecs\" : 1000,\r\n    \"pingMillisecs\" : 1000,\r\n    \"schedulerCapacity\" : \"$DEFAULT_CAPACITY\",\r\n    \"maxSwap\" : \"$DEFAULT_SWAP\",\r\n    \"maxJobSize\" : \"$DEFAULT_CAPACITY\",\r\n    \"outOfMemoryKillerEnabled\" : false,\r\n    \"maxBlockedRssStarvationRatio\" : 0.5,\r\n    \"logFolder\" : \"/tmp/wava\",\r\n    \"loggingLevel\" : \"FINE\",\r\n    \"maxLogSize\" : \"100MB\",\r\n    \"maxStatsLogSize\" : \"100MB\",\r\n    \"statsCpuStep\" : 15,\r\n    \"statsRssStep\" : \"50MB\",\r\n    \"statsSwapStep\" : \"50MB\",\r\n    \"statsIOStep\" : \"50MB\",\r\n    \"logStats\" : true\r\n  },\r\n  \"processCfg\" : {\r\n    \"nicenessRange\" : [ 1, 19 ],\r\n    \"cpuAfinity\" : \"$DEFAULT_CPU_AFINITY\"\r\n  },\r\n  \"groupCfg\" : {\r\n    \"dynamicGroupIdleSeconds\" : 10,\r\n    \"predefinedGroups\" : [ {\r\n      \"name\" : \"high\",\r\n      \"priority\" : -10,\r\n      \"timeToIdleSeconds\" : -1\r\n    }, {\r\n      \"name\" : \"low\",\r\n      \"priority\" : 10,\r\n      \"timeToIdleSeconds\" : -1\r\n    } ]\r\n  }\r\n}\r\n\r\n```\r\n### Configuration description\r\nProperty                                    | Description\r\n--------                                    | -----------\r\n`uICfg.ansiColors`                          | Use ANSI escape code sequences to highlight UI.\r\n`uICfg.sIMemoryUnits`                       | Use units from the International System for output memory values. `true`: kB based, `false`:[KiB](https://en.wikipedia.org/wiki/Kibibyte) based\r\n`schedulerCfg.nicenessHandlerClassName`     | FQN of the [`NicenessHandler`](wava-core/src/main/java/org/brutusin/wava/core/plug/NicenessHandler.java) implementation (see [`impl`](wava-core/src/main/java/org/brutusin/wava/core/plug/impl/niceness) package) to use.\r\n`schedulerCfg.memoryCgroupBasePath`         | Root path to the parent memory cgroup\r\n`schedulerCfg.refreshLoopSleepMillisecs`    | Sleeping time for the main looping thread.\r\n`schedulerCfg.pingMillisecs`                | Time interval between ping events to peer processes.\r\n`schedulerCfg.schedulerCapacity`            | Scheduler capacity. Maximum amount of physical memory permitted for all jobs. By default is 3/4 of total memory. Different memory units can be used, for example `4 GB`\r\n`schedulerCfg.maxSwap`                      | Maximum swap size to be used by all wava jobs. By default equals to the total amount of swap available in the system\r\n`schedulerCfg.maxJobSize`                   | Maximum value for a job memory claim. By default equal to the scheduler capacity\r\n`schedulerCfg.outOfMemoryKillerEnabled`     | Enable/disable the Out Of Memory Killer, triggered when a job is forced to page out and there is no enough swap memory available. If disabled the job is stopped until enough memory is available.\r\n`schedulerCfg.maxBlockedRssStarvationRatio` | Maximum ratio between the sum of memory claims of the blocked jobs divided by the scheduler capacity. If exceeded the [starvation prevention mechanism](#deadlock-prevention) is triggered.\r\n`schedulerCfg.logFolder`                    | Folder to store logs and global stats (if enabled).\r\n`schedulerCfg.loggingLevel`                 | Logging level (According to the [Java logging levels](https://docs.oracle.com/javase/7/docs/api/java/util/logging/Level.html))\r\n`schedulerCfg.maxLogSize`                   | Maximum size allowed overall logging files\r\n`schedulerCfg.maxStatsLogSize`              | Maximum size allowed overall stats files for global stats, and per job stats if enabled\r\n`schedulerCfg.statsCpuStep`                 | Stats cpu percentage precission\r\n`schedulerCfg.statsRssStep`                 | Stats rss memory precission\r\n`schedulerCfg.statsSwapStep`                | Stats swap memory precission\r\n`schedulerCfg.statsIOStep`                  | Stats io bandwidth precission (per second)\r\n`schedulerCfg.logStats`                     | `true` to enable global stats logging\r\n`processCfg.nicenessRange`                  | Minimum (most favorable) and maximum (less favorable) niceness to be assigned to a job process tree\r\n`processCfg.cpuAfinity`                     | CPU affinity to be set to the job processes. In a format supported by the `-c` parameter of [taskset](http://linuxcommand.org/man_pages/taskset1.html).\r\n`groupCfg.dynamicGroupIdleSeconds`          | Idle time for [dynamic groups](#priority-and-groups) in seconds.\r\n`groupCfg.predefinedGroups`                 | Set of groups to be available since startup.\r\n\r\n## Running\r\n\r\n### Starting scheduler (`wava -s`)\r\n```\r\n\u003e wava -s\u0026\r\nLogging to /tmp/wava/logs ...\r\n```\r\n\r\n### Submit a job  (`wava -r`)\r\n```\r\n\u003e wava -r -m 100MB -s /tmp/wava/my-job-stats bash -c \"while true; do date; done\" \u0026\u003e /dev/null \u0026\r\n```\r\n\r\n### List jobs  (`wava -j`)\r\n\r\n```\r\n\u003e wava -j\r\nJobs: 1 running; 0 bloqued; 0 queued\r\n  Available memory: 23.8 GB / 24.7 GB\r\n\r\n JOB INFO                            100.0 MB   PROCESS TREE STATS                      696.3 kB       0 B        0 B       92.7  COMMAND\r\n  JOB ID   PARENT GROUP    USER        JOB_RSS NICE    MAX_RSS   MAX_SWAP    MAX_IO         RSS       SWAP        IO        CPU%\r\n       1          default  root      100.0 MB     1  696.3 kB       0 B        0 B/s    696.3 kB       0 B        0 B/s     92.7 [bash, -c, while true; do date; done]\r\n\r\n```\r\n### Cancel job  (`wava -c`)\r\n```\r\n\u003e wava -c 1\r\nRunning job sucessfully cancelled\r\n```\r\n\r\n### Stop scheduler (`wava -x`)\r\n```\r\n\u003e wava -x\r\nStopping scheduler process ...\r\n```\r\n\r\n### Other commands\r\nExecute the `wava` command for detailed help\r\n```\r\n\u003e wava\r\n___________________________________________\r\n__  __ _____ __ __ _____\r\n/   /  /  _  /  |  /  _  \\\r\n|  /\\  |  _  \\  |  |  _  |\r\n\\__/\\__\\__|__/\\___/\\__|__/\r\n\r\n[W]hen [AVA]ilable scheduler 2.3.1-SNAPSHOT\r\n___________________________________________\r\nusage: wava [option]\r\n    -a,--about      information about the program\r\n    -c,--cancel     cancel a running or enqueued job\r\n    -g,--group      group management commands\r\n    -h,--help       print this message\r\n    -j,--jobs       view jobs\r\n    -r,--run        enqueue a job to be executed when enough physical memory is available\r\n    -s,--start      start core scheduler process\r\n    -t,--status     return core process status\r\n    -u,--update     update to lastest version\r\n    -v,--version    show wava version\r\n    -x,--exit       stop core process, terminating all jobs\r\n\r\n```\r\n\r\n## Support bugs and requests\r\nhttps://github.com/brutusin/wava/issues\r\n\r\n## Authors\r\n\r\n- Ignacio del Valle Alles (\u003chttps://github.com/idelvall/\u003e)\r\n\r\nContributions are always welcome and greatly appreciated!\r\n\r\n## License\r\nApache License, Version 2.0\r\n","funding_links":[],"categories":[],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrutusin%2Fwava","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbrutusin%2Fwava","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbrutusin%2Fwava/lists"}