{"id":37022343,"url":"https://github.com/verizon/ark","last_synced_at":"2026-01-14T02:40:03.407Z","repository":{"id":57736412,"uuid":"60277388","full_name":"Verizon/ark","owner":"Verizon","description":"Thread safe implementation of org.apache.mesos.Scheduler based on scalaz streams","archived":true,"fork":false,"pushed_at":"2016-11-19T07:23:14.000Z","size":47,"stargazers_count":25,"open_issues_count":0,"forks_count":5,"subscribers_count":9,"default_branch":"master","last_synced_at":"2023-07-04T22:38:16.966Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Verizon.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2016-06-02T15:54:23.000Z","updated_at":"2023-07-04T22:38:16.967Z","dependencies_parsed_at":"2022-08-23T18:30:06.026Z","dependency_job_id":null,"html_url":"https://github.com/Verizon/ark","commit_stats":null,"previous_names":[],"tags_count":1,"template":null,"template_full_name":null,"purl":"pkg:github/Verizon/ark","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Verizon%2Fark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Verizon%2Fark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Verizon%2Fark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Verizon%2Fark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Verizon","download_url":"https://codeload.github.com/Verizon/ark/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Verizon%2Fark/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28408711,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-14T01:52:23.358Z","status":"online","status_checked_at":"2026-01-14T02:00:06.678Z","response_time":107,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-14T02:40:02.790Z","updated_at":"2026-01-14T02:40:03.392Z","avatar_url":"https://github.com/Verizon.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Ark\n\n![image](docs/img/logo.png)\n\n[![Build Status](https://travis-ci.org/Verizon/ark.svg?branch=master)](https://travis-ci.org/Verizon/ark)\n[![Maven Central](https://maven-badges.herokuapp.com/maven-central/io.verizon.ark/core_2.10/badge.svg)](https://maven-badges.herokuapp.com/maven-central/io.verizon.ark/core_2.10)\n[![codecov](https://codecov.io/gh/Verizon/ark/branch/master/graph/badge.svg)](https://codecov.io/gh/Verizon/ark)\n\nThis library provides a functional scala implementation of `org.apache.mesos.Scheduler` interface provided by [Mesos java API](http://mesos.apache.org/api/latest/java/).\n\nThe goal of this library is to ease development of mesos schedulers by providing out-of-the-box implementations of common operational requirements of a framework, allowing developers to focus on domain logic implementation of task state transitions.\n\nFeatures:\n\n   * Pure functional implementation of mesos scheduler tasks state.\n   * Scalaz stream to queue all messages sent to the framework (from mesos master or custom user defined messages) to be processed one at a time making it completely thread safe.\n   * Recurring reconcialiation based on [Mesos Reconciliation Algorithm](http://mesos.apache.org/documentation/latest/reconciliation/).\n   * *TODO:* Re-registration on mesos master failures.\n   * *TODO:* High-Availability mode and leader election.\n\nFrom the current state of the project there is a clear path to implement missing features above by enhancing `oncue.mesos.Scheduler.processMessage` function.\n\n\n## Messages\n\nThe core of this Mesos Scheduler implementation is handled by a scalaz async message queue. When Mesos calls any of the functions provided by the `Scheduler` interface, the scheduler creates one or many `oncue.mesos.MesosMessage` and enqueues them in the scalaz stream.\n\n```scala\nsealed trait MesosMessage { def driver: org.apache.mesos.SchedulerDriver }\n```\n\nThe main scalaz stream is created inside `oncue.mesos.Scheduler` to handle calls from Mesos to the Scheduler interface. Users can provide any number of `scalaz.stream.Process[scalaz.concurrent.Task, CustomMessage]` when initializing the `Scheduler`. These custom streams get merged into the internal scalaz stream. This way the user can trigger any `CustomMesssage` to the scheduler which is handled by the same `processMessage` function that handles messages from Mesos.\n\n```scala\ntrait CustomMessage extends MesosMessage\n```\n\n## Reconciliation\n\nMesos has very good documentation on how to implement the [Reconciliation Algorithm](http://mesos.apache.org/documentation/latest/reconciliation/), since most frameworks need to perform reconciliation this was the first feature to address in a common Mesos scheduler library.\n\nReconciliation is triggered by sending a `ReconcileMessage` to the stream:\n\n```scala\ncase class ReconcileMessage(override val driver: SchedulerDriver) extends CustomMessage\n```\n\n`oncue.mesos.Scheduler` companion object provides a convenient function to initialize a timed reconciliation stream:\n\n```scala\ndef reconcileProcess(driver: SchedulerDriver, reconcileInterval: FiniteDuration): Process[Task, ReconcileMessage] = {\n  time.awakeEvery(reconcileInterval)(defaultExecutor, timeOutScheduler)\n    .map(_ =\u003e ReconcileMessage(driver))\n}\n```\n\nThe user can create a reconcile process by calling the function above and passing it to the scheduler `init` function, this will trigger reconcialiation every `reconcileInterval` and all offers will be declined until reconciliation is over.\n\n```scala\nval reconciliationInterval = 1 hour\nval customStreams = Seq( Scheduler.reconcileProcess(driver, reconciliationInterval) )\nscheduler.init(state, driver, customStreams).run\n```\n\n*TODO:* The wait time to reconcile all tasks is currently fixed, Mesos recommends to use truncated exponential back off to \"avoid a snowball effect in the case of the driver or master being backed up\".\n\n\n## Usage\n\nA full implementation of Mesos Scheduler would be required to implement `oncue.mesos.SchedulerState` and `oncue.mesos.SchedulerStateManager` traits and run the scheduler like this:\n\n```scala\n  // implement state and state manager\n  case class MyState( ... ) extends SchedulerState\n  class MyStateManager extends SchedulerState[MyState] { ... }\n\n  // initialize state and state manager\n  val initialState = MyState( ... )\n  val stateManager = new MyStateManager( ... )\n\n  // define framework info\n  val frameworkInfo = Protos.FrameworkInfo.newBuilder\n    .setName(\"my-framework\")\n    .setOtherFrameworkattributes( ... )\n    .build\n\n  // initialize scheduler and mesos driver\n  val scheduler = new oncue.mesos.Scheduler(stateManager)\n  val driver = new org.apache.mesos.MesosSchedulerDriver(scheduler, frameworkInfo, mesosMaster)\n\n  // shutdown scheduler on exit\n  sys addShutdownHook {\n    scheduler.shutdown(driver)\n  }\n\n  // Seq[Process[Task,CustomMessage]] pass custom state mutation messages\n  // Scheduler.reconcileProcess triggers reconciliation every \"reconciliationInterval\"\n  val reconciliationInterval = 1 hour\n  val customStreams = Seq(Scheduler.reconcileProcess(driver, reconciliationInterval))\n\n  // run scheduler (blocking)\n  scheduler.init(initialState, driver, customStreams).run\n```\n\n### Example\n\nThe provided example implementation creates a scheduler that triggers the provided task on every slave in the cluster.\nThis example also uses [http4s](http://http4s.org/) to set up REST endpoints to query current scheduler state by\nsending custom messages to the queue. User can query scheduler info and add or remove slaves from a blacklist.\n\nRunning example module on a local mesos cluster with 2 slaves using docker-machine on mac (see\nhttps://github.com/mesosphere/docker-containers/tree/master/mesos):\n\n1. Run ZK:\n   \n   ```bash\n   docker run -d --net=host netflixoss/exhibitor:1.5.2\n   ```\n   \n1. Run master:\n   \n   ```bash\n    docker run -d --net=host \\\n      -e LIBPROCESS_IP=$(docker-machine ip) \\\n      -e HOSTNAME=$(docker-machine ip) \\\n      -e MESOS_PORT=5050 \\\n      -e MESOS_ZK=zk://127.0.0.1:2181/mesos \\\n      -e MESOS_QUORUM=1 \\\n      -e MESOS_REGISTRY=in_memory \\\n      -e MESOS_LOG_DIR=/var/log/mesos \\\n      -e MESOS_WORK_DIR=/var/tmp/mesos \\\n      -v \"$(pwd)/log/mesos:/var/log/mesos\" \\\n      -v \"$(pwd)/tmp/mesos:/var/tmp/mesos\" \\\n      mesosphere/mesos-master:0.25.0-0.2.70.ubuntu1404\n   ```\n   \n1. Run slaves, notice `MESOS_PORT` and mount points change for `/var/log/mesos` and `/var/tmp/mesos`:\n   \n   ```bash\n   docker run -d --net=host --privileged \\\n      -e LIBPROCESS_IP=$(docker-machine ip) \\\n      -e HOSTNAME=$(docker-machine ip)  \\\n      -e MESOS_PORT=5051 \\\n      -e MESOS_MASTER=zk://127.0.0.1:2181/mesos \\\n      -e MESOS_SWITCH_USER=0 \\\n      -e MESOS_CONTAINERIZERS=docker,mesos \\\n      -e MESOS_LOG_DIR=/var/log/mesos \\\n      -e MESOS_WORK_DIR=/var/tmp/mesos \\\n      -v \"$(pwd)/log/mesos1:/var/log/mesos\" \\\n      -v \"$(pwd)/tmp/mesos1:/var/tmp/mesos\" \\\n      -v /var/run/docker.sock:/var/run/docker.sock \\\n      -v /cgroup:/cgroup \\\n      -v /sys:/sys \\\n      -v /usr/local/bin/docker:/usr/local/bin/docker \\\n      mesosphere/mesos-slave:0.25.0-0.2.70.ubuntu1404\n\n   docker run -d --net=host --privileged \\\n      -e LIBPROCESS_IP=$(docker-machine ip) \\\n      -e HOSTNAME=$(docker-machine ip)  \\\n      -e MESOS_PORT=5052 \\\n      -e MESOS_MASTER=zk://127.0.0.1:2181/mesos \\\n      -e MESOS_SWITCH_USER=0 \\\n      -e MESOS_CONTAINERIZERS=docker,mesos \\\n      -e MESOS_LOG_DIR=/var/log/mesos \\\n      -e MESOS_WORK_DIR=/var/tmp/mesos \\\n      -v \"$(pwd)/log/mesos2:/var/log/mesos\" \\\n      -v \"$(pwd)/tmp/mesos2:/var/tmp/mesos\" \\\n      -v /var/run/docker.sock:/var/run/docker.sock \\\n      -v /cgroup:/cgroup \\\n      -v /sys:/sys \\\n      -v /usr/local/bin/docker:/usr/local/bin/docker \\\n      mesosphere/mesos-slave:0.25.0-0.2.70.ubuntu1404\n   ```\n   \n1. Build scheduler assembly jar\n\n   ```bash\n   sbt \"project example\" assembly\n   ```\n1. Build scheduler container from example/Dockerfile:\n   \n   ```bash\n   docker build -t mysched example/\n   ```\n   \n1. Run scheduler container interactively:\n   \n   ```bash\n   docker run --rm --net=host -it \\\n      -e LIBPROCESS_IP=$(docker-machine ip) \\\n      -v $(pwd)/example/target/scala-2.10:/opt/app \\\n      mysched\n   ```\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fverizon%2Fark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fverizon%2Fark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fverizon%2Fark/lists"}