Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/comcast/go-leaderelection

GoLea is a Go library that provides the capability for a set of distributed processes to compete for leadership for a shared resource.
https://github.com/comcast/go-leaderelection

Last synced: 8 months ago
JSON representation

GoLea is a Go library that provides the capability for a set of distributed processes to compete for leadership for a shared resource.

Awesome Lists containing this project

README 

          
# GoLea ![Goalie](https://cloud.githubusercontent.com/assets/6370793/22409043/e0b28596-e640-11e6-9c7d-368409b4d082.png)



GoLea provides the capability for a set of distributed processes to compete for leadership for a shared resource. It is implemented using Zookeeper for the underlying support. For a general description of leader election, see the [wikipedia page describing leader election](https://en.wikipedia.org/wiki/Leader_election).



# Terms



Term


Description



Nominee


Nominee is a potential leader. Each client nominates itself. One will become Leader, the others will become Candidates.




Candidate


See Nominee.




Client


Client is the customer, or client, of the Library (see below). The Client uses the Library to negotiate leadership for a given resource.




Election Node


ElectionNode is the base Zookeeper node that leader candidates will place their nominations for leadership.

For the leader election examples in this package this node is called /election. For an actual application elections may be defined as `/election-type`. Using politics as an example, the `election-type` might be `president`. 
An application that has the concept of a *primary* and *standby* `election-type` might be *primary*. For applications that have multiple components the *Election Node* might have 2 or more Zookeeper nodes. An example might be `/component-type/primary` where multiple application components have the concept of `primary` and `standby`.



Resource


Resource is the target of the leadership request. If leadership is granted the leader is free to manage the resource in whatever way is appropriate to the application without worrying that other clients are concurrently managing that resource. *Resource* is a more general name for *Election Node*.




Follower


Follower is an entity that is not the leader for a Resource, but is actively monitoring the resource in the event the Leader is removed from its leadership position. If this happens, a Follower becomes a candidate for Leader which will be decided via an automatic, asynchronous, process.




Leader


Leader is the entity that has exclusive ownership for a given Resource.




Library


Library is this component - i.e., the Leader Election library.




Package


See Library.




ZK


Zookeeper.




# How to use



This section provides an overview of the various phases of an election. The file `Election_test.go` is a good example of how a client is expected to use the leader election package. The integration tests (see the `integrationtest directory`) may also provide some insight into how to use the package although their intent is primarily to test the package vs. provide examples.



Leader Election (LE) clients will have to import the following packages:



    import (

    	"github.com/go-zookeeper/zk"

    	"github.com/Comcast/go-leaderelection"

    )



Leader election clients must provide a ZK connection when creating an election instance. The rationale behind this is to prevent applications that participate in multiple elections from letting the library create multiple ZK connections behind the scenes. This allows the application to optimize the use of ZK connections.



## Create a *Resource*

A *resource* is represented as a ZK node. During the design and implementation of the library an explicit decision was made to not provide the capability in the library to create election resources. This decision may be revisited in the future.



    // Create the ZK connection

    zkConn, _, err := zk.Connect([]string{zkURL}, heartBeat)

    if err != nil {

        log.Printf("Error in zk.Connect (%s): %v",

            zkURL, err)

        return nil, err

    }



    // Create the election node in ZooKeeper

    _, err = zkConn.Create(electionNode,

        []byte(""), 0, zk.WorldACL(zk.PermAll))

    if err != nil {

        log.Printf("Error creating the election node (%s): %v",

            electionNode, err)

        return nil, err

    }



    candidate, err := leaderelection.NewElection(zkConn, electionNode)



## Delete an Election



    err := leaderelection.DeleteElection(zkConn, electionNode)

or



    leaderelection.EndElection()



## Request Leadership for an Election



Leadership election is inherently an asynchronous process. For candidates that win an election the inital response is fairly rapid. Even so, leaders can still receive errors from the election that will impact their leadership of a resource. The most common example is when the underlying election instance gets disconnected from Zookeeper. In the event of errors, leaders expected to resign their leadership. This is likely a futile process if the ZooKeeper connection is lost, but it's good practice to do this for any error.



Followers of an election can wait an indeterminate amount of time to become the leader of an election. If a follower becomes a leader they will be notified. In some cases the election may end before a follower can become a leader. Like leaders, followers can receive errors from the election. Like leaders, they should resign from the election in this case.



Finally, an election can be unilaterally ended by any actor in the application. As with errors, this results in a status notification to all candidates, including the leader. Candidates are expected to resign from the election in this case as well.



Given that leadership election is an asynchronous process clients should start the election as a goroutine as shown below. All events pertaining to the status of an election are communicated via channels. So the typical pattern is for a client to monitor the election status in a `for/select` loop as shown below.



    // Each candidation should register interest in participating for an election

    candidate, err := NewElection(zkConn, "/election", "myhostname")

    ...

    // Each candidate should begin their participation in the

    // by starting a goroutine.

    go candidate.ElectLeader()

    ...



## Monitor election and candidate's role in the election

    var status Status

    var ok bool



    for {

        select {

        case status, ok = <-leaderElector.Status():

            if !ok {

                fmt.Println("\t\t\tChannel closed, election is terminated!!!")

                respCh <- ElectionResponse{false, status.CandidateID}

                leaderElector.Resign()

                wg.Done()

                return

            }

            if status.Err != nil {

                fmt.Println("Received election status error <<", status.Err, ">> for candidate <", leaderElector.candidateID, ">.")

                leaderElector.Resign()

                wg.Done()

                return

            }



            fmt.Println("Candidate received status message: <", status, ">.")

            if status.Role == Leader {

                doLeaderStuff(leaderElector, status, respCh, connFailCh, waitFor)

                wg.Done()

                return

            }

        case <-connFailCh:

            fmt.Println("\t\t\tZK connection failed for candidate <", status.CandidateID, ">, exiting")

            respCh <- ElectionResponse{false, status.CandidateID}

            leaderElector.Resign()

            wg.Done()

            return

        case <-time.After(time.Second * 100):

            fmt.Println("\t\t\tERROR!!! Timer expired, stop waiting to become leader for", status.CandidateID)

            leaderElector.Resign()

            respCh <- ElectionResponse{false, status.CandidateID}

            wg.Done()

            return

    }

}



## Resign from an Election

    candidate.Resign()

## Leadership change

See the **Monitor election and candidate's role in the election** above.

## Query status

Candidates are always notified when an election's status changes. It is up to the client to cache this status if they need to reference it between status changes.



# Prerequisites



1. `go-leaderelection` uses `github.com/go-zookeeper/zk`.



1. All tests require the availability of a Zookeeper installation. `zkServer.sh` must be in the path. `Election_test.go` requires that Zookeeper be running. The integration tests control Zookeeper so Zookeeper should not be running when executing the integration tests.



Testing the package has additional prerequisites:



1. The integration tests leverage `github.com/Comcast/goint`. This package must be installed prior to executing integration tests.