Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/emqx/ekka

Autocluster and Autoheal for EMQX Broker
https://github.com/emqx/ekka

autocluster autoheal clustering erlang-library etcd kubernetes

Last synced: 3 months ago
JSON representation

Autocluster and Autoheal for EMQX Broker

Host: GitHub
URL: https://github.com/emqx/ekka
Owner: emqx
License: apache-2.0
Created: 2017-06-14T06:57:44.000Z (over 7 years ago)
Default Branch: master
Last Pushed: 2024-06-20T09:39:11.000Z (5 months ago)
Last Synced: 2024-07-22T18:37:10.818Z (3 months ago)
Topics: autocluster, autoheal, clustering, erlang-library, etcd, kubernetes
Language: Erlang
Homepage: https://www.emqx.com
Size: 821 KB
Stars: 97
Watchers: 29
Forks: 49
Open Issues: 8
Metadata Files:
- Readme: README.md
- License: LICENSE

Awesome Lists containing this project

README

        
# Ekka

Ekka - Autocluster for EMQX Broker. Ekka helps building a new distribution layer for EMQ X R2.3+.

```

----------             ----------

|  EMQX  |<--- MQTT--->|  EMQX  |

|--------|             |--------|

|  Ekka  |<----RPC---->|  Ekka  |

|--------|             |--------|

| Mnesia |<--Cluster-->| Mnesia |

|--------|             |--------|

| Kernel |<----TCP---->| Kernel |

----------             ----------

```

## Node discovery and Autocluster

Ekka supports erlang node discovery and autocluster using various strategies:

Strategy   | Description

-----------|--------------------------------------

manual     | Join cluster manually

static     | Static node list

dns        | DNS A Records

etcd       | etcd

k8s        | Kubernetes

The configuration example files are under 'etc/' folder.

### Cluster using static node list

Cuttlefish style config:

```

cluster.discovery = static

cluster.static.seeds = [email protected],[email protected]

```

Erlang config:

```

{cluster_discovery,

  {static, [

    {seeds, ['[email protected]', '[email protected]']}

  ]}},

```

### Cluster using DNS A records

Cuttlefish style config:

```

cluster.discovery = dns

## DNS name.

##

## Value: String

cluster.dns.name = localhost

## The App name is used to build 'node.name' with IP address.

##

## Value: String

cluster.dns.app = ekka

```

Erlang config:

```

{cluster_discovery,

  {dns, [

    {name, "localhost"},

    {app, "ekka"}

  ]}},

```

### Cluster using etcd

Cuttlefish style config:

```

cluster.discovery = etcd

## Etcd server list, seperated by ','.

##

## Value: String

cluster.etcd.server = http://127.0.0.1:2379

## The prefix helps build nodes path in etcd. Each node in the cluster

## will create a path in etcd: v2/keys///

##

## Value: String

cluster.etcd.prefix = ekkacl

## The TTL for node's path in etcd.

##

## Value: Duration

##

## Default: 1m, 1 minute

cluster.etcd.node_ttl = 1m

## Path to a file containing the client's private PEM-encoded key.

##

## Value: File

##

## cluster.etcd.keyfile = {{platform_etc_dir}}/certs/client-key.pem

## Path to the file containing the client's certificate

##

## Value: File

##

## cluster.etcd.certfile = {{platform_etc_dir}}/certs/client.pem

## Path to the file containing PEM-encoded CA certificates. The CA certificates

## are used during server authentication and when building the client certificate chain.

##

## Value: File

##

## cluster.etcd.cacertfile = {{platform_etc_dir}}/certs/ca.pem

```

Erlang config:

```

{cluster_discovery,

  {etcd, [

    {server, ["http://127.0.0.1:2379"]},

    {prefix, "ekkacluster"},

    %%{ssl_options, [

    %%    {keyfile, "path/to/client-key.pem"},

    %%    {certfile, "path/to/client.pem"},

    %%    {cacertfile, "path/to/ca.pem"}

    %%]},

    {node_ttl, 60000}

  ]}},

```

### Cluster using Kubernates

Cuttlefish style config:

```

cluster.discovery = k8s

## Kubernates API server list, seperated by ','.

##

## Value: String

## cluster.k8s.apiserver = http://10.110.111.204:8080

## The service name helps lookup EMQ nodes in the cluster.

##

## Value: String

## cluster.k8s.service_name = ekka

## The name space of k8s

##

## Value: String

## cluster.k8s.namespace = default

## The address type is used to extract host from k8s service.

##

## Value: ip | dns | hostname

## cluster.k8s.address_type = ip

## The app name helps build 'node.name'.

##

## Value: String

## cluster.k8s.app_name = ekka

## The suffix added to dns and hostname get from k8s service

##

## Value: String

## cluster.k8s.suffix = pod.cluster.local

```

Erlang config:

```

{cluster_discovery,

  {k8s, [

    {apiserver, "http://10.110.111.204:8080"},

    {namespace, "default"},

    {service_name, "ekka"},

    {address_type, ip},

    {app_name, "ekka"},

    {suffix, "pod.cluster.local"}

  ]}}

```

## Network partition and Autoheal

### Autoheal Design

When network partition occurs, the following steps to heal the cluster if autoheal is enabled:

1. Node reports the partitions to a leader node which has the oldest guid.

2. Leader node create a global netsplit view and choose one node in the majority as coordinator.

3. Leader node requests the coordinator to autoheal the network partition.

4. Coordinator node reboots all the nodes in the minority side.

### Enable autoheal

Erlang config:

```

{cluster_autoheal, true},

```

Cuttlefish style config:

```

cluster.autoheal = on

```

## Lock Service

Ekka implements a simple distributed lock service in 0.3 release. The Lock APIs:

Acquire lock:

```

-spec(acquire(resource()) -> {boolean(), [node()]}).

ekka_locker:acquire(Resource).

-spec(acquire(atom(), resource(), lock_type()) -> lock_result()).

ekka_locker:acquire(ekka_locker, Resource, Type).

```

Release lock:

```

-spec(release(resource()) -> lock_result()).

ekka_locker:release(Resource).

-spec(release(atom(), resource()) -> lock_result()).

ekka_locker:release(Name, Resource).

```

The lock type:

```

-type(lock_type() :: local | leader | quorum | all).

```

## Cluster without epmd

The ekka 0.6.0 release implements erlang distribiton without epmd.

See: http://erlang.org/pipermail/erlang-questions/2015-December/087013.html

For example:

```

## Dist port: 4370

erl -pa ebin -pa _build/default/lib/*/ebin -proto_dist ekka -start_epmd false -epmd_module ekka_epmd -name [email protected] -s ekka

## Dist port: 4371

erl -pa ebin -pa _build/default/lib/*/ebin -proto_dist ekka -start_epmd false -epmd_module ekka_epmd -name [email protected] -s ekka

## Dist port: 4372

erl -pa ebin -pa _build/default/lib/*/ebin -proto_dist ekka -start_epmd false -epmd_module ekka_epmd -name [email protected]  -s ekka

```

The erlang distribution port can be tuned by ekka `inet_dist_base_port` env. The default port is 4370.

## License

Apache License Version 2.0

## Author

EMQX Team.