Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/nats-io/nats-operator

NATS Operator
https://github.com/nats-io/nats-operator

cluster kubernetes message-queue nats operator pubsub

Last synced: about 1 month ago
JSON representation

NATS Operator

Host: GitHub
URL: https://github.com/nats-io/nats-operator
Owner: nats-io
License: apache-2.0
Created: 2016-12-04T16:07:52.000Z (about 8 years ago)
Default Branch: master
Last Pushed: 2023-11-15T22:02:19.000Z (about 1 year ago)
Last Synced: 2024-04-14T02:21:09.257Z (8 months ago)
Topics: cluster, kubernetes, message-queue, nats, operator, pubsub
Language: Go
Homepage: https://nats-io.github.io/k8s/
Size: 11.7 MB
Stars: 570
Watchers: 15
Forks: 115
Open Issues: 58
Metadata Files:
- Readme: README.md
- License: LICENSE
- Governance: GOVERNANCE.md

Awesome Lists containing this project

README

# NATS Operator

> :warning: The recommended way of running NATS on Kubernetes is by using the [Helm charts](https://github.com/nats-io/k8s/tree/master/helm/charts/nats). If looking for [JetStream](https://github.com/nats-io/jetstream) support, this is supported in the [Helm charts](https://github.com/nats-io/k8s/tree/master/helm/charts/nats#jetstream). The NATS Operator is not recommended to be used for new deployments.

[![License Apache 2.0](https://img.shields.io/badge/License-Apache2-blue.svg)](https://www.apache.org/licenses/LICENSE-2.0)
[![Build Status](https://travis-ci.org/nats-io/nats-operator.svg?branch=master)](https://travis-ci.org/nats-io/nats-operator)
[![Version](https://d25lcipzij17d.cloudfront.net/badge.svg?id=go&type=5&v=0.8.2)](https://github.com/nats-io/nats-operator/releases/tag/v0.8.2)

NATS Operator manages NATS clusters atop [Kubernetes][k8s-home] using [CRDs](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/). If looking to run NATS on K8S without the operator you can also find [Helm charts in the nats-io/k8s repo](https://github.com/nats-io/k8s#helm-charts-for-nats). You can also find more info about running NATS on Kubernetes in the [docs](https://docs.nats.io/nats-on-kubernetes/nats-kubernetes) as well as a minimal setup using `StatefulSets` only without using the operator to get started [here](https://docs.nats.io/nats-on-kubernetes/minimal-setup).

[k8s-home]: http://kubernetes.io

## Requirements

- Kubernetes v1.10+.
- [Configuration reloading](#configuration-reload) is only supported in Kubernetes v1.12+.
- [Authentication using service accounts](#auth-service-accounts) is only supported in Kubernetes v1.12+ having the `TokenRequest` API enabled.

## Introduction

NATS Operator provides a `NatsCluster` [Custom Resources Definition](https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-definitions/) (CRD) that models a NATS cluster.
This CRD allows for specifying the desired size and version for a NATS cluster, as well as several other advanced options:

```yaml
apiVersion: nats.io/v1alpha2
kind: NatsCluster
metadata:
name: example-nats-cluster
spec:
size: 3
version: "2.1.8"
```

NATS Operator monitors creation/modification/deletion of `NatsCluster` resources and reacts by attempting to perform the any necessary operations on the associated NATS clusters in order to align their current status with the desired one.

## Installing

NATS Operator supports two different operation modes:

* **Namespace-scoped (classic):** NATS Operator manages `NatsCluster` resources on the Kubernetes namespace where it is deployed.
* **Cluster-scoped (experimental):** NATS Operator manages `NatsCluster` resources across all namespaces in the Kubernetes cluster.

The operation mode must be chosen when installing NATS Operator and cannot be changed later.

### Namespace-scoped installation

To perform a namespace-scoped installation of NATS Operator in the Kubernetes cluster pointed at by the current context, you may run:

```console
$ kubectl apply -f https://github.com/nats-io/nats-operator/releases/latest/download/00-prereqs.yaml
$ kubectl apply -f https://github.com/nats-io/nats-operator/releases/latest/download/10-deployment.yaml
```

This will, by default, install NATS Operator in the `default` namespace and observe `NatsCluster` resources created in the `default` namespace, alone.
In order to install in a different namespace, you must first create said namespace and edit the manifests above in order to specify its name wherever necessary.

**WARNING:** To perform multiple namespace-scoped installations of NATS Operator, you must manually edit the `nats-operator-binding` cluster role binding in `deploy/00-prereqs.yaml` file in order to add all the required service accounts.
Failing to do so may cause all NATS Operator instances to malfunction.

**WARNING:** When performing a namespace-scoped installation of NATS Operator, you must make sure that all other namespace-scoped installations that may exist in the Kubernetes cluster share the same version.
Installing different versions of NATS Operator in the same Kubernetes cluster may cause unexpected behavior as the schema of the CRDs which NATS Operator registers may change between versions.

Alternatively, you may use [Helm](https://www.helm.sh/) to perform a namespace-scoped installation of NATS Operator.
To do so you may go to [helm/nats-operator](https://github.com/nats-io/nats-operator/tree/master/helm/nats-operator) and use the Helm charts found in that repo.

### Cluster-scoped installation (experimental)

Cluster-scoped installations of NATS Operator must live in the `nats-io` namespace.
This namespace must be created beforehand:

```console
$ kubectl create ns nats-io
```

Then, you must manually edit the manifests in `deployment/` in order to reference the `nats-io` namespace and to enable the `ClusterScoped` feature gate in the NATS Operator deployment.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nats-operator
namespace: nats-io
spec:
(...)
spec:
containers:
- name: nats-operator
(...)
args:
- nats-operator
- --feature-gates=ClusterScoped=true
(...)
```

Once you have done this, you may install NATS Operator by running:

**WARNING:** When performing a cluster-scoped installation of NATS Operator, you must make sure that there are no other deployments of NATS Operator in the Kubernetes cluster.
If you have a previous installation of NATS Operator, you must uninstall it before performing a cluster-scoped installation of NATS Operator.

## Creating a NATS cluster

Once NATS Operator has been installed, you will be able to confirm that two new CRDs have been registered in the cluster:

```console
$ kubectl get crd
NAME CREATED AT
natsclusters.nats.io 2019-01-11T17:16:36Z
natsserviceroles.nats.io 2019-01-11T17:16:40Z
```

To create a NATS cluster, you must create a `NatsCluster` resource representing the desired status of the cluster.
For example, to create a 3-node NATS cluster you may run:

```console
$ cat <
#### Using ServiceAccounts

> :warning: The ServiceAccounts uses a very rudimentary approach of config reloading and watching CRDs and advanced K8S APIs that may not be available in your cluster. Instead, the decentralized JWT approach should be preferred, to learn more: https://docs.nats.io/developing-with-nats/tutorials/jwt

The NATS Operator can define permissions based on Roles by using any present ServiceAccount in a namespace.
This feature requires a Kubernetes v1.12+ cluster having the `TokenRequest` API enabled.
To try this feature using `minikube` v0.30.0+, you can configure it to start as follows:

```console
$ minikube start \
--extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key \
--extra-config=apiserver.service-account-key-file=/var/lib/minikube/certs/sa.pub \
--extra-config=apiserver.service-account-issuer=api \
--extra-config=apiserver.service-account-api-audiences=api,spire-server \
--extra-config=apiserver.authorization-mode=Node,RBAC \
--extra-config=kubelet.authentication-token-webhook=true
```

Please note that availability of this feature across Kubernetes offerings may vary widely.

ServiceAccounts integration can then be enabled by setting the
`enableServiceAccounts` flag to true in the `NatsCluster` configuration.

```yaml
apiVersion: nats.io/v1alpha2
kind: NatsCluster
metadata:
name: example-nats
spec:
size: 3
version: "1.3.0"

pod:
# NOTE: Only supported in Kubernetes v1.12+.
enableConfigReload: true
auth:
# NOTE: Only supported in Kubernetes v1.12+ clusters having the "TokenRequest" API enabled.
enableServiceAccounts: true
```

Permissions for a `ServiceAccount` can be set by creating a
`NatsServiceRole` for that account. In the example below, there are
two accounts, one is an admin user that has more permissions.

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nats-admin-user
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nats-user
---
apiVersion: nats.io/v1alpha2
kind: NatsServiceRole
metadata:
name: nats-user
namespace: nats-io

# Specifies which NATS cluster will be mapping this account.
labels:
nats_cluster: example-nats
spec:
permissions:
publish: ["foo.*", "foo.bar.quux"]
subscribe: ["foo.bar"]
---
apiVersion: nats.io/v1alpha2
kind: NatsServiceRole
metadata:
name: nats-admin-user
namespace: nats-io
labels:
nats_cluster: example-nats
spec:
permissions:
publish: [">"]
subscribe: [">"]
```

The above will create two different Secrets which can then be mounted as volumes
for a Pod.

```sh
$ kubectl -n nats-io get secrets
NAME TYPE DATA AGE
...
nats-admin-user-example-nats-bound-token Opaque 1 43m
nats-user-example-nats-bound-token Opaque 1 43m
```

Please note that `NatsServiceRole` must be created in the same namespace as
`NatsCluster` is running, but `bound-token` will be created for `ServiceAccount`
resources that can be placed in various namespaces.

An example of mounting the secret in a `Pod` can be found below:

```yaml
apiVersion: v1
kind: Pod
metadata:
name: nats-user-pod
labels:
nats_cluster: example-nats
spec:
volumes:
- name: "token"
projected:
sources:
- secret:
name: "nats-user-example-nats-bound-token"
items:
- key: token
path: "token"
restartPolicy: Never
containers:
- name: nats-ops
command: ["/bin/sh"]
image: "wallyqs/nats-ops:latest"
tty: true
stdin: true
stdinOnce: true
volumeMounts:
- name: "token"
mountPath: "/var/run/secrets/nats.io"
```

Then within the `Pod` the token can be used to authenticate against
the server using the created token.

```sh
$ kubectl -n nats-io attach -it nats-user-pod

/go # nats-sub -s nats://nats-user:`cat /var/run/secrets/nats.io/token`@example-nats:4222 hello.world
Listening on [hello.world]
^C
/go # nats-sub -s nats://nats-admin-user:`cat /var/run/secrets/nats.io/token`@example-nats:4222 hello.world
Can't connect: nats: authorization violation
```

#### Using a single secret with the explicit configuration.

Authorization can also be set for the server by using a secret
where the permissions are defined in JSON:

```json
{
"users": [
{ "username": "user1", "password": "secret1" },
{ "username": "user2", "password": "secret2",
"permissions": {
"publish": ["hello.*"],
"subscribe": ["hello.world"]
}
}
],
"default_permissions": {
"publish": ["SANDBOX.*"],
"subscribe": ["PUBLIC.>"]
}
}
```

Example of creating a secret to set the permissions:

```sh
kubectl create secret generic nats-clients-auth --from-file=clients-auth.json
```

Now when creating a NATS cluster it is possible to set the permissions as
in the following example:

```yaml
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
pmetadata:
name: "example-nats-auth"
spec:
size: 3
version: "1.1.0"

auth:
# Definition in JSON of the users permissions
clientsAuthSecret: "nats-clients-auth"

# How long to wait for authentication
clientsAuthTimeout: 5
```

### Configuration Reload

On Kubernetes v1.12+ clusters it is possible to enable on-the-fly reloading of configuration for the servers that are part of the cluster.
This can also be combined with the authorization support, so in case the user permissions change, then the servers will reload and apply the new permissions.

```yaml
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
name: "example-nats-auth"
spec:
size: 3
version: "1.1.0"

pod:
# Enable on-the-fly NATS Server config reload
# NOTE: Only supported in Kubernetes v1.12+.
enableConfigReload: true

# Possible to customize version of reloader image
reloaderImage: connecteverything/nats-server-config-reloader
reloaderImageTag: "0.2.2-v1alpha2"
reloaderImagePullPolicy: "IfNotPresent"
auth:
# Definition in JSON of the users permissions
clientsAuthSecret: "nats-clients-auth"

# How long to wait for authentication
clientsAuthTimeout: 5
```

## Connecting operated NATS clusters to external NATS clusters

By using the `extraRoutes` field on the spec you can make the operated
NATS cluster create routes against clusters outside of Kubernetes:

```yaml
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
name: "nats"
spec:
size: 3
version: "1.4.1"

extraRoutes:
- route: "nats://nats-a.example.com:6222"
- route: "nats://nats-b.example.com:6222"
- route: "nats://nats-c.example.com:6222"
```

It is also possible to connect to another operated NATS cluster as follows:

```yaml
apiVersion: "nats.io/v1alpha2"
kind: "NatsCluster"
metadata:
name: "nats-v2-2"
spec:
size: 3
version: "1.4.1"

extraRoutes:
- cluster: "nats-v2-1"
```

## Resolvers

The operator only supports the `URL()` resolver, see [example/example-super-cluster.yaml](example/example-super-cluster.yaml#L56-L59)

## Development

### Building the Docker Image

To build the `nats-operator` Docker image:

```sh
$ docker build -f docker/operator/Dockerfile . -t
```

To build the `nats-server-config-reloader`:

```sh
$ docker build -f docker/reloader/Dockerfile . -t
```

You'll need Docker `17.06.0-ce` or higher.