https://github.com/superstreamlabs/memphis-k8s
Memphis.dev is an intelligent, frictionless message broker. Made to enable developers to build real-time and streaming features fast.
https://github.com/superstreamlabs/memphis-k8s
k8s kafka memphis message-queue messaging nats natsio rabbitmq
Last synced: about 1 year ago
JSON representation
Memphis.dev is an intelligent, frictionless message broker. Made to enable developers to build real-time and streaming features fast.
- Host: GitHub
- URL: https://github.com/superstreamlabs/memphis-k8s
- Owner: superstreamlabs
- License: apache-2.0
- Created: 2022-02-01T12:22:44.000Z (over 4 years ago)
- Default Branch: master
- Last Pushed: 2024-06-24T23:34:48.000Z (almost 2 years ago)
- Last Synced: 2025-03-31T09:11:33.513Z (about 1 year ago)
- Topics: k8s, kafka, memphis, message-queue, messaging, nats, natsio, rabbitmq
- Language: Mustache
- Homepage: https://memphis.dev
- Size: 3.52 MB
- Stars: 22
- Watchers: 2
- Forks: 13
- Open Issues: 4
-
Metadata Files:
- Readme: README.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
Awesome Lists containing this project
README
**[Memphis.dev](https://memphis.dev)** is a highly scalable, painless, and effortless data streaming platform.
Made to enable developers and data teams to collaborate and build
real-time and streaming apps fast.
# Memphis Kubernetes Deployment
Helm is a k8s package manager that allows users to deploy apps in a single, configurable command.
More information about Helm can be found [here](https://helm.sh/docs/topics/charts/).
Memphis is cloud-native and cloud-agnostic to any Kubernetes on **any cloud**.
## Requirements
**Minimum Requirements (Without high availability)**
ResourceQuantityMinimum Kubernetes version1.20 and aboveK8S Nodes1CPU2 CPUMemory4GB RAMStorage12GB PVC
***
**Recommended Requirements (With high availability)**
| Resource | Minimum Quantity |
| -------------------------- | ----------------- |
| Minimum Kubernetes version | 1.20 and above |
| K8S Nodes | 3 |
| CPU | 4 CPU |
| Memory | 8GB RAM |
| Storage | 12GB PVC Per node |
## Installation
Production
Production-grade Memphis with three memphis brokers configured in cluster-mode
```bash
helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && helm install memphis memphis/memphis --set global.cluster.enabled="true" --create-namespace --namespace memphis --wait
```
Dev
Standard installation of Memphis with a single broker
```bash
helm repo add memphis https://k8s.memphis.dev/charts/ --force-update &&
helm install memphis memphis/memphis --create-namespace --namespace memphis --wait
```
#### \* Optional \* Helm deployment options
| Option | Description | Default Value | Example |
| --- | --- | --- | --- |
| global.cluster.enabled | Cluster mode for HA and Performance | `"false"` | `"false"` |
| exporter.enabled | Prometheus exporter | `"false"` | `"false"` |
| exporter.serviceExposed.enbaled | Expose metrics port with memphis service | `"false"` | `"true"` |
| cluster.enabled | Enables Memphis cluster deployment. For fully HA configuration use global.cluster.enabled | `"false"` | `"true"` |
| cluster.replicas | Memphis broker replicas | `"3"` | `"5"` |
| memphis.image | Memphis image name | "memphisos/memphis:x.x.x-stable" | "memphisos/memphis:latest" |
| memphis.ui.port | Dashboard's (GUI) port | 9000 | 9000 |
| memphis.hosts.uiHostName | Which URL should be seen as the "UI hostname" | "" | `"https://memphis.example.com"` |
| memphis.hosts.restgwHostName | Which URL should be seen as the "REST Gateway hostname" | "" | `"https://restgw.memphis.example.com"` |
| memphis.hosts.brokerHostName | Which URL should be seen as the "broker hostname" | "" | `"memphis.example.com"` |
| memphis.configFile.logsRetentionInDays | Amount of days to retain system logs | 3 | 3 |
| memphis.configFile.gcProducerConsumerRetentionInHours | Amount of hours to retain producer/consumer in system | 3 | 3 |
| memphis.configFile.tieredStorageUploadIntervalSeconds | Interval in seconds between uploads to tiered storage | 8 | 8 |
| memphis.configFile.dlsRetentionHours | Amount of hours to retain messages in DLS | 3 | 3 |
| memphis.configFile.userPassBasedAuth | Authentication method selector.
`true = User + pass`
`false = User + connection token` | `"true"` | `"true"` |
| memphis.creds.rootPwd | Root password for the dashboard. Randomly generated. | "" | "superpass" |
| memphis.creds.connectionToken | Token for connecting an app to the Memphis Message Queue. Auto generated.Randomly generated. | "" | "connectionToken |
| memphis.creds.jwtSecret | For internal traffic. Randomly generated. | "" | "<JWT_TOKEN>" |
| memphis.creds.refreshJwtSecret | For internal traffic. Randomly generated. | "" | "<JWT_TOKEN>" |
| memphis.creds.encryptionSecretKey | Encryption secret key for internal encryption. Randomly generated. | "" | "" |
| memphis.creds.secretConfig.name | **\*Optional\***
Name of the secret | "memphis-creds" | "memphis-creds" |
| memphis.creds.secretConfig.existingSecret | **\*Optional\***
Is this secret an existing secret | "false" | "false" |
| memphis.creds.secretConfig.rootPwd_key | **\*Optional\***
Name of the key in secret | "ROOT_PASSWORD" | "rootPwd" |
| memphis.creds.secretConfig.connectionToken_key | **\*Optional\***
Name of the key in secret | "CONNECTION_TOKEN" | "connectionToken" |
| memphis.creds.secretConfig.jwtSecret_key | **\*Optional\***
Name of the key in secret | "JWT_SECRET" | "jwtSecret" |
| memphis.creds.secretConfig.refreshJwtSecret_key | **\*Optional\***
Name of the key in secret | "REFRESH_JWT_SECRET" | "refreshJwtSecret" |
| memphis.creds.secretConfig.encryptionSecretKey_key | **\*Optional\***
Name of the key in secret | "ENCRYPTION_SECRET_KEY" | "encryptionSecretKey" |
| memphis.creds.secretConfig.refreshJwtSecretRestGW_key | **\*Optional\***
Name of the key in secret | "REFRESH_JWT_SECRET_REST_GW" | "refreshJwtSecretRestGW" |
| memphis.creds.secretConfig.jwtSecretRestGW_key | **\*Optional\***
Name of the key in secret | "JWT_SECRET_REST_GW" | "jwtSecretRestGW" |
| memphis.extraEnvironmentVars.enabled | **\*Optional\***
List of additional environment variables for memphis. | "" | vars:
\- name: KEY
\- valye: value |
| memphis.tls.verify | **\*Optional\***
For encrypted client-memphis communication. Verification for the CA autority. SSL. | "" | `"true"` |
| memphis.tls.secret.name | **\*Optional\***
For encrypted client-memphis communication.
K8S secret name that holds the certs. SSL. | "" | `"memphis-client-tls-secret"` |
| memphis.tls.cert | **\*Optional\***
For encrypted client-memphis communication.
.pem file to use. SSL. | "" | `"memphis_client.pem"` |
| memphis.tls.key | **\*Optional\***
For encrypted client-memphis communication.
Private key file to use. SSL. | "" | `"memphis-key_client.pem"` |
| memphis.tls.ca | **\*Optional\***
For encrypted client-memphis communication.
CA file to use. SSL. | "" | `"rootCA.pem"` |
| websocket.enabled | **\*Optional\*** Memphis GUI using websockets for live rendering. | "" | `"false"` |
| websocket.port | Memphis GUI using websockets for live rendering. The port can be configured | "7770" | "77777" |
| websocket.host | Websocket host can be handled on separate LB/DNS. | "localhost" | "ws.example.com" |
| websocket.noTLS | Websocket can be configured with tls, default is noTLS. | "true" | "false" |
| websocket.tls.secret.name | **\*Optional\*** Memphis GUI using websockets for live rendering.
K8S secret name for the certs | "" | `"memphis-ws-tls-secret"` |
| websocket.tls.cert | **\*Optional\***
Memphis GUI using websockets for live rendering.
.pem file to use | "" | `"memphis_local.pem"` |
| websocket.tls.key | **\*Optional\***
Memphis GUI using websockets for live rendering.
key file | "" | `"memphis-key_local.pem"` |
| metadata.postgresql.username | **\*Optional\***
Username for postgres db | "postgres" | "postgres" |
| metadata.postgresql.existingSecret | **\*Optional\***
An ability to provide predefined secret for metadata PostgreSQL credentials | "" | "metadata-creds.yaml" |
| metadata.pgpool.existingSecret | **\*Optional\***
An ability to provide predefined secret for metadata PG credentials | "" | "metadata-creds.yaml" |
| metadata.pgpool.tls.enabled | **\*Optional\***
Enabling TLS-based communication with PG | "false" | "false" |
| metadata.pgpool.tls.certificatesSecret | **\*Optional\***
PG TLS cert secret to be used | "" | "tls-secret" |
| metadata.pgpool.tls.certFilename | **\*Optional\***
PG TLS cert file to be used | "" | "tls.crt" |
| metadata.pgpool.tls.certKeyFilename | **\*Optional\***
PG TLS key to be used | "" | "tls.key" |
| metadata.pgpool.tls.certCAFilename | **\*Optional\***
PG TLS cert CA to be used | "" | "ca.crt" |
| metadata.external.enabled | **\*Optional\***
For using external PG instead of deploying dedicated one for Memphis | "false" | "true" |
| metadata.external.dbTlsMutual | **\*Optional\***
External PG TLS-basec communication | "true" | "true" |
| metadata.external.dbName | **\*Optional\***
External PG db name | "" | "memphis" |
| metadata.external.dbHost | **\*Optional\***
External PG db hostname | "" | "metadata.example.url" |
| metadata.external.dbPort | **\*Optional\***
External PG db port | "" | 5432 |
| metadata.external.dbUser | **\*Optional\***
External PG db user | "" | "postgres" |
| metadata.external.dbPass | **\*Optional\***
External PG db password | "" | "12345678" |
| metadata.external.secret.enabled | **\*Optional\***
Enable an option to use secret for password store | "false" | "true" |
| metadata.external.secret.name | **\*Optional\***
Secret name | "" | "metadata-secret" |
| metadata.external.secret.dbPass_key | **\*Optional\***
Name of the key in the secret | "" | "dbPass" |
| restGateway.enabled | **\*Optional\***
Memphis Rest Gateway can be disabled if not in use | "true" | "false" |
| restGateway.jwtSecret | **\*Optional\***
Manual Jwt Token configurtion | "" | "" |
| restGateway.refreshJwtSecret | **\*Optional\***
Manual Refresh Jwt Token configurtion | "" | "" |
| auth.enabled | **\*Optional\***
Enable using predefined parameters | "false" | "true" |
| auth.enabled.mgmt | **\*Optional\***
Management users that will be created at first deployment | "" | "" |
| auth.enabled.client | **\*Optional\***
Client users that will be created at first deployment | "" | "" |
Here is how to run an installation command with additional options -
```
helm install memphis --set cluster.replicas=3,memphis.creds.rootPwd=rootpassword" memphis/memphis --create-namespace --namespace memphis
```
### Deployed pods
- **memphis.** Memphis broker.
- **memphis-rest-gateway.** Memphis REST Gateway.
- **memphis-metadata.** Metadata store.
- **memphis-metadata-coordinator.** Metadata coordinator
For more information on each component, please head to the [architecture section](../../memphis/architecture.md#key-components).
## Deploy Memphis with TLS (encrypted communication via SSL)
### 0. Optional: Create self-signed certificates
a) Generate a self-signed certificate using `mkcert`
```bash
$ mkcert -client \
-cert-file memphis_client.pem \
-key-file memphis-key_client.pem \
"127.0.0.1" "localhost" "*.memphis.dev" ::1 \
email@localhost valera@Valeras-MBP-2.lan
```
b) Find the `rootCA`
```
$ mkcert -CAROOT
```
c) Create self-signed certificates for client
```bash
$ mkcert -client -cert-file client.pem -key-file key-client.pem localhost ::1
```
### 1. Create namespace + secret for the TLS certs
a) Create a dedicated namespace for memphis
```bash
kubectl create namespace memphis
```
b) Create a k8s secret with the required certs
```bash
kubectl create secret generic memphis-client-tls-secret \
--from-file=memphis_client.pem \
--from-file=memphis-key_client.pem \
--from-file=rootCA.pem -n memphis
```
```yaml
tls:
secret:
name: memphis-client-tls-secret
ca: "rootCA.pem"
cert: "memphis_client.pem"
key: "memphis-key_client.pem"
```
### 2. Deploy Memphis with the generated certificate
```bash
helm install memphis memphis \
--create-namespace --namespace memphis --wait \
--set \
global.cluster.enabled="true",\
memphis.tls.verify="true",\
memphis.tls.cert="memphis_client.pem",\
memphis.tls.key="memphis-key_client.pem",\
memphis.tls.secret.name="memphis-client-tls-secret",\
memphis.tls.ca="rootCA.pem"
```
## Upgrade existing deployment
### For adding TLS support
1. Create a k8s secret with the provided TLS certs
```
kubectl create secret generic memphis-client-tls-secret \
--from-file=memphis_client.pem \
--from-file=memphis-key_client.pem \
--from-file=rootCA.pem -n memphis
```
2. Upgrade Memphis to use the TLS certs
```bash
helm upgrade memphis memphis -n memphis --reuse-values \
--set \
memphis.tls.verify="true",\
memphis.tls.cert="memphis_client.pem",\
memphis.tls.key="memphis-key_client.pem",\
memphis.tls.secret.name="tls-client-secret",\
memphis.tls.ca="rootCA.pem"
```
## Deployment diagram
