{"id":20812566,"url":"https://github.com/superstreamlabs/memphis-k8s","last_synced_at":"2025-05-07T10:45:14.820Z","repository":{"id":37008861,"uuid":"454377089","full_name":"superstreamlabs/memphis-k8s","owner":"superstreamlabs","description":"Memphis.dev is an intelligent, frictionless message broker. Made to enable developers to build real-time and streaming features fast.","archived":false,"fork":false,"pushed_at":"2024-06-24T23:34:48.000Z","size":3689,"stargazers_count":22,"open_issues_count":4,"forks_count":13,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-03-31T09:11:33.513Z","etag":null,"topics":["k8s","kafka","memphis","message-queue","messaging","nats","natsio","rabbitmq"],"latest_commit_sha":null,"homepage":"https://memphis.dev","language":"Mustache","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/superstreamlabs.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-01T12:22:44.000Z","updated_at":"2024-10-09T21:09:22.000Z","dependencies_parsed_at":"2023-12-20T18:27:35.367Z","dependency_job_id":"215d65b8-6785-4f52-9e57-cfb71be1919f","html_url":"https://github.com/superstreamlabs/memphis-k8s","commit_stats":null,"previous_names":["superstreamlabs/memphis-k8s"],"tags_count":26,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-k8s","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-k8s/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-k8s/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/superstreamlabs%2Fmemphis-k8s/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/superstreamlabs","download_url":"https://codeload.github.com/superstreamlabs/memphis-k8s/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252862834,"owners_count":21815925,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["k8s","kafka","memphis","message-queue","messaging","nats","natsio","rabbitmq"],"created_at":"2024-11-17T20:55:31.425Z","updated_at":"2025-05-07T10:45:14.799Z","avatar_url":"https://github.com/superstreamlabs.png","language":"Mustache","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003ca href=\"![Github (2)](https://github.com/memphisdev/memphis.js/assets/107035359/731a59be-0f46-4a94-84c3-c0b2a07fe01c)\"\u003e![Github (2)](https://github.com/memphisdev/memphis.js/assets/107035359/281222f9-8f93-4a20-9de8-7c26541bded7)\u003c/a\u003e\n\u003cp align=\"center\"\u003e\n\u003ca href=\"https://memphis.dev/discord\"\u003e\u003cimg src=\"https://img.shields.io/discord/963333392844328961?color=6557ff\u0026label=discord\" alt=\"Discord\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/memphisdev/memphis/issues?q=is%3Aissue+is%3Aclosed\"\u003e\u003cimg src=\"https://img.shields.io/github/issues-closed/memphisdev/memphis?color=6557ff\"\u003e\u003c/a\u003e \n  \u003cimg src=\"https://img.shields.io/npm/dw/memphis-dev?color=ffc633\u0026label=installations\"\u003e\n\u003ca href=\"https://github.com/memphisdev/memphis/blob/master/CODE_OF_CONDUCT.md\"\u003e\u003cimg src=\"https://img.shields.io/badge/Code%20of%20Conduct-v1.0-ff69b4.svg?color=ffc633\" alt=\"Code Of Conduct\"\u003e\u003c/a\u003e \n\u003cimg alt=\"GitHub release (latest by date)\" src=\"https://img.shields.io/github/v/release/memphisdev/memphis?color=61dfc6\"\u003e\n\u003cimg src=\"https://img.shields.io/github/last-commit/memphisdev/memphis?color=61dfc6\u0026label=last%20commit\"\u003e\n\u003c/p\u003e\n\n\u003cdiv align=\"center\"\u003e\n  \n  \u003cimg width=\"200\" alt=\"CNCF Silver Member\" src=\"https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/color/cncf-member-silver-color.svg#gh-light-mode-only\"\u003e\n  \u003cimg width=\"200\" alt=\"CNCF Silver Member\" src=\"https://github.com/cncf/artwork/raw/master/other/cncf-member/silver/white/cncf-member-silver-white.svg#gh-dark-mode-only\"\u003e\n \n\n  \u003ch4\u003e\n\n**[Memphis.dev](https://memphis.dev)** is a highly scalable, painless, and effortless data streaming platform.\u003cbr\u003e\nMade to enable developers and data teams to collaborate and build\u003cbr\u003e\nreal-time and streaming apps fast.\n\n  \u003c/h4\u003e\n  \n\u003c/div\u003e\n\n# Memphis Kubernetes Deployment\n\nHelm is a k8s package manager that allows users to deploy apps in a single, configurable command.\u003cbr\u003e\nMore information about Helm can be found [here](https://helm.sh/docs/topics/charts/).\n\nMemphis is cloud-native and cloud-agnostic to any Kubernetes on **any cloud**.\n\n## Requirements\n\n**Minimum Requirements (Without high availability)**\n\n\u003ctable\u003e\u003cthead\u003e\u003ctr\u003e\u003cth\u003eResource\u003c/th\u003e\u003cth\u003eQuantity\u003c/th\u003e\u003cth data-hidden\u003e\u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd\u003eMinimum Kubernetes version\u003c/td\u003e\u003ctd\u003e1.20 and above\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eK8S Nodes\u003c/td\u003e\u003ctd\u003e1\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eCPU\u003c/td\u003e\u003ctd\u003e2 CPU\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eMemory\u003c/td\u003e\u003ctd\u003e4GB RAM\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd\u003eStorage\u003c/td\u003e\u003ctd\u003e12GB PVC\u003c/td\u003e\u003ctd\u003e\u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\n\n***\n\n**Recommended Requirements (With high availability)**\n\n| Resource                   | Minimum Quantity  |\n| -------------------------- | ----------------- |\n| Minimum Kubernetes version | 1.20 and above    |\n| K8S Nodes                  | 3                 |\n| CPU                        | 4 CPU             |\n| Memory                     | 8GB RAM           |\n| Storage                    | 12GB PVC Per node |\n\n## Installation\n\n\u003cdetails\u003e\n\n\u003csummary\u003eProduction\u003c/summary\u003e\n\nProduction-grade Memphis with three memphis brokers configured in cluster-mode\n\n```bash\nhelm repo add memphis https://k8s.memphis.dev/charts/ --force-update \u0026\u0026 helm install memphis memphis/memphis --set global.cluster.enabled=\"true\" --create-namespace --namespace memphis --wait\n```\n\n\u003c/details\u003e\n\n\u003cdetails\u003e\n\n\u003csummary\u003eDev\u003c/summary\u003e\n\nStandard installation of Memphis with a single broker\n\n```bash\nhelm repo add memphis https://k8s.memphis.dev/charts/ --force-update \u0026\u0026 \nhelm install memphis memphis/memphis --create-namespace --namespace memphis --wait\n```\n\n\u003c/details\u003e\n\n#### \\* Optional \\* Helm deployment options\n\n| Option | Description | Default Value | Example |\n| --- | --- | --- | --- |\n| global.cluster.enabled | Cluster mode for HA and Performance | `\"false\"` | `\"false\"` |\n| exporter.enabled | Prometheus exporter | `\"false\"` | `\"false\"` |\n| exporter.serviceExposed.enbaled | Expose metrics port with memphis service | `\"false\"` | `\"true\"` |\n| cluster.enabled | Enables Memphis cluster deployment. For fully HA configuration use global.cluster.enabled | `\"false\"` | `\"true\"` |\n| cluster.replicas | Memphis broker replicas | `\"3\"` | `\"5\"` |\n| memphis.image | Memphis image name | \"memphisos/memphis:x.x.x-stable\" | \"memphisos/memphis:latest\" |\n| memphis.ui.port | Dashboard's (GUI) port | 9000 | 9000 |\n| memphis.hosts.uiHostName | Which URL should be seen as the \"UI hostname\" | \"\"  | `\"https://memphis.example.com\"` |\n| memphis.hosts.restgwHostName | Which URL should be seen as the \"REST Gateway hostname\" | \"\"  | `\"https://restgw.memphis.example.com\"` |\n| memphis.hosts.brokerHostName | Which URL should be seen as the \"broker hostname\" | \"\"  | `\"memphis.example.com\"` |\n| memphis.configFile.logsRetentionInDays | Amount of days to retain system logs | 3   | 3   |\n| memphis.configFile.gcProducerConsumerRetentionInHours | Amount of hours to retain producer/consumer in system | 3  | 3  |\n| memphis.configFile.tieredStorageUploadIntervalSeconds | Interval in seconds between uploads to tiered storage | 8  | 8  |\n| memphis.configFile.dlsRetentionHours | Amount of hours to retain messages in DLS | 3  | 3  |\n| memphis.configFile.userPassBasedAuth | Authentication method selector.  \u003cbr\u003e`true = User + pass`  \u003cbr\u003e`false = User + connection token` | `\"true\"` | `\"true\"` |\n| memphis.creds.rootPwd | Root password for the dashboard. Randomly generated. | \"\"  | \"superpass\" |\n| memphis.creds.connectionToken | Token for connecting an app to the Memphis Message Queue. Auto generated.Randomly generated. | \"\"  | \"connectionToken |\n| memphis.creds.jwtSecret | For internal traffic. Randomly generated. | \"\"  | \"\u0026lt;JWT_TOKEN\u0026gt;\" |\n| memphis.creds.refreshJwtSecret | For internal traffic. Randomly generated. | \"\"  | \"\u0026lt;JWT_TOKEN\u0026gt;\" |\n| memphis.creds.encryptionSecretKey | Encryption secret key for internal encryption. Randomly generated. | \"\"  | \"\"  |\n| memphis.creds.secretConfig.name | **\\*Optional\\***  \u003cbr\u003eName of the secret | \"memphis-creds\"  | \"memphis-creds\" |\n| memphis.creds.secretConfig.existingSecret | **\\*Optional\\***  \u003cbr\u003eIs this secret an existing secret | \"false\"  | \"false\" |\n| memphis.creds.secretConfig.rootPwd_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"ROOT_PASSWORD\"  | \"rootPwd\" |\n| memphis.creds.secretConfig.connectionToken_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"CONNECTION_TOKEN\"  | \"connectionToken\" |\n| memphis.creds.secretConfig.jwtSecret_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"JWT_SECRET\"  | \"jwtSecret\" |\n| memphis.creds.secretConfig.refreshJwtSecret_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"REFRESH_JWT_SECRET\"  | \"refreshJwtSecret\" |\n| memphis.creds.secretConfig.encryptionSecretKey_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"ENCRYPTION_SECRET_KEY\"  | \"encryptionSecretKey\" |\n| memphis.creds.secretConfig.refreshJwtSecretRestGW_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"REFRESH_JWT_SECRET_REST_GW\"  | \"refreshJwtSecretRestGW\" |\n| memphis.creds.secretConfig.jwtSecretRestGW_key | **\\*Optional\\***  \u003cbr\u003eName of the key in secret | \"JWT_SECRET_REST_GW\"  | \"jwtSecretRestGW\" |\n| memphis.extraEnvironmentVars.enabled | **\\*Optional\\***  \u003cbr\u003eList of additional environment variables for memphis. | \"\"  | vars:  \u003cbr\u003e\\- name: KEY  \u003cbr\u003e\\- valye: value |\n| memphis.tls.verify | **\\*Optional\\***  \u003cbr\u003eFor encrypted client-memphis communication. Verification for the CA autority. SSL. | \"\"  | `\"true\"` |\n| memphis.tls.secret.name | **\\*Optional\\***  \u003cbr\u003eFor encrypted client-memphis communication.  \u003cbr\u003eK8S secret name that holds the certs. SSL. | \"\"  | `\"memphis-client-tls-secret\"` |\n| memphis.tls.cert | **\\*Optional\\***  \u003cbr\u003eFor encrypted client-memphis communication.  \u003cbr\u003e.pem file to use. SSL. | \"\"  | `\"memphis_client.pem\"` |\n| memphis.tls.key | **\\*Optional\\***  \u003cbr\u003eFor encrypted client-memphis communication.  \u003cbr\u003ePrivate key file to use. SSL. | \"\"  | `\"memphis-key_client.pem\"` |\n| memphis.tls.ca | **\\*Optional\\***  \u003cbr\u003eFor encrypted client-memphis communication.  \u003cbr\u003eCA file to use. SSL. | \"\"  | `\"rootCA.pem\"` |\n| websocket.enabled | **\\*Optional\\*** Memphis GUI using websockets for live rendering. | \"\"  | `\"false\"` |\n| websocket.port | Memphis GUI using websockets for live rendering. The port can be configured | \"7770\" | \"77777\" |\n| websocket.host | Websocket host can be handled on separate LB/DNS. | \"localhost\" | \"ws.example.com\" |\n| websocket.noTLS | Websocket can be configured with tls, default is noTLS. | \"true\" | \"false\" |\n| websocket.tls.secret.name | **\\*Optional\\*** Memphis GUI using websockets for live rendering.  \u003cbr\u003eK8S secret name for the certs | \"\"  | `\"memphis-ws-tls-secret\"` |\n| websocket.tls.cert | **\\*Optional\\***  \u003cbr\u003eMemphis GUI using websockets for live rendering.  \u003cbr\u003e.pem file to use | \"\"  | `\"memphis_local.pem\"` |\n| websocket.tls.key | **\\*Optional\\***  \u003cbr\u003eMemphis GUI using websockets for live rendering.  \u003cbr\u003ekey file | \"\"  | `\"memphis-key_local.pem\"` |\n| metadata.postgresql.username | **\\*Optional\\***  \u003cbr\u003eUsername for postgres db | \"postgres\" | \"postgres\" |\n| metadata.postgresql.existingSecret |  **\\*Optional\\*** \u003cbr\u003e An ability to provide predefined secret for metadata PostgreSQL credentials | \"\" | \"metadata-creds.yaml\" |\n| metadata.pgpool.existingSecret |  **\\*Optional\\*** \u003cbr\u003e An ability to provide predefined secret for metadata PG credentials | \"\" | \"metadata-creds.yaml\" |\n| metadata.pgpool.tls.enabled | **\\*Optional\\***  \u003cbr\u003eEnabling TLS-based communication with PG | \"false\" | \"false\" |\n| metadata.pgpool.tls.certificatesSecret | **\\*Optional\\***  \u003cbr\u003ePG TLS cert secret to be used | \"\"  | \"tls-secret\" |\n| metadata.pgpool.tls.certFilename | **\\*Optional\\***  \u003cbr\u003ePG TLS cert file to be used | \"\"  | \"tls.crt\" |\n| metadata.pgpool.tls.certKeyFilename | **\\*Optional\\***  \u003cbr\u003ePG TLS key to be used | \"\"  | \"tls.key\" |\n| metadata.pgpool.tls.certCAFilename | **\\*Optional\\***  \u003cbr\u003ePG TLS cert CA to be used | \"\"  | \"ca.crt\" |\n| metadata.external.enabled | **\\*Optional\\***  \u003cbr\u003eFor using external PG instead of deploying dedicated one for Memphis | \"false\" | \"true\" |\n| metadata.external.dbTlsMutual | **\\*Optional\\***  \u003cbr\u003eExternal PG TLS-basec communication | \"true\" | \"true\" |\n| metadata.external.dbName | **\\*Optional\\***  \u003cbr\u003eExternal PG db name | \"\"  | \"memphis\" |\n| metadata.external.dbHost | **\\*Optional\\***  \u003cbr\u003eExternal PG db hostname | \"\"  | \"metadata.example.url\" |\n| metadata.external.dbPort | **\\*Optional\\***  \u003cbr\u003eExternal PG db port | \"\"  | 5432 |\n| metadata.external.dbUser | **\\*Optional\\***  \u003cbr\u003eExternal PG db user | \"\"  | \"postgres\" |\n| metadata.external.dbPass | **\\*Optional\\***  \u003cbr\u003eExternal PG db password | \"\"  | \"12345678\" |\n| metadata.external.secret.enabled | **\\*Optional\\***  \u003cbr\u003eEnable an option to use secret for password store | \"false\" | \"true\" |\n| metadata.external.secret.name | **\\*Optional\\***  \u003cbr\u003eSecret name | \"\"  | \"metadata-secret\" |\n| metadata.external.secret.dbPass_key | **\\*Optional\\***  \u003cbr\u003eName of the key in the secret | \"\"  | \"dbPass\" |\n| restGateway.enabled | **\\*Optional\\***  \u003cbr\u003eMemphis Rest Gateway can be disabled if not in use | \"true\" | \"false\" |\n| restGateway.jwtSecret | **\\*Optional\\***  \u003cbr\u003eManual Jwt Token configurtion | \"\"  | \"\"  |\n| restGateway.refreshJwtSecret | **\\*Optional\\***  \u003cbr\u003eManual Refresh Jwt Token configurtion | \"\"  | \"\"  |\n| auth.enabled | **\\*Optional\\***  \u003cbr\u003eEnable using predefined parameters | \"false\"  | \"true\"  |\n| auth.enabled.mgmt | **\\*Optional\\***  \u003cbr\u003eManagement users that will be created at first deployment | \"\"  | \"\"  |\n| auth.enabled.client | **\\*Optional\\***  \u003cbr\u003eClient users that will be created at first deployment | \"\"  | \"\"  |\nHere is how to run an installation command with additional options -\u0026#x20;\n\n```\nhelm install memphis --set cluster.replicas=3,memphis.creds.rootPwd=rootpassword\" memphis/memphis --create-namespace --namespace memphis\n```\n\n### Deployed pods\n\n- **memphis.** Memphis broker.\n- **memphis-rest-gateway.** Memphis REST Gateway.\n- **memphis-metadata.** Metadata store.\n- **memphis-metadata-coordinator.** Metadata coordinator\n\nFor more information on each component, please head to the [architecture section](../../memphis/architecture.md#key-components).\n\n## Deploy Memphis with TLS (encrypted communication via SSL)\n\n### 0. Optional: Create self-signed certificates\n\na) Generate a self-signed certificate using `mkcert`\n\n```bash\n$ mkcert -client \\\n-cert-file memphis_client.pem \\\n-key-file memphis-key_client.pem  \\\n\"127.0.0.1\" \"localhost\" \"*.memphis.dev\" ::1 \\\nemail@localhost valera@Valeras-MBP-2.lan\n```\n\nb) Find the `rootCA`\n\n```\n$ mkcert -CAROOT\n```\n\nc) Create self-signed certificates for client\n\n```bash\n$ mkcert -client -cert-file client.pem -key-file key-client.pem  localhost ::1 \n```\n\n### 1. Create namespace + secret for the TLS certs\n\na) Create a dedicated namespace for memphis\n\n```bash\nkubectl create namespace memphis\n```\n\nb) Create a k8s secret with the required certs\n\n\n```bash\nkubectl create secret generic memphis-client-tls-secret \\\n--from-file=memphis_client.pem \\\n--from-file=memphis-key_client.pem \\\n--from-file=rootCA.pem -n memphis\n```\n\n```yaml\ntls:\n  secret:\n    name: memphis-client-tls-secret\n  ca: \"rootCA.pem\"\n  cert: \"memphis_client.pem\"\n  key: \"memphis-key_client.pem\"\n```\n\n### 2. Deploy Memphis with the generated certificate\n\n```bash\nhelm install memphis memphis \\\n--create-namespace --namespace memphis --wait \\\n--set \\\nglobal.cluster.enabled=\"true\",\\\nmemphis.tls.verify=\"true\",\\\nmemphis.tls.cert=\"memphis_client.pem\",\\\nmemphis.tls.key=\"memphis-key_client.pem\",\\\nmemphis.tls.secret.name=\"memphis-client-tls-secret\",\\\nmemphis.tls.ca=\"rootCA.pem\"\n```\n\n## Upgrade existing deployment\n\n### For adding TLS support\n\n1. Create a k8s secret with the provided TLS certs\n\n```\nkubectl create secret generic memphis-client-tls-secret \\\n--from-file=memphis_client.pem \\\n--from-file=memphis-key_client.pem \\\n--from-file=rootCA.pem -n memphis\n```\n\n2. Upgrade Memphis to use the TLS certs\n\n```bash\nhelm upgrade memphis memphis -n memphis --reuse-values \\\n--set \\\nmemphis.tls.verify=\"true\",\\\nmemphis.tls.cert=\"memphis_client.pem\",\\\nmemphis.tls.key=\"memphis-key_client.pem\",\\\nmemphis.tls.secret.name=\"tls-client-secret\",\\\nmemphis.tls.ca=\"rootCA.pem\"\n```\n\n## Deployment diagram\n\n![Memphis Architecture (1)](https://user-images.githubusercontent.com/70286779/229374721-963cd3e6-e425-44cd-8467-233e6fc5e680.jpeg)\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperstreamlabs%2Fmemphis-k8s","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsuperstreamlabs%2Fmemphis-k8s","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsuperstreamlabs%2Fmemphis-k8s/lists"}