Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/superstreamlabs/memphis
Memphis.dev is a highly scalable and effortless data streaming platform
https://github.com/superstreamlabs/memphis
data data-engineering data-pipeline data-stream-processing data-streaming enrichment golang kubernetes message-broker message-bus message-queue messaging-queue microservices schema-registry
Last synced: about 1 month ago
JSON representation
Memphis.dev is a highly scalable and effortless data streaming platform
- Host: GitHub
- URL: https://github.com/superstreamlabs/memphis
- Owner: superstreamlabs
- License: other
- Created: 2022-02-01T12:07:05.000Z (almost 3 years ago)
- Default Branch: master
- Last Pushed: 2024-05-22T12:25:35.000Z (6 months ago)
- Last Synced: 2024-05-22T12:51:15.413Z (6 months ago)
- Topics: data, data-engineering, data-pipeline, data-stream-processing, data-streaming, enrichment, golang, kubernetes, message-broker, message-bus, message-queue, messaging-queue, microservices, schema-registry
- Language: Go
- Homepage: https://docs.memphis.dev
- Size: 465 MB
- Stars: 3,169
- Watchers: 33
- Forks: 210
- Open Issues: 109
-
Metadata Files:
- Readme: README.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: .github/CODEOWNERS
Awesome Lists containing this project
- awesome-repositories - superstreamlabs/memphis - Memphis.dev is a highly scalable and effortless data streaming platform (Go)
- awesome-ccamel - superstreamlabs/memphis - Memphis.dev is a highly scalable and effortless data streaming platform (Go)
README
Please pay attention that Memphis.dev is no longer supported officially by the Superstream team (formerly Memphis.dev) and was released to the public.
**[Memphis.dev](https://memphis.dev)** Is The First Data Streaming Platform Designed For Backend Developers
To Build Event-driven And Real-time Features Faster Than Ever.
##  About
Before Memphis came along, handling ingestion and processing of events on a large scale took months to adopt and was a capability reserved for the top 20% of mega-companies. Now, Memphis opens the door for the other 80% to unleash their event and data streaming superpowers quickly, easily, and with great cost-effectiveness.
## 🚀 Getting Started
[Tutorials](https://docs.memphis.dev/memphis/getting-started/tutorials) | [Videos](https://www.youtube.com/playlist?list=PL_7iYjqhtXpWpZT2U0zDYo2eGOoGmg2mm)
#### ☸ Kubernetes
```shell
helm repo add memphis https://k8s.memphis.dev/charts/ --force-update && \
helm install my-memphis memphis/memphis --create-namespace --namespace memphis
```
#### 🐳 Docker Compose
```shell
curl -s https://memphisdev.github.io/memphis-docker/docker-compose.yml -o docker-compose.yml && \
docker compose -f docker-compose.yml -p memphis up
```
## ✨ Key Features [v1.4.4](https://docs.memphis.dev/memphis/release-notes/releases/v1.4.4-latest)
 Production-ready message broker in under 3 minutes
 Easy-to-use UI, CLI, and SDKs
 Data-level observability
 Dead-Letter Queue with automatic message retransmit
 Schemaverse - Embedded schema management for produced data (Protobuf/JSON/GraphQL/Avro)
 Functions for real-time processing
 Graph visualization
 Storage tiering
 SDKs: Node.JS, Go, Python, Typescript, NestJS, REST, .NET, Kotlin
 Kubernetes-native
 Community driven
End-to-end Observability
"Feel" your producers, consumers, and records
Quickly analyze system health using a graph overview
Never lose a message with automatic dead-letter
Save up 96% storage costs with Storage tiering
Increase data quality with schemas
## Public case studies
- [Dstny - Building the next-gen in-house communication using Memphis.dev](https://memphis.dev/blog/how-dstny-building-the-future-of-in-house-communication-using-memphis-dev/)
- [Gastromatic - Synchronizing data using Memphis.dev](https://medium.com/gastromatic/synchronizing-data-using-memphis-dev-a-case-study-2e6e9a7b5512)
- [KELA - Real-time cyber threats identification](https://memphis.dev/blog/how-kela-is-using-memphis-dev-for-real-time-cyber-threats-identification/)
- [Handling millions of discord messages](https://memphis.dev/blog/how-cactusfire-handles-millions-of-daily-discord-messages-using-memphis-dev/)
## Network diagram
## SDKs supported features
Feature | Go | Python | JS | .NET | Java | Rust
------------- | ------------- | ------------- | ------------- | ------------- | ------------- | -------------
Connection | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Disconnection | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Create a station | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Destroy a station | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Retention | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Retention values | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Storage types | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Create a new schema | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Enforce a schema Protobuf | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Enforce a schema Json | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :construction: (WIP)
Enforce a schema GraphQL | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Enforce a schema Avro | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Detach a schema | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Produce | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Add headers | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Async produce | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Message ID | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | ? | :white_check_mark:
Destroy a producer | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | Partial | :white_check_mark:
Consume | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Context to message handler | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | Not Applicable
Ack a message | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Nack a Message | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Send a message to dead-letter | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Get message time sent | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
Fetch | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x:
Message delay | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Get Headers | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Get message sequence number | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Destroying a Consumer | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark:
Check if broker is connected | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark:
Consumer prefetch | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark:
Schema deserialization | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x:
## 👉 Use-cases
- Async task management
- Real-time streaming pipelines
- Data ingestion
- Cloud Messaging
- Services (microservices, service mesh)
- Event/Data Streaming (observability, analytics, ML/AI)
- Queuing
- N:N communication patterns
- Ingest Grafana Loki logs at scale
## Support 🙋♂️🤝
### Have any questions ❓
We welcome you to our discord server with your questions, doubts and feedback.
### Create a bug 🐞 report
If you see an error message or run into an issue, please [create bug report](https://github.com/memphisdev/memphis/issues/new?assignees=&labels=type%3A%20bug&template=bug_report.md&title=). This effort is valued and it will help all Memphis{dev} users.
### Submit a feature 💡 request
If you have an idea, or you think that we're missing a capability that would make development easier and more robust, please [Submit feature request](https://github.com/memphisdev/memphis/issues/new?assignees=&labels=type%3A%20feature%20request).
If an issue❗with similar feature request already exists, don't forget to leave a "+1".
If you add some more information such as your thoughts and vision about the feature, your comments will be embraced warmly :)
## Contributing
Memphis.dev is an open-source project.
We are committed to a fully transparent development process and appreciate highly any contributions.
Whether you are helping us fix bugs, proposing new features, improving our documentation or spreading the word - we would love to have you as part of the Memphis.dev community.
Please refer to our [Contribution Guidelines](./CONTRIBUTING.md) and [Code of Conduct](./CODE_OF_CONDUCT.md).
## Contributors ✨
Thanks goes to these wonderful people ❤:
## License 📃
Memphis is open-sourced and operates under the "Memphis Business Source License 1.0" license
Built out of Apache 2.0, the main difference between the licenses is:
"You may make use of the Licensed Work (i) only as part of your own product or service, provided it is not a message broker or a message queue product or service; and (ii) provided that you do not use, provide, distribute, or make available the Licensed Work as a Service. A “Service” is a commercial offering, product, hosted, or managed service, that allows third parties (other than your own employees and contractors acting on your behalf) to access and/or use the Licensed Work or a substantial set of the features or functionality of the Licensed Work to third parties as a software-as-a-service, platform-as-a-service, infrastructure-as-a-service or other similar services that compete with Licensor products or services."
Please check out [License](./LICENSE) to read the full text.