awesome-distributed-system-projects
🚀 List of distributed system projects for inspiration and learning to build distributed services from real world examples
https://github.com/roma-glushko/awesome-distributed-system-projects
Last synced: 27 days ago
JSON representation
Components
Vectors
- Dragonboat - a high performance multi-group Raft consensus library in pure Go
- Golimit - Uber ringpop based distributed and decentralized rate limiter
- Tenacity - general-purpose retrying library
- ex_hash_ring - pure Elixir consistent hash ring implementation based on the excellent C hash-ring lib
- raft - Raft consensus implementation
- NuRaft - Raft implementation derived from the cornerstone project
- Hyx - Lightweight fault tolerance primitives for your resilient and modern Python microservices
- Migdalor - a Kubernetes native peer discovery for Python asyncio nodes
- skiplist - a Golang implementation of the skiplist data structure
- Waltz - a quorum-based distributed write-ahead log for replicating transactions
Databases
Analytical Databases
- BaikalDB is a distributed HTAP MySQL-compatible database designed for petabytes scale
- AresDB - a GPU-powered real-time analytics storage and query engine
Column Databases
- Apache Cassandra - a highly-scalable partitioned row store. Rows are organized into tables with a required primary key
- scylladb - a real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB
- FrostDB - an embeddable wide-column columnar database written in Go
Graph Databases
- neo4j - Graph Database
- edgedb - a graph-relational database
- nebula - a distributed, fast open-source graph database featuring horizontal scalability and high availability
- EliasDB - a graph-based lightweight database
Key-Value Databases
- memcached - a high performance multithreaded event-based key/value cache store intended to be used in a distributed system
- redis - an in-memory database with various value types that persists on disk
- TiKV - a distributed transactional key-value database, originally created to complement TiDB
- leveldb - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values
- goleveldb - a LevelDB implemented in Golang
- summitdb - an in-memory, NoSQL key/value database. It persists to disk, uses the Raft consensus algorithm, is ACID compliant, and built on a transactional and strongly-consistent model - defined JS scripting
- pupdb - a simple file-based key-value database
- pickledb - an open source key-value store using Python's json module
- KeyDB - a faster drop-in multithreaded alternative to Redis
- Dragonfly - an in-memory data store fully compatible with Redis and Memcache and designed using modern algorithms
- BuntDB - a low-level, in-memory, key/value store in pure Go.
- ConstDB - a redis-like cache store that implements CRDTs and active-active replications.
- GhostDB - a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale
- Hive - a lightweight and blazing fast key-value database written in pure Dart. Inspired by Bitcask
- PumpkinDB - an immutable Ordered Key-Value Database Engine
- FlashDB - a simple, in-memory, key/value store in pure Go.
- Lazer - a PHP flat file database based on JSON files
- Scribble - a tiny JSON database in Golang
- FlyDB - a high-performance KV storage engine based on bitcask paper supports redis protocol and the corresponding data structure
- Dice - an extremely simple Golang-based in-memory KV store that speaks Redis dialect
- Engula - a distributed key-value store, used as a cache, database, and storage engine
- rosedb - a fast, stable, and embedded NoSQL database based on bitcask, supports a variety of data structures such as string, list, hash, set, and sorted set
- BadgerDB - an embeddable, persistent and fast key-value (KV) database written in pure Go
NoSQL, Document Databases
- MongoDB - document database designed for ease of development and scaling
- FerretDB - an proxy, converting the MongoDB 6.0+ wire protocol queries to SQL - using PostgreSQL as a database engine
- tinydb - a lightweight document oriented database written in pure Python
- SleekDB - a simple flat file NoSQL like database implemented in PHP without any third-party dependencies that store data in plain JSON files
- BonsaiDB - an ACID, transactional KV or document dev-friendly database with configurable delayed on-disk data storing
- CloverDB - a lightweight document-oriented NoSQL database written in pure Golang
- LiteDB - NoSQL Document Store in a single data file
Permission Databases
- SpiceDB - a Google Zanzibar-inspired, database system for creating and managing security-critical application permissions
- Keto - a Google Zanzibar-inspired open source database, gRPC, REST APIs, newSQL, and an easy and granular permission language. Supports ACL, RBAC
Relational, SQL, NewSQL Databases
- CockroachDB - a distributed fault-tolerant SQL database built on a transactional and strongly-consistent key-value store
- YugabyteDB - a cloud native distributed SQL database for mission-critical applications
- RQLite - a lightweight, distributed relational database, which uses SQLite as its storage engine
- Kingbus - a distributed MySQL binlog store based on raft [Raft
- YDB is an open-source Distributed SQL Database that combines high availability and scalability with strict consistency and ACID transactions
- RadonDB - an open source, Cloud-native MySQL database for unlimited scalability and performance
Time Series
- VictoriaMetrics - fast, cost-effective monitoring solution and time series database
- influxdb - scalable datastore for metrics, events, and real-time analytics
- trino - fast distributed SQL query engine for big data analytics
- Apache Doris - an easy-to-use, high performance and unified analytics database
- FiloDB - a distributed, prometheus-compatible, real-time, in-memory, massively scalable, multi-schema time series / event / operational database
- tstorage is a lightweight local on-disk storage engine for time-series data with a straightforward API
- CnosDB is a high-performance, high-compression, and easy-to-use open-source distributed time-series database. Used in fields such as IoT, industrial internet, connected cars, and IT operations
- LinDB - a scalable, high performance, high availability distributed time series database
- CeresDB - a high-performance, distributed, cloud native time-series database
Vectors
- Qdrant - a vector similarity search engine and vector database
- milvus - an open-source vector database built to power embedding similarity search and AI applications
- Weaviate - an open source vector database that stores both objects and vectors
- tobias-mayer/vector-db - a simple vector database that can be used to search for similar vectors in logarithmic time
- DANNY - a decentralized vector database for building vector search applications
- DANNY - a decentralized vector database for building vector search applications
- requiemdb - a permanent storage for OTEL data
- Jocko - a Kafka/distributed commit log service in Go. [Serf + Raft
- oklog - a distributed and coordination-free log management system for big ol' clusters [Archived
- elasticell - a distributed HA Redis-compatible NoSQL database with strong consistency and reliability
- CouchDB - a highly available, partition tolerant, eventually consistent document database - master setups with automatic conflict detection.
- Apache HBase - a Hadoop database, a distributed, scalable, big data store
- Tair - a high-performance and high-availability distributed fast-access memory (MDB)/persistent (LDB) storage service
- immudb - an immutable database based on zero trust, Key/Value & SQL, tamperproof, data change history
- toydb - distributed SQL database in Rust, written as a learning project
- DB3 Network - a decentralized firebase firestore alternative
- ZODB - an ACID transactional object-oriented database
Data Processing
Vectors
File Systems
Vectors
- JuiceFS - Hadoop-compatible AWS S3-compatible high-performance POSIX file system
- GlusterFS - distributed storage that can scale to several petabytes
- GlusterFS - highly reliable, scalable and efficient distributed file system
- SeaweedFS - a simple Hadoop-compatible AWS S3-compatible distributed highly scalable distributed file system
Frameworks
Vectors
- gosiris - an actor framework for Golang
- cotyledon - a framework for defining long-running services - notify, rate limiting for worker spawning, and more.
- atomix - fully featured framework for building fault-tolerant distributed systems [REST + Raft
- orbit - virtual actor framework for building distributed systems
- hemera - A Node.js microservices toolkit for the NATS messaging system [RPC
- Tooz - centralizing the most common distributed primitives like group membership protocol, lock service and leader election by providing a coordination API helping developers to build distributed applications
- Nebula - powerful framework for building highly concurrent, distributed, and resilient message-driven applications
- Service Weaver - A framework that allows to write applications as modular binary and deploy it as a set of microservices
- Dapr - portable, serverless, event-driven runtime that works as a sidecar and makes it easy for developers to build resilient, stateless and stateful microservices
- ergo - port of Erlang/OTP approaches in Golang
Gateways
Vectors
- MLFLow Gateway - an LLM proxy
- Glide - an open reliable fast LLM/model gateway for rapid development of GenAI apps
- Traefik - a cloud-native app proxy
- Kong - a cloud-native feature-rich API gatewat
- Skipper - an HTTP router and reverse proxy for service composition
- janus - a lightweight API gateway and management platform
- Lura - ultra performance API gateway with middlewares
Locking
Vectors
OS
Vectors
Other Resources
Vectors
- Build Your Own Redis with C/C++
- Berkley CS186: Intro into Database Systems
- MIT 6.830: Database Systems
- awesome-scalability - Reading list for illustrating the patterns of scalable, reliable, and performant large-scale systems
- awesome-distributed-systems - curated list on awesome material on distributed systems
- awesome-database-learning - a list of learning materials to understand databases internals
- Writing a sqlite clone from scratch in C
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
- Berkley CS186: Intro into Database Systems
Queues
Vectors
- nsq - realtime fault tolerant distributed messaging platform designed to operate at scale, handling billions of messages per day [Raft + gRPC
- Sandglass - distributed, horizontally scalable, persistent, time ordered message queue
- dnpipes - distributed version of Unix named pipes comparable to AWS SQS
- GatewayWorker - distributed realtime messaging framework based on workerman
- ZeroMQ - abstraction of asynchronous message queues, multiple messaging patterns, message filtering (subscriptions), seamless access to multiple transport protocols and more
- Apache Pulsar - distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API
- Apache ActiveMQ - high performance Apache 2.0 licensed Message Broker
Schedulers
Vectors
Search Engines
Vectors
- ElasticSearch - distributed, RESTful search and analytics engine
- Apache Lucene - a high-performance, full featured text search engine library
- MeiliSearch - Lightning Fast, Ultra Relevant, and Typo-Tolerant Search Engine
- FlexSearch - memory-flexible full-text search library
- RiotSearch - distributed, Simple and efficient full text search engine
- Typesense - fast, typo tolerant, fuzzy search engine
- Sonic - fast, lightweight & schema-less search backend. An alternative to Elasticsearch that runs on a few MBs of RAM
Service Discovery
Vectors
Streaming
Vectors
- Kafka - a distributed, highly scalable, elastic, fault-tolerant, and secure event streaming platform
- faust - a distributed stream processing library that ports the ideas from Kafka Streams to Python
- Liftbridge - a lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system
- RisingWave - a distributed SQL database for stream processing, designed to reduce the complexity and cost of building real-time applications