An open API service indexing awesome lists of open source software.

awesome-system-design

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
https://github.com/madd86/awesome-system-design

Last synced: about 1 month ago
JSON representation

Advanced

  • Distributed Computing - Wikipedia article broadening the view of distributed system design.
  • Fallacies of Distributed Computing - Wikipedia article introducing the topic of fallacies of distributed computing and its effects.
  • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
  • CAP Theorem - IBM Article about CAP Theorem, Microservices and NoSQL DBs.
  • Pattern: Microservice Architecture - Good article talking about Microservice architecture as well as its drawbacks.
  • Top 10 Secure Coding Practices - Brief article talking about good practices for code securities.
  • Scalable Web Architecture and Distributed Systems - Good article about distributed systems as well as some of the potential tools.
  • Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services - [Paid 💵] - Book that talks about disitributed systems as well as lightly demonstrating some code of what it looks like.
  • Designing Data Intensive Applications - [Paid 💵] - Goes in depth to explain various resources we use when working with distributed systems, as well as how it came to be and what problems it aims to solve.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • Distributed Systems (3rd Edition) - [Free 👍] - Great overview of and in-depth introduction to distributed systems. Recommended for intermediate level readers.
  • The evolution of Reddit Architecture - Overview of how Reddit system design scaled.
  • CSE138 Distributed Systems by UCSC - Undergraduate course on distributed systems from UCSC (2020).
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
  • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
  • Monolith to Microservices - [Free 👍] - Written by the same author as the one above, this book will cover the migration from Monolith to Microservices, it's recommended you start by the previous book.
  • CSE138 Distributed Systems by UCSC - Undergraduate course on distributed systems from UCSC (2020).
  • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
  • Building Microservices - [Free 👍] - Awesome book that talks about designing sytem architecture with microservices in depth, includes most relevant topics in this regard.

Distributed File Systems

Graph

  • HDFS - Hadoop File System is a a widely popular choice among its big data competitors, providing high throughput access.
  • CephFS - Unified, distributed storage system.
  • GlusterFS - Scale-out NAS file system.
  • MooseFS - POSIX-compliant distributed file system.

Hadoop Ecosystem

Dashboard

  • Ambari - Dashboard that integrates most of hadoop related technologies for easy management and executions.

Data Ingestion

  • Apache Kafka - Widely popular message broker with low latency for data streaming.
  • Sqoop - Efficiently transfer data between Hadoop and structured datastores such as relational databases.
  • Flume - Distributed, highly available and efficient in collecting, aggregating and moving large amounts of log data.

DB

  • HBase - [3.6k ⭐] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project.

Processing

  • Tez - Solves a similar problem to Spark and MapReduce, it's more efficient than MapReduce because it calculates the most efficient way of doing it.
  • Map Reduce - MapReduce, as the name implies, maps data and reduce the results.
  • Spark - Powerful data processing to not only process data like Tez (and MapReduce), it can process streams of data in real time, apply regression analysis algorithms in ML and much more.
  • Apex - *Retired project, it's a YARN-native platform that unifies stream and batch processing.

Query

  • Hive - Query hadoop stored data in SQL.
  • Pig - Scriping language that looks like SQL to query hadoop data.

Resource Management

  • YARN - 'Yet Another Resource Negotiator', works like a kernel to manage computer resources across the clusters.

Workflow Scheduler

  • Oozie - Create workflows in xml to execute jobs (from other hadoop-ecosystem applications) in steps, allows for parallel execution as well.

Introduction / Interviews

Load Balancers

Cloud

  • Amazon Elastic Load Balancing - Popular choice for amazon customers, supports lambda functions, highly scalable.
  • Google Load Balancing - Popular choice for google customers, comes with auto-scaling feature, very fast, has intergrated CDN.
  • Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.
  • DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.
  • Azure Load Balancing - Popular choice for Microsoft's Azure customers. Supports internal and external traffics, ipv6, monitorining and the standard load balancing set of features.
  • Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.

Hardware

  • F5 - Robust hardware load balancer option, supporting multiple protocols (IP, TCP, FTP, UDP, HTTP).
  • TP-Link - Cheaper alternative that works as a load balancer.
  • Barracuda - One of the top choices for load balancing when it comes to in-house servers. Top security measures built in, comprehensive reports and monitoring outbound traffic for data loss prevention.

Open Source Software

  • SeeSaw - [5.1k ⭐] - Used by Google, developed in Go, linux-based virtual load balancer server.
  • HAProxy - Widely popular option, provides high-availability, proxy, TCP/HTTP load balancing. Used by Reddit, Imgur, MaxCDN, GitHub, AirBNB.
  • Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.
  • Neutrino - Used by eBay, built with Scala and Netty. Supports round-robin and least-connection algorithms.
  • Openresty - Nginx + Lua, perfect combination.
  • Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.

Message Broker

Graph

  • Amazon MQ - Open source message broker from Amazon.
  • Apache ActiveMQ - It's a multi-protocol, java based messaging server.
  • RabbitMQ - Widely popular lightweight
  • IronMQ - Very fast and highly scalable messaging broker. (not open source)
  • Apache Pulsar - Created by yahoo, also highly scalable, low latency, geo-replication and multi-tenacy.
  • Azure Service Bus - A fully managed enterprise integration message broker.
  • Kestrel - Written in Scala and speaks the memcached protocol. It works much like Kafka.

NoSQL

Cache (Key-Value)

  • Couchbase - Inspired by memcached, adding features such as replication and persistance.
  • Oracle Coherence - [126 ⭐] - High scaling, low latency in-memory caching.
  • Memcached - [10.2k ⭐] - One of the first in-memory caching database, high performing and multi-threaded.
  • Apache Ignite - [3.3k ⭐] - In memory caching with ACID properties.
  • Redis - [44k ⭐] - Widely used in-memory caching database with many added features such as persistent storage and supporting strings, lists, sets, hashses, streams, bitmaps, etc.

Document Store

  • MongoDB - One of the most popular 'NoSQL' database for general purpose.
  • CouchDB - [4.6k ⭐] - ACID compliant NoSQL document-store DB, provides a RESTful HTTP API for reading and updating database documents.
  • RethinkDB - [23.8k ⭐] - Document-store DB.
  • ElasticSearch - [49.9k ⭐] - Widely popular 'NoSQL' database for fast and scalable search engines.

Graph

  • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
  • Amazon Neptune - Fast, reliable and fully managed graph database service.
  • ArangoDB - [10k ⭐] - Flexible database for documents, key-value, graphs. Uses its own query language, AQL.
  • Neo4j - [7.9k ⭐] - Good support for a graph db, ACID compliant and flexible.

Store (Key-Value)

  • Apple FoundationDB - [10k ⭐] - Multi-model (many data types in a single database), ACID key-value store. Easily scalable and fault tolerant.
  • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.

Wide Column Store

  • Amazon DynamoDB - Key-Value and Document database, highly performant, scalable and secure.
  • Google Bigtable - Scalable and performant 'NoSQL' database for large analytical and operational workload.
  • Scylla - [4.9k ⭐] - 'NoSQL' data store using seastar framework, compatible with Cassandra.

Relational Database Management System

  • MariaDB - MariaDB is a fork of MySQL server.
  • MySQL - Widely used relational database.
  • PostgresSQL - Relational database that has been gaining popularity.
  • SQLite - Another widely used database that is built into all mobile phones and most computers.
  • Sql Server - Widely used relational database.

Resource Management

Graph

  • Kubernetes - Highly popular way to deploy, manage and automatically scale a cluster of containers on bare-metal or virtual servers.

REST Framework

Resource Management

  • ASP.NET Core MVC - A rich framework for building web apps and APIs using the Model-View-Controller design pattern in C# or F#. Number 6 on TechEmpower Composite Benchmarks for web frameworks.
  • Gin - [40.6k ⭐] - Blazingly fast microservice framework using Golang, high throughput capacity.
  • Phoenix - [15.5k ⭐] - Distributed processing, easily scalable, support for channels and live chat. This framework - written in Elixir, uses BEAM and Erlang, very efficient for large scale systems and supports high throughput.
  • Express.js - [49.6k ⭐] - Fast node.js rest api that can perform well under many scenarios.
  • Rails - [46.2k ⭐] - Written in Ruby, Rails delivers quick apis from prototype to production in an efficient manner.
  • Play Framework - [11.6k ⭐] - Very fast, high throughput framework written in Scala/Java that is RESTful by default.
  • Flask - [51.6k ⭐] - A lightweight Python Microframework for fast prototyping and production.
  • FastAPI - [22.7k ⭐] - A lightweight Python Microframework inspired in Flask but more modern, using Python async.
  • Django REST - [18.4k ⭐] - Written in Python, Django Rest is a powerful and flexible REST API. The efficiency and time to market resembles Rails.
  • Fastify - [15.4k ⭐] - A Node.js web framework highly focused on providing the best developer experience with the least overhead and a powerful plugin architecture.

Stream Processing

Graph

  • Apache Flink - Based on the concept of streams and transofrmations. Uses maven, handles batch tasks as data streams with finite boundaries. Low latency, high throughput.
  • Amazon Kinesis Streams - Durable, scalable, real-tme service. Collects gigabytes of data per second from hundreds of thousands of sources, including database event streams, website clickstreams, financial transactions, etc.
  • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.
  • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.