An open API service indexing awesome lists of open source software.

awesome-system-design

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
https://github.com/madd86/awesome-system-design

Last synced: 19 days ago
JSON representation

  • Advanced

    • Distributed Computing - Wikipedia article broadening the view of distributed system design.
    • Fallacies of Distributed Computing - Wikipedia article introducing the topic of fallacies of distributed computing and its effects.
    • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
    • CAP Theorem - IBM Article about CAP Theorem, Microservices and NoSQL DBs.
    • Pattern: Microservice Architecture - Good article talking about Microservice architecture as well as its drawbacks.
    • Top 10 Secure Coding Practices - Brief article talking about good practices for code securities.
    • Scalable Web Architecture and Distributed Systems - Good article about distributed systems as well as some of the potential tools.
    • Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services - [Paid 💵] - Book that talks about disitributed systems as well as lightly demonstrating some code of what it looks like.
    • Designing Data Intensive Applications - [Paid 💵] - Goes in depth to explain various resources we use when working with distributed systems, as well as how it came to be and what problems it aims to solve.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • Distributed Systems (3rd Edition) - [Free 👍] - Great overview of and in-depth introduction to distributed systems. Recommended for intermediate level readers.
    • The evolution of Reddit Architecture - Overview of how Reddit system design scaled.
    • CSE138 Distributed Systems by UCSC - Undergraduate course on distributed systems from UCSC (2020).
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • Monolith to Microservices - [Free 👍] - Written by the same author as the one above, this book will cover the migration from Monolith to Microservices, it's recommended you start by the previous book.
    • CSE138 Distributed Systems by UCSC - Undergraduate course on distributed systems from UCSC (2020).
    • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
    • Building Microservices - [Free 👍] - Awesome book that talks about designing sytem architecture with microservices in depth, includes most relevant topics in this regard.
  • Distributed File Systems

    • Graph

      • HDFS - Hadoop File System is a a widely popular choice among its big data competitors, providing high throughput access.
      • CephFS - Unified, distributed storage system.
      • GlusterFS - Scale-out NAS file system.
      • MooseFS - POSIX-compliant distributed file system.
  • Hadoop Ecosystem

    • Dashboard

      • Ambari - Dashboard that integrates most of hadoop related technologies for easy management and executions.
    • Data Ingestion

      • Apache Kafka - Widely popular message broker with low latency for data streaming.
      • Sqoop - Efficiently transfer data between Hadoop and structured datastores such as relational databases.
      • Flume - Distributed, highly available and efficient in collecting, aggregating and moving large amounts of log data.
    • DB

      • HBase - [3.6k ⭐] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project.
    • Processing

      • Tez - Solves a similar problem to Spark and MapReduce, it's more efficient than MapReduce because it calculates the most efficient way of doing it.
      • Map Reduce - MapReduce, as the name implies, maps data and reduce the results.
      • Spark - Powerful data processing to not only process data like Tez (and MapReduce), it can process streams of data in real time, apply regression analysis algorithms in ML and much more.
      • Apex - \*Retired project, it's a YARN-native platform that unifies stream and batch processing.
    • Query

      • Hive - Query hadoop stored data in SQL.
      • Pig - Scriping language that looks like SQL to query hadoop data.
    • Resource Management

      • YARN - 'Yet Another Resource Negotiator', works like a kernel to manage computer resources across the clusters.
    • Workflow Scheduler

      • Oozie - Create workflows in xml to execute jobs (from other hadoop-ecosystem applications) in steps, allows for parallel execution as well.
  • Introduction / Interviews

  • Load Balancers

    • Cloud

      • Amazon Elastic Load Balancing - Popular choice for amazon customers, supports lambda functions, highly scalable.
      • Google Load Balancing - Popular choice for google customers, comes with auto-scaling feature, very fast, has intergrated CDN.
      • Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.
      • DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.
      • Azure Load Balancing - Popular choice for Microsoft's Azure customers. Supports internal and external traffics, ipv6, monitorining and the standard load balancing set of features.
      • Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.
    • Hardware

      • F5 - Robust hardware load balancer option, supporting multiple protocols (IP, TCP, FTP, UDP, HTTP).
      • TP-Link - Cheaper alternative that works as a load balancer.
      • Barracuda - One of the top choices for load balancing when it comes to in-house servers. Top security measures built in, comprehensive reports and monitoring outbound traffic for data loss prevention.
    • Open Source Software

      • SeeSaw - [5.1k ⭐] - Used by Google, developed in Go, linux-based virtual load balancer server.
      • HAProxy - Widely popular option, provides high-availability, proxy, TCP/HTTP load balancing. Used by Reddit, Imgur, MaxCDN, GitHub, AirBNB.
      • Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.
      • Neutrino - Used by eBay, built with Scala and Netty. Supports round-robin and least-connection algorithms.
      • Openresty - Nginx + Lua, perfect combination.
      • Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.
  • Message Broker

    • Graph

      • Amazon MQ - Open source message broker from Amazon.
      • Apache ActiveMQ - It's a multi-protocol, java based messaging server.
      • RabbitMQ - Widely popular lightweight
      • IronMQ - Very fast and highly scalable messaging broker. (not open source)
      • Apache Pulsar - Created by yahoo, also highly scalable, low latency, geo-replication and multi-tenacy.
      • Azure Service Bus - A fully managed enterprise integration message broker.
      • Kestrel - Written in Scala and speaks the memcached protocol. It works much like Kafka.
  • NoSQL

    • Cache (Key-Value)

      • Couchbase - Inspired by memcached, adding features such as replication and persistance.
      • Oracle Coherence - [126 ⭐] - High scaling, low latency in-memory caching.
      • Memcached - [10.2k ⭐] - One of the first in-memory caching database, high performing and multi-threaded.
      • Apache Ignite - [3.3k ⭐] - In memory caching with ACID properties.
      • Redis - [44k ⭐] - Widely used in-memory caching database with many added features such as persistent storage and supporting strings, lists, sets, hashses, streams, bitmaps, etc.
    • Document Store

      • MongoDB - One of the most popular 'NoSQL' database for general purpose.
      • CouchDB - [4.6k ⭐] - ACID compliant NoSQL document-store DB, provides a RESTful HTTP API for reading and updating database documents.
      • RethinkDB - [23.8k ⭐] - Document-store DB.
      • ElasticSearch - [49.9k ⭐] - Widely popular 'NoSQL' database for fast and scalable search engines.
    • Graph

      • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
      • Amazon Neptune - Fast, reliable and fully managed graph database service.
      • ArangoDB - [10k ⭐] - Flexible database for documents, key-value, graphs. Uses its own query language, AQL.
      • Neo4j - [7.9k ⭐] - Good support for a graph db, ACID compliant and flexible.
    • Store (Key-Value)

      • Apple FoundationDB - [10k ⭐] - Multi-model (many data types in a single database), ACID key-value store. Easily scalable and fault tolerant.
      • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
    • Wide Column Store

      • Amazon DynamoDB - Key-Value and Document database, highly performant, scalable and secure.
      • Google Bigtable - Scalable and performant 'NoSQL' database for large analytical and operational workload.
      • Scylla - [4.9k ⭐] - 'NoSQL' data store using seastar framework, compatible with Cassandra.
  • Relational Database Management System

    • MariaDB - MariaDB is a fork of MySQL server.
    • MySQL - Widely used relational database.
    • PostgresSQL - Relational database that has been gaining popularity.
    • SQLite - Another widely used database that is built into all mobile phones and most computers.
    • Sql Server - Widely used relational database.
  • Resource Management

    • Graph

      • Kubernetes - Highly popular way to deploy, manage and automatically scale a cluster of containers on bare-metal or virtual servers.
  • REST Framework

    • Resource Management

      • ASP.NET Core MVC - A rich framework for building web apps and APIs using the Model-View-Controller design pattern in C# or F#. Number 6 on [TechEmpower Composite Benchmarks](https://www.techempower.com/benchmarks/#section=data-r19&hw=ph&test=composite) for web frameworks.
      • Gin - [40.6k ⭐] - Blazingly fast microservice framework using Golang, high throughput capacity.
      • Phoenix - [15.5k ⭐] - Distributed processing, easily scalable, support for channels and live chat. This framework - written in Elixir, uses BEAM and Erlang, very efficient for large scale systems and supports high throughput.
      • Express.js - [49.6k ⭐] - Fast node.js rest api that can perform well under many scenarios.
      • Rails - [46.2k ⭐] - Written in Ruby, Rails delivers quick apis from prototype to production in an efficient manner.
      • Play Framework - [11.6k ⭐] - Very fast, high throughput framework written in Scala/Java that is RESTful by default.
      • Flask - [51.6k ⭐] - A lightweight Python Microframework for fast prototyping and production.
      • FastAPI - [22.7k ⭐] - A lightweight Python Microframework inspired in Flask but more modern, using Python async.
      • Django REST - [18.4k ⭐] - Written in Python, Django Rest is a powerful and flexible REST API. The efficiency and time to market resembles Rails.
      • Fastify - [15.4k ⭐] - A Node.js web framework highly focused on providing the best developer experience with the least overhead and a powerful plugin architecture.
  • Stream Processing

    • Graph

      • Apache Flink - Based on the concept of streams and transofrmations. Uses maven, handles batch tasks as data streams with finite boundaries. Low latency, high throughput.
      • Amazon Kinesis Streams - Durable, scalable, real-tme service. Collects gigabytes of data per second from hundreds of thousands of sources, including database event streams, website clickstreams, financial transactions, etc.
      • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.
      • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.