Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

awesome-system-design

A curated list of awesome System Design (A.K.A. Distributed Systems) resources.
https://github.com/madd86/awesome-system-design

Last synced: 3 days ago
JSON representation

  • Advanced

    • Scalable Web Architecture and Distributed Systems - Good article about distributed systems as well as some of the potential tools.
    • Distributed Computing - Wikipedia article broadening the view of distributed system design.
    • Fallacies of Distributed Computing - Wikipedia article introducing the topic of fallacies of distributed computing and its effects.
    • Fallacies of Distributed Computing Explained - In depth explanation of the fallacies mentioned above.
    • CAP Theorem - IBM Article about CAP Theorem, Microservices and NoSQL DBs.
    • Pattern: Microservice Architecture - Good article talking about Microservice architecture as well as its drawbacks.
    • Taxonomy of Distributed Systems - 11 Page lecture classifying distributed systems and specifically why we need them.
    • Top 10 Secure Coding Practices - Brief article talking about good practices for code securities.
    • Scalable Web Architecture and Distributed Systems - Good article about distributed systems as well as some of the potential tools.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • Building Microservices - [Free 👍] - Awesome book that talks about designing sytem architecture with microservices in depth, includes most relevant topics in this regard.
    • Distributed Systems (3rd Edition) - [Free 👍] - Great overview of and in-depth introduction to distributed systems. Recommended for intermediate level readers.
    • The evolution of Reddit Architecture - Overview of how Reddit system design scaled.
    • CSE138 Distributed Systems by UCSC - Undergraduate course on distributed systems from UCSC (2020).
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services - [Paid 💵] - Book that talks about disitributed systems as well as lightly demonstrating some code of what it looks like.
    • Designing Data Intensive Applications - [Paid 💵] - Goes in depth to explain various resources we use when working with distributed systems, as well as how it came to be and what problems it aims to solve.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.
    • The System Design Manual - [Paid 💵] - Covers the core aspects of distributed systems, like: network fundamentals, the theory underpinning distributed systems, architectural patterns of scalable systems, stability patterns that harden systems against failures and operational best-practices on how to maintain large-scale systems with a small team.

  • NoSQL

    • Graph

      • Cosmos DB - Microsoft's globally distributed, multi-model database service. Eastically and independently scale throughput and storage. SQL, MongoDB, Cassandra, Tables, Gremlin, and Spark APIs.
      • Amazon Neptune - Fast, reliable and fully managed graph database service.

    • Cache (Key-Value)

      • Couchbase - Inspired by memcached, adding features such as replication and persistance.
      • Oracle Coherence - [126 ⭐] - High scaling, low latency in-memory caching.
      • Memcached - [10.2k ⭐] - One of the first in-memory caching database, high performing and multi-threaded.

    • Document Store

      • MongoDB - One of the most popular 'NoSQL' database for general purpose.

    • Wide Column Store

      • Amazon DynamoDB - Key-Value and Document database, highly performant, scalable and secure.
      • Google Bigtable - Scalable and performant 'NoSQL' database for large analytical and operational workload.
      • Cassandra - Facebook-born project very fast, easily scalable, with option to include consistency with each operation.
      • Scylla - [4.9k ⭐] - 'NoSQL' data store using seastar framework, compatible with Cassandra.

  • Distributed File Systems

    • Graph

      • XtreemFS - Fault tolerant file system.
      • HDFS - Hadoop File System is a a widely popular choice among its big data competitors, providing high throughput access.
      • Lustre - File system for computer clusters.
      • CephFS - Unified, distributed storage system.
      • GlusterFS - Scale-out NAS file system.
      • MooseFS - POSIX-compliant distributed file system.

  • Hadoop Ecosystem

    • Resource Management

      • MESOS - Works like a Linux Kernel by managing CPU, memory, storage and other resources across the cluster.
      • YARN - 'Yet Another Resource Negotiator', works like a kernel to manage computer resources across the clusters.
      • MESOS - Works like a Linux Kernel by managing CPU, memory, storage and other resources across the cluster.

    • Dashboard

      • Ambari - Dashboard that integrates most of hadoop related technologies for easy management and executions.

    • Data Ingestion

      • Sqoop - Efficiently transfer data between Hadoop and structured datastores such as relational databases.
      • Flume - Distributed, highly available and efficient in collecting, aggregating and moving large amounts of log data.
      • Apache Kafka - Widely popular message broker with low latency for data streaming.

    • Workflow Scheduler

      • Oozie - Create workflows in xml to execute jobs (from other hadoop-ecosystem applications) in steps, allows for parallel execution as well.

    • Query

      • Hive - Query hadoop stored data in SQL.
      • Pig - Scriping language that looks like SQL to query hadoop data.

    • Processing

      • Tez - Solves a similar problem to Spark and MapReduce, it's more efficient than MapReduce because it calculates the most efficient way of doing it.
      • Map Reduce - MapReduce, as the name implies, maps data and reduce the results.
      • Spark - Powerful data processing to not only process data like Tez (and MapReduce), it can process streams of data in real time, apply regression analysis algorithms in ML and much more.
      • Apex - \*Retired project, it's a YARN-native platform that unifies stream and batch processing.

    • DB

      • HBase - [3.6k ⭐] - Modeled after Google's Bigtable and written in Java. Developed as a part of Apache Hadoop project.

  • Introduction / Interviews

  • Relational Database Management System

    • MariaDB - MariaDB is a fork of MySQL server.
    • MySQL - Widely used relational database.
    • PostgresSQL - Relational database that has been gaining popularity.
    • SQLite - Another widely used database that is built into all mobile phones and most computers.
    • Sql Server - Widely used relational database.

  • Message Broker

    • Graph

      • Apache Pulsar - Created by yahoo, also highly scalable, low latency, geo-replication and multi-tenacy.
      • Amazon MQ - Open source message broker from Amazon.
      • Apache ActiveMQ - It's a multi-protocol, java based messaging server.
      • RabbitMQ - Widely popular lightweight
      • IronMQ - Very fast and highly scalable messaging broker. (not open source)
      • Azure Service Bus - A fully managed enterprise integration message broker.

  • Resource Management

    • Graph

      • Kubernetes - Highly popular way to deploy, manage and automatically scale a cluster of containers on bare-metal or virtual servers.

  • Stream Processing

    • Graph

      • Apache Samza - Build stateful applications that process data in real time from multiple sources, including Kafka. Easy and inexpensive multi-subscriber model, can eliminate backpressure and has reliable persistency with low latency.
      • Apache Flink - Based on the concept of streams and transofrmations. Uses maven, handles batch tasks as data streams with finite boundaries. Low latency, high throughput.
      • Amazon Kinesis Streams - Durable, scalable, real-tme service. Collects gigabytes of data per second from hundreds of thousands of sources, including database event streams, website clickstreams, financial transactions, etc.
      • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.
      • Azure Stream Analytics - Real-time analytics service that is designed for mission-critical workloads.

  • Load Balancers

    • Open Source Software

      • SeeSaw - [5.1k ⭐] - Used by Google, developed in Go, linux-based virtual load balancer server.
      • HAProxy - Widely popular option, provides high-availability, proxy, TCP/HTTP load balancing. Used by Reddit, Imgur, MaxCDN, GitHub, AirBNB.
      • Zevenet - Supports L3, L4 and L7. Easy install with a docker repo. Supports advanced health-check monitorining.
      • Neutrino - Used by eBay, built with Scala and Netty. Supports round-robin and least-connection algorithms.

    • Hardware

      • F5 - Robust hardware load balancer option, supporting multiple protocols (IP, TCP, FTP, UDP, HTTP).
      • TP-Link - Cheaper alternative that works as a load balancer.
      • Barracuda - One of the top choices for load balancing when it comes to in-house servers. Top security measures built in, comprehensive reports and monitoring outbound traffic for data loss prevention.

    • Cloud

      • Amazon Elastic Load Balancing - Popular choice for amazon customers, supports lambda functions, highly scalable.
      • Google Load Balancing - Popular choice for google customers, comes with auto-scaling feature, very fast, has intergrated CDN.
      • Cloudflare Load Balancing - Scalable load balancing by Cloudflare, feature fast failover and a dashboard.
      • DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.
      • Azure Load Balancing - Popular choice for Microsoft's Azure customers. Supports internal and external traffics, ipv6, monitorining and the standard load balancing set of features.
      • DigitalOcean Load Balancing - If you're a digitalocean customer, this is a good option, very cheap, regional availability, scalable, easy to deploy among your other droplets.

  • REST Framework

    • Resource Management

      • ASP.NET Core MVC - A rich framework for building web apps and APIs using the Model-View-Controller design pattern in C# or F#. Number 6 on [TechEmpower Composite Benchmarks](https://www.techempower.com/benchmarks/#section=data-r19&hw=ph&test=composite) for web frameworks.
Categories
Advanced 64 Introduction / Interviews 24 Hadoop Ecosystem 15 Load Balancers 13 NoSQL 10 Message Broker 6 Distributed File Systems 6 Relational Database Management System 5 Stream Processing 5 Resource Management 1 REST Framework 1
Sub Categories
Graph 20 Cloud 6 Open Source Software 4 Processing 4 Resource Management 4 Wide Column Store 4 Hardware 3 Cache (Key-Value) 3 Data Ingestion 3 Query 2 Dashboard 1 Document Store 1 Workflow Scheduler 1 DB 1