{"id":13410967,"url":"https://github.com/InterviewReady/system-design-resources","last_synced_at":"2025-03-14T16:33:25.238Z","repository":{"id":36965469,"uuid":"493934907","full_name":"InterviewReady/system-design-resources","owner":"InterviewReady","description":"These are the best resources for System Design on the Internet","archived":false,"fork":false,"pushed_at":"2024-08-22T04:25:10.000Z","size":115,"stargazers_count":16165,"open_issues_count":0,"forks_count":2021,"subscribers_count":302,"default_branch":"main","last_synced_at":"2024-12-28T11:15:13.272Z","etag":null,"topics":["cache","fault-tolerance","scalability","system-design"],"latest_commit_sha":null,"homepage":"https://interviewready.io","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"gpl-3.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/InterviewReady.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":"SECURITY.md","support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-05-19T05:37:49.000Z","updated_at":"2024-12-28T09:55:28.000Z","dependencies_parsed_at":"2023-11-26T06:23:58.922Z","dependency_job_id":"ce3c2d7b-d511-4947-b5ba-87f7e0982273","html_url":"https://github.com/InterviewReady/system-design-resources","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InterviewReady%2Fsystem-design-resources","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InterviewReady%2Fsystem-design-resources/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InterviewReady%2Fsystem-design-resources/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/InterviewReady%2Fsystem-design-resources/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/InterviewReady","download_url":"https://codeload.github.com/InterviewReady/system-design-resources/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":243610610,"owners_count":20318996,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cache","fault-tolerance","scalability","system-design"],"created_at":"2024-07-30T20:01:10.532Z","updated_at":"2025-03-14T16:33:24.896Z","avatar_url":"https://github.com/InterviewReady.png","language":null,"readme":"# System Design Resources\nThese are the best resources for System Design on the Internet.\n\n# Table of Contents\n\n- [Video Processing](#video-processing)\n- [Cluster and Workflow Management](#cluster-and-workflow-management)\n- [Intra-Service Messaging](#intra-service-messaging)\n- [Message Queue Antipattern](#message-queue-antipattern)\n- [Service Mesh](#service-mesh)\n- [Practical System Design](#practical-system-design)\n- [Distributed File System](#distributed-file-system)\n- [Time Series Databases](#time-series-databases)\n- [Rate Limiting](#rate-limiting)\n- [In Memory Database - Redis](#in-memory-database---redis)\n- [Network Protocols](#network-protocols)\n- [Chess Engine Design](#chess-engine-design)\n- [Subscription Management System](#subscription-management-system)\n- [Google Docs](#google-docs)\n- [API Design](#api-design)\n- [NoSQL Database Internals](#nosql-database-internals)\n- [NoSQL Database Algorithms](#nosql-database-algorithms)\n- [Database Replication](#database-replication)\n- [Containers and Docker](#containers-and-docker)\n- [Capacity Estimation](#capacity-estimation)\n- [Publisher Subscriber](#publisher-subscriber)\n- [Event Driven Architectures](#event-driven-architectures)\n- [Software Architectures](#software-architectures)\n- [Microservices](#microservices)\n- [Distributed Transactions consistency Patterns](#distributed-transactions-consistency-patterns)\n- [Load Balancing](#load-balancing)\n- [Alerts and Anomaly Detection](#alerts-and-anomaly-detection)\n- [Distributed Logging](#distributed-logging)\n- [Metrics and Text Search Engine](#metrics-and-text-search-engine)\n- [Single Point of Failure](#single-point-of-failure)\n- [Location Based Services](#location-based-services)\n- [Batch Processing](#batch-processing)\n- [Real Time Stream Processing](#real-time-stream-processing)\n- [Caching](#caching)\n- [Distributed Consensus](#distributed-consensus)\n- [Authorization](#authorization)\n- [Content Delivery Network](#content-delivery-network)\n- [Testing Distributed Systems](#testing-distributed-systems)\n- [System Design Resources](#system-design-resources)\n\n##\n## Video Processing\n- [Transcoding Videos at Scale](https://www.egnyte.com/blog/2018/12/transcoding-how-we-serve-videos-at-scale/)\n- [Facebook Video Broadcasting](https://engineering.fb.com/ios/under-the-hood-broadcasting-live-video-to-millions/)\n- [Netflix Video Encoding at Scale](https://netflixtechblog.com/high-quality-video-encoding-at-scale-d159db052746)\n- [Netflix Shot based encoding](https://netflixtechblog.com/optimized-shot-based-encodes-now-streaming-4b9464204830)\n\n##\n## Cluster and Workflow Management\n- [Facebook Cluster Management](https://engineering.fb.com/data-center-engineering/twine/)\n- [Google Autopilot - Autoscaling](https://dl.acm.org/doi/pdf/10.1145/3342195.3387524)\n- [Netflix Workflow Orchestration](https://netflix.github.io/conductor/)\n- [Opensource Workflow Management](https://github.com/spotify/luigi)\n- [Meta Hardware Management](https://engineering.fb.com/2020/12/09/data-center-engineering/how-facebook-keeps-its-large-scale-infrastructure-hardware-up-and-running/)\n- [Meta Capacity Assignment](https://engineering.fb.com/2022/09/06/data-center-engineering/viewing-the-world-as-a-computer-global-capacity-management/)\n- [Amazon EC2](https://www.allthingsdistributed.com/2015/07/under-the-hood-of-the-amazon-ec2-container-service.html)\n\n##\n## Intra-Service Messaging\n- [What is a message queue](https://www.cloudamqp.com/blog/what-is-message-queuing.html)\n- [AirBnb Idempotency](https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb)\n- [Meta Async Task Computing](https://engineering.fb.com/2023/01/31/production-engineering/meta-asynchronous-computing/)\n\n## Message Queue Antipattern\n- [DB as queue Antipattern](https://en.wikipedia.org/wiki/Database-as-IPC)\n- [Using a database as a message queue](https://softwareengineering.stackexchange.com/questions/231410/why-database-as-queue-so-bad)\n- [Anti-pattern of DB as a queue](http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html)\n- [Drawbacks of DB as a queue](https://www.cloudamqp.com/blog/why-is-a-database-not-the-right-tool-for-a-queue-based-system.html)\n\n##\n## Service Mesh\n- [Kubernetes Service Mesh](https://akomljen.com/kubernetes-service-mesh/)\n- [Kubernetes Sidecar](https://www.weave.works/blog/introduction-to-service-meshes-on-kubernetes-and-progressive-delivery)\n- [Service Mesh](https://www.weave.works/blog/introduction-to-service-meshes-on-kubernetes-and-progressive-delivery)\n- [NginX Service Mesh](https://docs.nginx.com/nginx-service-mesh/about/what-is-nsm/)\n- [Data Plane and Control Plane](https://blog.envoyproxy.io/service-mesh-data-plane-vs-control-plane-2774e720f7fc)\n\n##\n## Practical System Design\n- [Facebook Messenger Optimisations](https://spectrum.ieee.org/how-facebooks-software-engineers-prepare-messenger-for-new-years-eve)\n- [YouTube Architecture](http://highscalability.com/youtube-architecture)\n- [YouTube scalability 2012](https://www.youtube.com/watch?v=w5WVu624fY8)\n- [Distributed Design Patterns](http://horicky.blogspot.com/2010/10/scalable-system-design-patterns.html)\n- [Monolith to Microservice](https://martinfowler.com/articles/break-monolith-into-microservices.html)\n- [Zerodha Tech Stack](https://zerodha.tech/blog/hello-world/)\n\n##\n## Distributed File System\n- [Open Source Distributed File System](https://docs.ceph.com/en/latest/architecture/)\n- [Amazon S3 Performance hacks](https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/)\n- [Amazon S3 object expiration](https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/)\n\n##\n## Time Series Databases\n- [Pinterest Time Series Database](https://medium.com/pinterest-engineering/goku-building-a-scalable-and-high-performant-time-series-database-system-a8ff5758a181)\n- [Uber Time Series DB](https://eng.uber.com/aresdb/)\n- [TimeSeries Relational DB](https://blog.timescale.com/blog/time-series-data-why-and-how-to-use-a-relational-database-instead-of-nosql-d0cd6975e87c)\n- [Facebook Gorilla Time Series DB](http://www.vldb.org/pvldb/vol8/p1816-teller.pdf)\n\n##\n## Rate Limiting\n- [Circuit Breaker Algorithm](https://martinfowler.com/bliki/CircuitBreaker.html)\n- [Uber Rate Limiter](https://github.com/uber-go/ratelimit/blob/master/ratelimit.go)\n\n##\n## In Memory Database - Redis\n- [Redis Official Documentation](https://redis.com/)\n- [Learn Redis through Redis University](https://university.redis.com/)\n- [Redis Open Source Repo](https://github.com/redis/redis)\n- [Redis Architecture](https://medium.com/opstree-technology/redis-cluster-architecture-replication-sharding-and-failover-86871e783ac0)\n\n##\n## Network Protocols\n- [What is HTTP](https://engineering.cred.club/head-of-line-hol-blocking-in-http-1-and-http-2-50b24e9e3372)\n- [QUIC Protocol](https://www.akamai.com/blog/performance/http3-and-quic-past-present-and-future)\n- [TCP Protocol algorithms](https://ee.lbl.gov/papers/congavoid.pdf) (First 10 pages are important)\n- [WebRTC](https://webrtc.github.io/webrtc-org/blog/2012/07/23/a-great-introduction-to-webrtc.html)\n- [WebSockets](https://datatracker.ietf.org/doc/html/rfc6455#section-1.2)\n- [Dynamic Source Routing using QUIC](https://fb.watch/fSEbI4KHlA/)\n\n##\n## Chess Engine Design\n- [Chess Engine Building](https://www.youtube.com/watch?v=U4ogK0MIzqk)\n\n##\n## Subscription Management System\n- [Subscription Manager](https://netflixtechblog.com/building-a-rule-based-platform-to-manage-netflix-membership-skus-at-scale-e3c0f82aa7bc)\n\n##\n## Google Docs\n- [Operational Transform](http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation)\n- [Google Docs](https://www.youtube.com/watch?v=uOFzWZrsPV0\u0026list=PLX)\n- [Lumiere](https://www.arxiv.org/abs/2401.12945)\n\n\n## \n## API Design\n\n- [API Design at Airbnb](https://medium.com/airbnb-engineering/building-services-at-airbnb-part-1-c4c1d8fa811b)\n- [Swagger APIs](https://swagger.io/docs/specification/about/)\n\n##\n## NoSQL Database Internals\n\n- [Cassandra Architecture](https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/architecture/archIntro.html)\n- [Google BigTable Architecture](https://static.googleusercontent.com/media/research.google.com/en//archive/bigtable-osdi06.pdf)\n- [Amazon Dynamo DB Internals](https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html)\n- [Design Patterns in Amazon Dynamo DB](https://www.youtube.com/watch?v=HaEPXoXVf2k)\n- [Internals of Amazon Dynamo DB](https://www.youtube.com/watch?v=yvBR71D0nAQ)\n\n##\n## NoSQL Database Algorithms\n\n- [Hyperloglog Algorithm](https://odino.org/my-favorite-data-structure-hyperloglog/)\n- [Log Structured Merge Tree](https://www.cs.umb.edu/~poneil/lsmtree.pdf)\n- [Sorted String Tables and Compaction Strategies](https://github.com/scylladb/scylla/wiki/SSTable-compaction-and-compaction-strategies)\n- [Leveled Compaction Cassandra](https://www.datastax.com/blog/leveled-compaction-apache-cassandra)\n- [Scylla DB Compaction](https://github.com/scylladb/scylla/wiki/SSTable-compaction-and-compaction-strategies)\n- [Indexing in Cassandra](https://www.bmc.com/blogs/cassandra-clustering-columns-partition-composite-key/)\n\n##\n## Database Replication\n\n- [Database replication](https://dev.mysql.com/doc/refman/8.0/en/replication.html)\n- [Netflix Data replication - Change Data Capture](https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b)\n- [LinkedIn Logging Usecases](https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying)\n- [Uber Trillions of indexes in LedgerStore](https://www.uber.com/en-IN/blog/how-ledgerstore-supports-trillions-of-indexes)\n\n##\n## Containers and Docker\n\n- [Facebook Twine Containerization](https://engineering.fb.com/developer-tools/zookeeper-twine/)\n- [CloudFlare Containerization](https://blog.cloudflare.com/cloud-computing-without-containers/)\n- [Docker Architecture](https://docs.docker.com/get-started/overview/#docker-architecture)\n\n##\n## Capacity Estimation\n\n- [Google Capacity Estimation](https://www.youtube.com/watch?v=modXC5IWTJI)\n- [Scalability at YouTube 2012](https://www.youtube.com/watch?v=G-lGCC4KKok)\n- [Back of envelope Calculations at AWS](https://www.youtube.com/watch?v=-3qetLv2Yp0)\n- [Capacity Estimation](http://static.googleusercontent.com/media/research.google.com/en//people/jeff/stanford-295-talk.pdf)\n\n##\n## Publisher Subscriber\n\n- [Oracle Publisher Subscriber](https://docs.oracle.com/cd/B10501_01/appdev.920/a96590/adg15pub.htm)\n- [Amazon Pub Sub Messaging](https://aws.amazon.com/pub-sub-messaging/)\n- [Asynchronous processing](http://blog.codepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/)\n- [Async Request Response](https://www.enterpriseintegrationpatterns.com/patterns/conversation/RequestResponse.html)\n\n##\n## Event Driven Architectures\n\n- [Martin Fowler- Event Driven Architecture](https://www.youtube.com/watch?v=STKCRSUsyP0)\n- [Event Driven Architecture](https://martinfowler.com/articles/201701-event-driven.html)\n\n##\n## Software Architectures\n\n- [Hexagonal Architecture](https://netflixtechblog.com/ready-for-changes-with-hexagonal-architecture-b315ec967749)\n- [Hexagonal architecture (Alistair Cockburn)](https://alistair.cockburn.us/hexagonal-architecture/)\n- [The Clean Code by Robert C. Martin (Uncle Bob)](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html)\n- [CQRS](https://martinfowler.com/bliki/CQRS.html)\n- [DomainDrivenDesign](https://martinfowler.com/bliki/DomainDrivenDesign.html)\n\n##\n## Microservices\n\n- [Monolith Architecture](https://buttercms.com/books/microservices-for-startups/should-you-always-start-with-a-monolith/)\n- [Monoliths vs Microservices](https://articles.microservices.com/monolithic-vs-microservices-architecture-5c4848858f59)\n- [Microservices](http://highscalability.com/blog/2018/4/5/do-you-have-too-many-microservices-five-design-attributes-th.html)\n- [Uber Nanoservices antipattern](https://www.youtube.com/watch?v=kb-m2fasdDY)\n- [Uber Domain oriented microservice](https://eng.uber.com/microservice-architecture/)\n\n##\n## Distributed Transactions consistency Patterns\n\n- [Transactional outbox](https://microservices.io/patterns/data/transactional-outbox.html)\n- [SAGAS Long lived transactions (LLTs)](https://www.cs.cornell.edu/andru/cs711/2002fa/reading/sagas.pdf)\n\n##\n## Load Balancing\n\n- [Load Balancer with Sticky Sessions](https://stackoverflow.com/questions/10494431/sticky-and-non-sticky-sessions)\n- [NetScaler what is load balancing](https://www.netscaler.com/articles/what-is-load-balancing)\n- [Nginx Load Balancing](https://www.nginx.com/resources/glossary/load-balancing/)\n- [Consistent hashing](https://michaelnielsen.org/blog/consistent-hashing/)\n- [Minimizing connection churn](https://netflixtechblog.com/curbing-connection-churn-in-zuul-2feb273a3598#5e4d)\n- [Google Subsetting Algorithm](https://queue.acm.org/detail.cfm?id=3570937)\n\n##\n## Alerts and Anomaly Detection\n\n- [Outlier Detection](https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e)\n- [Anomaly Detection](https://towardsdatascience.com/machine-learning-for-anomaly-detection-and-condition-monitoring-d4614e7de770)\n- [Uber Real Time Monitoring and Root Cause Analysis Argos](https://eng.uber.com/argos-real-time-alerts/)\n- [Microsoft Anomaly Detection](https://www.youtube.com/watch?v=12Xq9OLdQwQ\u0026t=0s)\n- [Facebook Data Engineering](https://engineering.fb.com/2016/05/09/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/)\n- [LinkedIn Real Time Alerting](https://engineering.linkedin.com/blog/2019/06/smart-alerts-in-thirdeye--linkedins-real-time-monitoring-platfor)\n- [LinkedIn Isolation Forests](https://engineering.linkedin.com/blog/2019/isolation-forest)\n\n##\n## Distributed Logging\n\n- [Uber Distributed Request Tracing](https://eng.uber.com/distributed-tracing/)\n- [Pintrest Logging](https://medium.com/@Pinterest_Engineering/open-sourcing-singer-pinterests-performant-and-reliable-logging-agent-610fecf35566)\n- [Google Monitoring Infrastructure](https://www.facebook.com/atscaleevents/videos/959344524420015/)\n\n##\n## Metrics and Text Search Engine\n\n- [Facebook real-time text search engine](https://www.facebook.com/watch/?v=432864835468)\n- [Elastic Search Time Based Querying](https://www.elastic.co/guide/en/elasticsearch/guide/current/time-based.html)\n- [Elastic Search Aggregation](https://www.elastic.co/guide/en/elasticsearch/guide/current/aggregations.html)\n\n##\n## Single Point of Failure\n\n- [Avoiding Single Points of Failure](https://medium.com/the-cloud-architect/patterns-for-resilient-architecture-part-3-16e8601c488e)\n- [Netflix Multi-Region Availability](https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b)\n- [Oracle Single Points of failure](https://docs.oracle.com/cd/E19693-01/819-0992/fjdch/index.html)\n- [DNS single point of failure 2004](http://www.tenereillo.com/GSLBPageOfShame.htm)\n- [DNS traffic management by Shopify](https://shopify.engineering/introduction-dns-traffic-management)\n- [Sharding](https://medium.com/@jeeyoungk/how-sharding-works-b4dec46b3f6)\n\n##\n## Location Based Services\n\n- [Google S2 library](https://blog.christianperone.com/2015/08/googles-s2-geometry-on-the-sphere-cells-and-hilbert-curve/)\n\n##\n## Batch Processing\n\n- [Map Reduce Architecture](https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf)\n\n##\n## Real Time Stream Processing\n\n- [LinkedIn Brooklin- Real-time data streaming](https://engineering.linkedin.com/blog/2019/brooklin-open-source)\n- [Netflix Real Time Stream Processing](https://netflixtechblog.com/keystone-real-time-stream-processing-platform-a3ee651812a)\n- [KSQLDB for Kafka](https://docs.ksqldb.io/en/latest/operate-and-deploy/how-it-works/)\n- [Netflix Psyberg](https://netflixtechblog.com/1-streamlining-membership-data-engineering-at-netflix-with-psyberg-f68830617dd1)\n\n##\n## Caching\n\n- [Google Guava Cache](https://github.com/google/guava/wiki/CachesExplained)\n- [Caching (See the README)](https://github.com/ben-manes/caffeine/)\n- [Caching](http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html)\n- [Microsoft Caching Guide](https://docs.microsoft.com/en-us/previous-versions/msp-n-p/dn589802(v%3dpandp.10))\n- [Caching patterns](https://hazelcast.com/blog/a-hitchhikers-guide-to-caching-patterns/)\n- [Uber's Integrated Cache for 40M RPS](https://www.uber.com/en-IN/blog/how-uber-serves-over-40-million-reads-per-second-using-an-integrated-cache)\n\n##\n## Distributed consensus\n\n- [Paxos](http://ifeanyi.co/posts/understanding-consensus/)\n- [Raft](https://raft.github.io/)\n\n##\n## Authorization\n\n- [Designing an Authorization Model for an Enterprise](https://cerbos.dev/blog/designing-an-authorization-model-for-an-enterprise)\n- [The Architectural Patterns of Cloud-native Authorization Systems](https://www.aserto.com/blog/5-laws-cloud-native-authorization)\n\n##\n## Content Delivery Network\n\n- [AWS CloudFront CDN with S3](https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-s3-amazon-cloudfront-a-match-made-in-the-cloud/)\n\n##\n## Testing Distributed Systems\n\n- [Deterministic Testing](https://www.youtube.com/watch?v=4fFDFbi3toc)\n- [TLA+ by Leslie Lamport](https://lamport.azurewebsites.net/tla/tla.html)\n- [Jepsen](https://jenpsen.io)\n\n##\n## System Design Resources\n\n- [Designing Data-Intensive Applications Book](https://amzn.to/3SyNAOy)\n- [WhitePapers](https://interviewready.io/blog/white-papers-worth-reading-for-software-engineers)\n- [InterviewReady Videos](https://interviewready.io?source=github)\n- [System Design Online Judge](https://interviewready.io/question-list/system-design-judge)\n","funding_links":[],"categories":["Others","miscellaneous","Index","Don't forget to give a :star: to make the project popular","📈 Resources:","▶️ Content"],"sub_categories":["System Design","🐈 GitHub Repositories:"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FInterviewReady%2Fsystem-design-resources","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FInterviewReady%2Fsystem-design-resources","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FInterviewReady%2Fsystem-design-resources/lists"}