https://github.com/developer-y/scalable-software-architecture

Collection of tech talks, papers and web links on Distributed Systems, Scalability and System Design.
https://github.com/developer-y/scalable-software-architecture
Last synced: 14 days ago
JSON representation
Collection of tech talks, papers and web links on Distributed Systems, Scalability and System Design.
Host: GitHub
URL: https://github.com/developer-y/scalable-software-architecture
Owner: Developer-Y
License: mit
Created: 2016-10-08T05:39:39.000Z (over 9 years ago)
Default Branch: master
Last Pushed: 2023-11-08T17:14:18.000Z (over 2 years ago)
Last Synced: 2025-03-26T03:12:54.016Z (11 months ago)
Homepage:
Size: 40 KB
Stars: 2,043
Watchers: 107
Forks: 408
Open Issues: 0
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project

README

          # Scalable-Software-Architecture

Collection of tech talks, papers and web links on Distributed Systems, Scalability and System Design.

## Tech Talks

#### General Advice on System Design and Scalability

  * [Lecture - Scalability - Harvard Web Development, David Malan](https://www.youtube.com/watch?v=-W9F__D3oY4)

  * [Building Software Systems At Google and Lessons Learned](https://www.youtube.com/watch?v=modXC5IWTJI)

  * [Scalable Internet Architectures - Theo Schlossnagle](https://www.youtube.com/watch?v=2WuT2rdLK5A)

  * [Seattle Conference on Scalability - Jeff Dean](https://www.youtube.com/watch?v=YFGFvCCFk0g)

  * [Best Practices for Scaling Web Apps](https://www.youtube.com/watch?v=tQ2V9QSv48M)

  * [Building a Scalable Architecture for Web Apps](https://www.youtube.com/watch?v=yeyugGZO2qc)

  * [Web application architecture: The whole stack - Allen Holub](https://www.youtube.com/watch?v=SgEFNRf67Cw)

  * [Scalable Distributed Design](https://www.youtube.com/watch?v=E9GFA-YkzFY)

  * [Building Software at Google Scale Tech Talk](https://www.youtube.com/watch?v=2qv3fcXW1mg)

  * [Seattle Conference on Scalability: Scaling Google for Every User](https://www.youtube.com/watch?v=LT1UFZSbcxE)

  * [Seattle Conference on Scalability: Lessons In Building Scalable Systems](https://www.youtube.com/watch?v=mS48X9oEar0)

  * [Workers, Queues, and Cache](https://www.infoq.com/presentations/workers-queues-cache)

  * [Velocity 2012: Jay Parikh, "Building for a Billion Users"](https://www.youtube.com/watch?v=oodS71YtkGU)

  * [Building Large Systems at Google](https://www.youtube.com/watch?v=_cltA0W-ehQ)

  * [3000 images per second - Henna Kerman - @Scale 2016](https://www.youtube.com/watch?v=CEZ6oO1u1u8)

  * [Scaling to over 1,000,000 requests per second](https://www.youtube.com/watch?v=v-KfhCGW3MA)

  * [Jeff Dean: "Achieving Rapid Response Times in Large Online Services" Keynote - Velocity 2014](https://www.youtube.com/watch?v=1-3Ahy7Fxsc)

  * [Getting Things Done at Scale](https://www.infoq.com/presentations/cultural-change-management)

  * [Scale-oriented Architecture with APIs](https://www.infoq.com/presentations/queues-proxy-microservices-automation)

  * [You Won't Believe How the Biggest Sites Build Scalable and Resilient Systems!](https://www.infoq.com/presentations/scalable-resilient-systems)

  * [Scalable Distributed Design](https://www.youtube.com/watch?v=E9GFA-YkzFY)

  * [Distributed Patterns you should know by Eric Redmond](https://www.youtube.com/watch?v=otWvr8VBRx0)

  * [GOTO 2012 • Runaway Complexity in Big Data Systems...and a Plan to Stop it • Nathan Marz](https://www.youtube.com/watch?v=ucHjyb6jv08)

  * [Seattle Conference on Scalability: Abstractions for handling large datasets](https://www.youtube.com/watch?v=oRwFpQKgRps)

  * [High Performance Web Sites and YSlow](https://www.youtube.com/watch?v=BTHvs3V8DBA)

  

#### Company/Product specific tech talks

  * [Seattle Conference on Scalability: YouTube Scalability](https://www.youtube.com/watch?v=w5WVu624fY8)

  * [How to answer design question: How do you design a twitter?](https://www.youtube.com/watch?v=gX8S7b8UYl8)

  * [Operations at Twitter: Scaling Beyond 100 Million Users](https://www.youtube.com/watch?v=z8LU0Cj6BOU)

  * [How We've Scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc)

  * [Keynote: Twitter's search architecture](https://www.youtube.com/watch?v=AguWva8P_DI)

  * [Marco Cecconi - "The Architecture of StackOverflow"](https://www.youtube.com/watch?v=t6kM2EM6so4)

  * [Scalability at YouTube](https://www.youtube.com/watch?v=G-lGCC4KKok)

  * [Lessons of Scale at Facebook](https://www.youtube.com/watch?v=QCHiNEw73AU)

  * [Scale at Facebook](https://www.infoq.com/presentations/Scale-at-Facebook)

  * [Flight Lightning - Scaling Twitter core infrastructure](https://www.youtube.com/watch?v=6OvrFkLSoZ0)

  * [Scaling Instagram with Mike Krieger](https://www.youtube.com/watch?v=oNA2C1vC8FQ)

  * [GOTO 2014 • Scaling Pinterest • Marty Weiner](https://www.youtube.com/watch?v=jQNCuD_hxdQ) ([InfoQ link](https://www.infoq.com/presentations/Pinterest))

  * [Real-Time Delivery Architecture at Twitter](https://www.youtube.com/watch?v=J5auCY4ajK8)

  * [OSCON 2014: How Instagram.com Works; Pete Hunt](https://www.youtube.com/watch?v=VkTCL6Nqm6Y)

  * [Scaling the Data Infrastructure at Instagram](https://www.youtube.com/watch?v=1sPgogJlKWM)

  * [O'Reilly MySQL CE 2011: Jeremy Cole, "Big and Small Data at @Twitter"](https://www.youtube.com/watch?v=5cKTP36HVgI)

  * [O'Reilly Webcast: How Pinterest Architected and Built Their Sharded MySQL Datastore](https://www.youtube.com/watch?v=aFJm1YlRr5Q)

  * [Timelines at Scale (Twitter)](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)

  * [Architecture at Scale at ESPN](https://www.infoq.com/presentations/Architecture-Scale-ESPN)

  * [Building Highly-resilient Systems at Pinterest](https://www.infoq.com/presentations/pinterest-resilient-systems)

  * [Scaling Uber](https://www.infoq.com/presentations/uber-scalability-arch)

  * [How Zoom works](http://highscalability.com/blog/2020/5/14/a-short-on-how-zoom-works.html)

  * [Scaling Engineering Culture at Twitter](https://www.youtube.com/watch?v=H9wTuT6yQlc)

  * [Keynote - Systems at Facebook Scale](https://www.youtube.com/watch?v=dlixGkelP9U)

  * [Scaling YouTube's Backend: The Vitess Trade-offs](https://www.youtube.com/watch?v=5yDO-tmIoXY)

  * [Hacker Way: Rethinking Web App Development at Facebook](https://www.youtube.com/watch?v=nYkdrAPrdcw)

  * [Data Platform Architecture, Evolution, and Philosophy at Netflix](https://www.youtube.com/watch?v=YT-ztW5QNUQ)

  * [Data Platform Architecture, Evolution, and Philosophy at Netflix](https://www.youtube.com/watch?v=uH8T7JMzloM)

  * [Structure, Personalization, Scale: A Deep Dive into LinkedIn Search](https://www.infoq.com/presentations/linkedin-search)

  * [Scaling Foursquare: From Check-ins to Recommendations](https://www.infoq.com/presentations/scale-foursquare)

  * [How Netflix Leverages Multiple Regions to Increase Availability: An Active-Active Case Study](https://www.infoq.com/presentations/netflix-availability-regions)

  * [That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp](https://www.infoq.com/presentations/whatsapp-scalability)

  * [How Facebook Scales Big Data Systems](https://www.infoq.com/presentations/scale-facebook-big-data)

  * [Scaling Uber's Real-time Market Platform](https://www.infoq.com/presentations/uber-market-platform)

  * [Software Development & Architecture @ LinkedIn](https://www.infoq.com/presentations/linkedin-architecture-practices)

  * [Etsy Search: How We Index and Query 26 Million One-of-a-kind Items](https://www.infoq.com/presentations/etsy-search-ecosystem)

  * [Scalability Lessons from eBay, Google, and Real-time Games](https://www.infoq.com/presentations/scalability-ebay-google-kixeye)

  * [How SoundCloud Uses Cassandra](https://www.infoq.com/presentations/soundcloud-cassandra)

  * [Service Architectures at Scale: Lessons from Google and eBay](https://www.infoq.com/presentations/service-arch-scale-google-ebay)

  * [Solidifying the Cloud: How Google Backs up the Internet](https://www.infoq.com/presentations/google-cloud-backup-failure)

  * [Real-Time Systems at Twitter](https://www.infoq.com/presentations/real-time-twitter)

  * [Serving user intent : Facebook style notifications using HBase and Event streams](https://www.youtube.com/watch?v=hIxXaH0CrVQ)

  * [Netflix's Distributed Computing Strategies: Optimistic Design for the Eventual Consistency Model](https://www.youtube.com/watch?v=6R1WhWkh6pg)

#### Distributed Computing

  * [Intro to Hadoop and MapReduce (Udacity)](https://www.youtube.com/playlist?list=PLAwxTw4SYaPkXJ6LAV96gH8yxIfGaN3H-)

  * [Introduction to Hadoop](https://www.youtube.com/watch?v=Pq3OyQO-l3E)

  * [MapReduce Flow Chart](https://www.youtube.com/watch?v=6OemZEJdMp8)

  * [Distributed Computing CS 61A UC Berkeley](https://www.youtube.com/watch?v=zGTZ-wwA9O8&list=PL6BsET-8jgYWkCySVpSbOj-MCk6xC0i4N)

  * [MapReduce  CS 61A UC Berkeley](https://www.youtube.com/watch?v=6mABLxxtIlg&list=PL6BsET-8jgYVivQ1AS4UsPR2tyr830OHj)

  * [Cluster Computing and MapReduce Lecture 1](https://www.youtube.com/watch?v=yjPBkvYh-ss)

  * [Cluster Computing and MapReduce Lecture 2](https://www.youtube.com/watch?v=-vD6PUdf3Js)

  * [Introducing Apache Hadoop: The Modern Data Operating System](https://www.youtube.com/watch?v=d2xeNpfzsYI)

#### Distributed Database/Large-Scale Storage

  * [Spanner: Google’s Globally-Distributed Database](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/corbett) ([Youtube link](https://www.youtube.com/watch?v=NthK17nbpYs))

  * [Spanner - multi-version, globally- distributed, and synchronously-replicated database](https://www.youtube.com/watch?v=Q7pcMn0_tac)

  * [BigTable: A Distributed Structured Storage System](https://www.youtube.com/watch?v=2cXBNQClehA) ([slides](http://static.googleusercontent.com/media/research.google.com/en//people/jeff/bigtable-uw-2005.pdf)) 

  * [Large-Scale Low-Latency Storage for the Social Network - Data@Scale](https://www.youtube.com/watch?v=5RfFhMwRAic)

  * [Structured Data at Box: How We're Building for Scale](https://www.youtube.com/watch?v=KHrw0Xo8tYo)

  * [F4 - Photo Storage at Facebook](https://www.youtube.com/watch?v=34e_g-Ji_30)

  * [The Storage Technologies Behind Facebook Messages](https://www.youtube.com/watch?v=cSNGGAKJqwk)

  * [Cold Storage at Facebook](https://www.youtube.com/watch?v=FGZqToLuwU0)

  * [Taking Storage for a Ride with Uber](https://www.youtube.com/watch?v=Dg76cNaeB4s)

  * [Zen: Pinterest's Graph Storage Service - @Scale 2014 - Data](https://www.youtube.com/watch?v=yI0vHfgK6oI) ([With Slides](https://www.infoq.com/presentations/zen-pinterest-graph-storage-service))

  * [Storage Systems at a Rapidly Scaling Startup with a Small Team - Data@Scale](https://www.youtube.com/watch?v=bLyv8zKa5DU)

  * [f4: Facebook's Warm BLOB Storage System](https://www.youtube.com/watch?v=imlP3lxwGgQ)

#### Distributed graph processing

  * [Giraph](https://www.youtube.com/watch?v=xyvs7NQdfZk)

  * [Apache Giraph Large Scale Graph Processing On Hadoop](https://www.youtube.com/watch?v=uXw2TKHsqvM)

  * [Processing Over a Billion Edges on Apache Giraph](https://www.youtube.com/watch?v=b5Qmz4zPj-M)

  * [Graph Search: The Power of Connected Data](https://www.infoq.com/presentations/graph-search-connected-data)

  * [Using Graph Partitioning in Distributed Systems Design](https://www.youtube.com/watch?v=QHkhyY9atkE)

  * [Let Me Graph That For You: Building a Graph Database Application](https://www.infoq.com/presentations/graph-db-tools)

  * [GraphChi: Large-Scale Graph Computation on Just a PC](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/kyrola)

  * [PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez)

  

#### Stream Processing

  * [Heron: Real-time Stream Data Processing at Twitter](https://www.youtube.com/watch?v=pUaFOuGgmco)

  * [Samza in LinkedIn: How LinkedIn Processes Billions of Events Everyday in Real-time](https://www.infoq.com/presentations/samza-linkedin-2014)

  * [Mantis: Netflix's Event Stream Processing System](https://www.infoq.com/presentations/mantis)

  * [High Throughput Stream Processing with ACID Guarantees](https://www.infoq.com/presentations/acid-stream-processing)

  * [Martin Kleppmann — Event Sourcing and Stream Processing at Scale](https://www.youtube.com/watch?v=avi-TZI9t2I)

  * [ETE 2012 - Nathan Marz on Storm](https://www.youtube.com/watch?v=bdps8tE0gYo)

  * [Cassandra NYC 2011: Nathan Marz - The Storm and Cassandra Realtime Computation Stack](https://www.youtube.com/watch?v=cF8a_FZwULI)

  

#### API Design

  * [How To Design A Good API and Why it Matters](https://www.youtube.com/watch?v=aAb7hSCtvGw)

  * [How to Design Great APIs - Parse Developer Day 2013](https://www.youtube.com/watch?v=qCdpTji8nxo)

  * [Google I/O 2010 - How Google builds APIs](https://www.youtube.com/watch?v=nyu5ZxGUfgs)

  * [Designing a Beautiful REST+JSON API](https://www.youtube.com/watch?v=5WXYw4J4QOU)

  

#### Web Services and SOA

  * [Introduction to Service Design and Engineering - University of Trento, Italy](https://www.youtube.com/playlist?list=PLBdajHWwi0JCn87QuFT3e58mekU0-6WUT)

  * [REST+JSON API Design - Best Practices for Developers](https://www.youtube.com/watch?v=hdSrT4yjS1g)

  * [What is a Service Oriented Architecture?](https://www.youtube.com/watch?v=nRayJZmj2oY)

  * [Webinar : Practical SOA for the Solution Architect](https://www.youtube.com/watch?v=1KXKppaOgtY)

#### Caching

  * [Scaling Redis at Twitter](https://www.youtube.com/watch?v=rP9EKvWt0zo)

  * [Facebook and memcached - Tech Talk](https://www.youtube.com/watch?v=UH7wkvcf0ys)

  * [Scaling Memcache at Facebook](https://www.youtube.com/watch?v=6phA3IAcEJ8)

  * [How Netflix and reddit scale to handle massive demand](https://www.youtube.com/watch?v=zVqA7Ogutiw)

  * [An analysis of Facebook photo caching](https://www.youtube.com/watch?v=ENaQScyvOzY)

#### NoSQL

  * [Introduction to NoSQL • Martin Fowler](https://www.youtube.com/watch?v=qI_g07C_Q5I)

  * [NoSQL Distilled to an hour by Martin Fowler](https://www.youtube.com/watch?v=ASiU89Gl0F0)

  * [NoSQL Distilled • Pramod Sadalage](https://www.youtube.com/watch?v=l68zxgZS_wc)

  * [Tech Talk: Cassandra Data Modeling](https://www.youtube.com/watch?v=tg6eIht-00M)

  * [NoSQL Explained](https://www.youtube.com/watch?v=Q6gn2kuOTV0)

  * [Big Data Architecture Patterns](https://www.youtube.com/watch?v=-N9i-YXoQBE)

  * [Graph Databases Exposed](https://www.youtube.com/watch?v=A44CQ4P7oWw)

#### Messaging

  * [Queue It! What Job Queues Can Do for You!](https://www.infoq.com/presentations/job-queue-gearman-beanstalkd)

  * [Joydeep Sen Sarma - Messaging architecture at Facebook](https://www.youtube.com/watch?v=5hUmdoMOrpo)

  * [Messaging at Scale at Instagram](https://www.youtube.com/watch?v=E708csv4XgY)

  * [Building a Distributed Data Ingestion System with RabbitMQ](https://www.infoq.com/presentations/data-replication-rabbitmq)

  * [scaling web applications with message queues - Lenz Gschwendtner](https://www.youtube.com/watch?v=aOrGq9yb6og)

  

#### Object Oriented Analysis and Design

  * [Software Architecture & Design | Udacity](https://www.youtube.com/playlist?list=PLAwxTw4SYaPkMTetlG7xKWaI5ZAZFX8fL)

  * [OOSE: Software Dev Using UML and Java](https://www.youtube.com/playlist?list=PLJ9pm_Rc9HesnkwKlal_buSIHA-jTZMpO)

  * [Eric Evans — Tackling Complexity in the Heart of Software](https://www.youtube.com/watch?v=dnUFEg68ESM)

  * [Domain Driven Design](https://www.youtube.com/watch?v=BPAZwFNxgrg)

  * [Domain-Driven Design](https://www.youtube.com/watch?v=RNUn2R7TptM)

  * [How You Can Architect and Develop Enterprise Mission-Critical Applications with Domain-Driven Design]( https://www.youtube.com/watch?v=aieoAWXNjl0)

  * [DDD: putting the model to work](https://www.infoq.com/presentations/model-to-work-evans)

  * [Eric Evans on DDD: Strategic Design](https://www.infoq.com/presentations/strategic-design-evans)

  * [Architecting and Implementing Domain-Driven Design Patterns with Microsoft .NET](https://www.youtube.com/watch?v=DaQCGy_GIPE)

  * [SOLID Design Patterns in C#](https://www.youtube.com/playlist?list=PL8m4NUhTQU48oiGCSgCP1FiJEcg_xJzyQ)

  * [Object Oriented Design](https://www.youtube.com/playlist?list=PLGLfVvz_LVvS5P7khyR4xDp7T9lCk9PgE)

  * [Design Patterns Video Tutorial](https://www.youtube.com/playlist?list=PLF206E906175C7E07)

  * [Object Oriented Design Interview Question: Design a Car Parking Lot.](https://www.youtube.com/watch?v=2vtT6TBnOAM)

  * [Google's Clean Code Talks](https://www.youtube.com/playlist?list=PL693EFD059797C21E)

  * [NYC Tech Talk Series: How Google Backs Up the Internet](https://www.youtube.com/watch?v=eNliOm9NtCM)

  * [Robert C Martin(Uncle Bob) -Clean Architecture and Design-2012](https://www.youtube.com/watch?v=asLUTiJJqdE)

  * [Robert C Martin - Clean Architecture and Design](https://www.youtube.com/watch?v=Nsjsiz2A9mg)

  * [Robert C Martin - The Single Responsibility Principle](https://www.youtube.com/watch?v=Gt0M_OHKhQE)

  * [Robert C Martin - Clean Architecture](https://www.youtube.com/watch?v=Nltqi7ODZTM)

  * [The S.O.L.I.D. Principles of OO and Agile Design - by Uncle Bob Martin](https://www.youtube.com/watch?v=t86v3N4OshQ)

  * [Solid Principles by Uncle Bob Martin](https://www.youtube.com/watch?v=oar-T2KovwE)

  * [The Principles of Clean Architecture by Uncle Bob Martin](https://www.youtube.com/watch?v=o_TH-Y78tt4)

  * [Unleash Your Domain - Greg Young](https://vimeo.com/19428577)

  

#### Misc

  * [Differential Synchronization](https://www.youtube.com/watch?v=S2Hp_1jqpY8)

  * [Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303) | AWS re:Invent 2013](https://www.youtube.com/watch?v=LpNbjXFPyZ0)

  * [Deepak Agarwal: Recommender Systems - The Art and Science of Matching Items to Users](https://www.youtube.com/watch?v=bRzOBGLCRbc)

  * [Lecture 12 -Analyzing Big Data with Twitter: Recommender Systems by Alpa Jain](https://www.youtube.com/watch?v=NSscbT7JwxY)

  * [Transactions across Datacenters](https://www.youtube.com/watch?v=srOgpXECblk)  ([Slides](http://snarfed.org/transactions_across_datacenters_io.html))

  * [Finding the Needle in the Haystack - or - Troubleshooting Distributed Systems](https://www.infoq.com/presentations/mondemand-troubleshoot-distributed-system)

  * [Finding the Needle in a Big Data Haystack](https://www.infoq.com/presentations/search-hadoop-data-hub)

  * [Large scale image processing on the fly in 25ms with Google's first Network Engineer](https://www.youtube.com/watch?v=H_9cH1tiigs)

  * [Bringing Push Notifications to the Mobile Web](https://www.youtube.com/watch?v=HbmcnjWFGbY)

  

---------------------------------------------------------------

## Papers

#### General

  * [papers-we-love](https://github.com/papers-we-love/papers-we-love)

  * [Google Research](http://research.google.com/pubs/papers.html)

  * [Facebook Research](https://research.facebook.com/publications/)

  * [MIT PDOS](https://pdos.csail.mit.edu/papers/)

  * [Distributed Systems Reading List](https://dancres.github.io/Pages/)

  * [Hints for Computer System Design](http://research.microsoft.com/en-us/um/people/blampson/33-Hints/Acrobat.pdf)

  * The Little Manual of API Design

  * [On Designing and Deploying Internet-Scale Services](https://www.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton.pdf)

  * [Time, Clocks, and the Ordering of Events in a Distributed System](http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf)

  * [Above the Clouds: A Berkeley View of Cloud Computing](http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf)

  * [The Byzantine Generals Problem](http://www.andrew.cmu.edu/course/15-749/READINGS/required/resilience/lamport82.pdf)

  * [How to Design a Good API and Why it Matters - Google Research](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/32713.pdf)

  * Twitter - Automatic Management of Partitioned, Replicated Search Services

  * [High-Availability at Massive Scale: Building Google’s Data Infrastructure for Ads](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44686.pdf)

  * [Making Reliable Distributed Systems in the Presence of Software Errors](http://www.erlang.org/download/armstrong_thesis_2003.pdf)

  * [Fallacies of Distributed Computing Explained](http://www.rgoarchitects.com/Files/fallacies.pdf)

#### Search

  * [The Anatomy of a Large-Scale Hypertextual Web Search Engine (Google Paper)](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/334.pdf)  (Weblink)

  * [The PageRank Citation Ranking: Bringing Order to the Web](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf)

  * [Web Search for a Planet: The Google Cluster Architecture](http://research.google.com/archive/googlecluster-ieee.pdf)

  * [Unicorn: A System for Searching the Social Graph](http://db.disi.unitn.eu/pages/VLDBProgram/pdf/industry/p871-curtiss.pdf)  ([FB link](https://research.facebook.com/publications/unicorn-a-system-for-searching-the-social-graph/))

#### P2P

  * [Chord: A scalable peer-to-peer lookup service for Internet applications](https://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf)

  * [Building peer-to-peer systems with Chord, a distributed lookup service](https://pdos.csail.mit.edu/papers/chord:hotos01/hotos8.pdf)

  * [Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web](https://www.akamai.com/us/en/multimedia/documents/technical-publication/consistent-hashing-and-random-trees-distributed-caching-protocols-for-relieving-hot-spots-on-the-world-wide-web-technical-publication.pdf)  ([CS 168: Consistent Hashing](http://web.stanford.edu/class/cs168/l/l1.pdf)  | [Algorithmic Nuggets in Content Delivery](http://www.sigcomm.org/sites/default/files/ccr/papers/2015/July/0000000-0000009.pdf))

  * Web Caching with Consistent Hashing ([Web link](http://www8.org/w8-papers/2a-webserver/caching/paper2.html))

  * [Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems](https://github.com/papers-we-love/papers-we-love/blob/master/distributed_systems/pastry-scalable-decentralized-object-location-and-routing-for-large-scale-peer-to-peer-systems.pdf)

  * [Simple Efficient Load Balancing Algorithms for Peer-to-Peer Systems](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/33339.pdf)

#### Distributed Computing

  * [MapReduce: Simplified Data Processing on Large Clusters](http://research.google.com/archive/mapreduce-osdi04.pdf)

  * [Resident Distributed Datasets: a Fault-Tolerant Abstraction for In-Memory Cluster Computing (Zahari et al.)](https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf)

  * [Kafka: a Distributed Messaging System for Log Processing](http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf)

#### Distributed Database/Large-Scale Storage

  * [Dynamo: Amazon's Highly Available Key-value Datastore](http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)

  * [The Google File System](http://research.google.com/archive/gfs-sosp2003.pdf)

  * [Bigtable: A Distributed Storage System for Structured Data](http://research.google.com/archive/bigtable-osdi06.pdf)

  * [Spanner: Google's Globally-Distributed Database - Google Research](http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf)

  * TAO: Facebook’s Distributed Data Store for the Social Graph

  * [F1: A Distributed SQL Database That Scales](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41344.pdf)

  * [Scuba: Diving into Data at Facebook](http://db.disi.unitn.eu/pages/VLDBProgram/pdf/industry/p767-wiener.pdf)

  * f4: Facebook’s Warm BLOB Storage System

  * [Finding a needle in Haystack: Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf)

  * [Cassandra - A Decentralized Structured Storage System](https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf)

#### Consistency

  * [Consistency Tradeoffs in Modern Distributed Database System Design](http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf)

  * [Paxos Made Live - An Engineering Perspective](http://www.cs.utexas.edu/users/lorenzo/corsi/cs380d/papers/paper2-1.pdf)

  * [Paxos Made Simple](https://github.com/papers-we-love/papers-we-love/blob/master/distributed_systems/paxos-made-simple.pdf)

  * [Existential Consistency: Measuring and Understanding Consistency at Facebook](https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xpt1/t39.2365-6/12057015_1010569855670644_348748003_n/Existential_Consistency_Measuring_and_Understanding_Consistency_at_Facebook.pdf)

  * [In Search of an Understandable Consensus Algorithm](https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf)

#### Distributed Graph processing

  * [SQLGraph: An Efficient Relational-Based Property Graph Store](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43287.pdf)

  * [One Trillion Edges: Graph Processing at FacebookScale](http://www.vldb.org/pvldb/vol8/p1804-ching.pdf)

  * [Pregel: A System for Large-Scale Graph Processing](http://kowshik.github.io/JPregel/pregel_paper.pdf)

  * [Dremel: Interactive Analysis of Web-Scale Datasets](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf)

#### Company/Product specific

  * [Scaling Memcache at Facebook](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf)

  * [Realtime Data Processing at Facebook](https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xat1/t39.2365-6/13331599_975087972607457_1796386216_n/Realtime_Data_Processing_at_Facebook.pdf)

  * [Holistic Configuration Management at Facebook](https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xta1/t39.2365-6/12057212_1623087361291843_1316956191_n/Holistic_Configuration_Management_at_Facebook.pdf)

  * [The Unified Logging Infrastructure for Data Analytics at Twitter](http://vldb.org/pvldb/vol5/p1771_georgelee_vldb2012.pdf)

  * [Scaling Big Data Mining Infrastructure: The Twitter Experience](http://www.kdd.org/sites/default/files/issues/14-2-2012-12/V14-02-02-Lin.pdf)

  * [Large-scale cluster management at Google with Borg](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf)

#### Misc

  * [Differential Synchronization](http://research.google.com/pubs/archive/35605.pdf)

  * A Low-bandwidth Network File System

  * [Maglev: A Fast and Reliable Software Network Load Balancer](http://research.google.com/pubs/archive/44824.pdf)

  * [The Chubby Lock Service for Loosely-Coupled Distributed Systems](http://research.google.com/archive/chubby-osdi06.pdf)

  * [Everything You Always Wanted to Know About Synchronization but Were Afraid to Ask](http://sigops.org/sosp/sosp13/papers/p33-david.pdf)

  * [Transactional storage for geo-replicated systems](https://github.com/papers-we-love/papers-we-love/blob/master/datastores/transactional-storage-for-geo-replicated-systems.pdf)

  * [Highly Available Transactions: Virtues and Limitations](http://www.bailis.org/papers/hat-vldb2014.pdf)

  * [The Log-Structured Merge-Tree](http://www.cs.umb.edu/~poneil/lsmtree.pdf)

--------------------------------------------------------

## Web Links

#### General

  * https://www.hiredintech.com/

  * https://github.com/checkcheckzz/system-design-interview

  * https://github.com/shashank88/system_design

  * http://highscalability.com/ ([All Time Favorites](http://highscalability.com/all-time-favorites/))

  * http://blog.gainlo.co/index.php/category/system-design-interview-questions/

  * http://www.allthingsdistributed.com/archives.html  (Back-to-Basics series)

  * [ Scalability for dummies - Part 1](http://www.lecloud.net/post/7295452622/scalability-for-dummies-part-1-clones)  ([Part 2](http://www.lecloud.net/post/7994751381/scalability-for-dummies-part-2-database) | [Part 3](http://www.lecloud.net/post/9246290032/scalability-for-dummies-part-3-cache) | [Part 4](http://www.lecloud.net/post/9699762917/scalability-for-dummies-part-4-asynchronism))

  * [Scalable Web Architecture and Distributed Systems](http://aosabook.org/en/distsys.html)

  * [Introduction to Architecting Systems for Scale](http://lethain.com/introduction-to-architecting-systems-for-scale/)

  * [Software Architect Roadmap](https://roadmap.sh/software-architect)  

#### Examples 

Highscalability.com has wide collection of articles on Scalable architecture. Individual web links will be added below if they are not already highlighted in popular sites like highscalability.

  * [Trending at Instagram](http://instagram-engineering.tumblr.com/post/122961624217/trending-at-instagram)

  * [Search Architecture - Instagram](http://instagram-engineering.tumblr.com/post/124162066737/search-architecture)

  * [THE UBER ENGINEERING TECH STACK, PART I](https://eng.uber.com/tech-stack-part-one/) ([PART II](https://eng.uber.com/tech-stack-part-two/))

  * [Processing Payments At Scale](https://engineering.groupon.com/2016/misc/processing-payments-at-scale/)

  * [Personalized Group Recommendations on Flickr](http://code.flickr.net/2016/09/30/personalized-group-recommendations-on-flickr/)

  * [Building The LinkedIn Knowledge Graph](https://engineering.linkedin.com/blog/2016/10/building-the-linkedin-knowledge-graph)

  * [Personal recommendations for the Foursquare homescreen](http://engineering.foursquare.com/2015/08/06/personal-recommendations-for-the-foursquare-homescreen/)

--------------------------------------------------

  

## Books

  * [Microsoft Application Architecture Guide, 2nd Edition (Online)](https://msdn.microsoft.com/en-us/library/ff650706.aspx)

  * [The Architecture of Open Source Applications](http://aosabook.org/en/index.html)

  * Design Patterns: Elements of Reusable Object-Oriented Software

  * Head First Design Patterns

  * Patterns of Enterprise Application Architecture

  * Domain Driven Design by Eric Evans

  * Agile Software Development, Principles, Patterns and Practices by Robert Martin

--------------------------------------------------
ecosyste.ms

Data

Tools

Indexes

Applications

Experiments

Awesome

https://github.com/developer-y/scalable-software-architecture

Awesome Lists containing this project

README
