{"id":13620928,"url":"https://github.com/Developer-Y/Scalable-Software-Architecture","last_synced_at":"2025-04-14T22:32:58.235Z","repository":{"id":37736030,"uuid":"70306859","full_name":"Developer-Y/Scalable-Software-Architecture","owner":"Developer-Y","description":"Collection of tech talks, papers and web links on Distributed Systems, Scalability and System Design.","archived":false,"fork":false,"pushed_at":"2023-11-08T17:14:18.000Z","size":41,"stargazers_count":2043,"open_issues_count":0,"forks_count":408,"subscribers_count":107,"default_branch":"master","last_synced_at":"2025-03-26T03:12:54.016Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Developer-Y.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-08T05:39:39.000Z","updated_at":"2025-03-25T08:29:53.000Z","dependencies_parsed_at":"2023-02-15T15:30:58.726Z","dependency_job_id":"ac670fdc-970d-41bb-b20c-021f8761ff4c","html_url":"https://github.com/Developer-Y/Scalable-Software-Architecture","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Developer-Y%2FScalable-Software-Architecture","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Developer-Y%2FScalable-Software-Architecture/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Developer-Y%2FScalable-Software-Architecture/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Developer-Y%2FScalable-Software-Architecture/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Developer-Y","download_url":"https://codeload.github.com/Developer-Y/Scalable-Software-Architecture/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248972439,"owners_count":21191794,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-01T21:01:00.812Z","updated_at":"2025-04-14T22:32:53.220Z","avatar_url":"https://github.com/Developer-Y.png","language":null,"readme":"# Scalable-Software-Architecture\n\nCollection of tech talks, papers and web links on Distributed Systems, Scalability and System Design.\n\n## Tech Talks\n\n#### General Advice on System Design and Scalability\n  * [Lecture - Scalability - Harvard Web Development, David Malan](https://www.youtube.com/watch?v=-W9F__D3oY4)\n  * [Building Software Systems At Google and Lessons Learned](https://www.youtube.com/watch?v=modXC5IWTJI)\n  * [Scalable Internet Architectures - Theo Schlossnagle](https://www.youtube.com/watch?v=2WuT2rdLK5A)\n  * [Seattle Conference on Scalability - Jeff Dean](https://www.youtube.com/watch?v=YFGFvCCFk0g)\n  * [Best Practices for Scaling Web Apps](https://www.youtube.com/watch?v=tQ2V9QSv48M)\n  * [Building a Scalable Architecture for Web Apps](https://www.youtube.com/watch?v=yeyugGZO2qc)\n  * [Web application architecture: The whole stack - Allen Holub](https://www.youtube.com/watch?v=SgEFNRf67Cw)\n  * [Scalable Distributed Design](https://www.youtube.com/watch?v=E9GFA-YkzFY)\n  * [Building Software at Google Scale Tech Talk](https://www.youtube.com/watch?v=2qv3fcXW1mg)\n  * [Seattle Conference on Scalability: Scaling Google for Every User](https://www.youtube.com/watch?v=LT1UFZSbcxE)\n  * [Seattle Conference on Scalability: Lessons In Building Scalable Systems](https://www.youtube.com/watch?v=mS48X9oEar0)\n  * [Workers, Queues, and Cache](https://www.infoq.com/presentations/workers-queues-cache)\n  * [Velocity 2012: Jay Parikh, \"Building for a Billion Users\"](https://www.youtube.com/watch?v=oodS71YtkGU)\n  * [Building Large Systems at Google](https://www.youtube.com/watch?v=_cltA0W-ehQ)\n  * [3000 images per second - Henna Kerman - @Scale 2016](https://www.youtube.com/watch?v=CEZ6oO1u1u8)\n  * [Scaling to over 1,000,000 requests per second](https://www.youtube.com/watch?v=v-KfhCGW3MA)\n  * [Jeff Dean: \"Achieving Rapid Response Times in Large Online Services\" Keynote - Velocity 2014](https://www.youtube.com/watch?v=1-3Ahy7Fxsc)\n  * [Getting Things Done at Scale](https://www.infoq.com/presentations/cultural-change-management)\n  * [Scale-oriented Architecture with APIs](https://www.infoq.com/presentations/queues-proxy-microservices-automation)\n  * [You Won't Believe How the Biggest Sites Build Scalable and Resilient Systems!](https://www.infoq.com/presentations/scalable-resilient-systems)\n  * [Scalable Distributed Design](https://www.youtube.com/watch?v=E9GFA-YkzFY)\n  * [Distributed Patterns you should know by Eric Redmond](https://www.youtube.com/watch?v=otWvr8VBRx0)\n  * [GOTO 2012 • Runaway Complexity in Big Data Systems...and a Plan to Stop it • Nathan Marz](https://www.youtube.com/watch?v=ucHjyb6jv08)\n  * [Seattle Conference on Scalability: Abstractions for handling large datasets](https://www.youtube.com/watch?v=oRwFpQKgRps)\n  * [High Performance Web Sites and YSlow](https://www.youtube.com/watch?v=BTHvs3V8DBA)\n  \n\n#### Company/Product specific tech talks\n  * [Seattle Conference on Scalability: YouTube Scalability](https://www.youtube.com/watch?v=w5WVu624fY8)\n  * [How to answer design question: How do you design a twitter?](https://www.youtube.com/watch?v=gX8S7b8UYl8)\n  * [Operations at Twitter: Scaling Beyond 100 Million Users](https://www.youtube.com/watch?v=z8LU0Cj6BOU)\n  * [How We've Scaled Dropbox](https://www.youtube.com/watch?v=PE4gwstWhmc)\n  * [Keynote: Twitter's search architecture](https://www.youtube.com/watch?v=AguWva8P_DI)\n  * [Marco Cecconi - \"The Architecture of StackOverflow\"](https://www.youtube.com/watch?v=t6kM2EM6so4)\n  * [Scalability at YouTube](https://www.youtube.com/watch?v=G-lGCC4KKok)\n  * [Lessons of Scale at Facebook](https://www.youtube.com/watch?v=QCHiNEw73AU)\n  * [Scale at Facebook](https://www.infoq.com/presentations/Scale-at-Facebook)\n  * [Flight Lightning - Scaling Twitter core infrastructure](https://www.youtube.com/watch?v=6OvrFkLSoZ0)\n  * [Scaling Instagram with Mike Krieger](https://www.youtube.com/watch?v=oNA2C1vC8FQ)\n  * [GOTO 2014 • Scaling Pinterest • Marty Weiner](https://www.youtube.com/watch?v=jQNCuD_hxdQ) ([InfoQ link](https://www.infoq.com/presentations/Pinterest))\n  * [Real-Time Delivery Architecture at Twitter](https://www.youtube.com/watch?v=J5auCY4ajK8)\n  * [OSCON 2014: How Instagram.com Works; Pete Hunt](https://www.youtube.com/watch?v=VkTCL6Nqm6Y)\n  * [Scaling the Data Infrastructure at Instagram](https://www.youtube.com/watch?v=1sPgogJlKWM)\n  * [O'Reilly MySQL CE 2011: Jeremy Cole, \"Big and Small Data at @Twitter\"](https://www.youtube.com/watch?v=5cKTP36HVgI)\n  * [O'Reilly Webcast: How Pinterest Architected and Built Their Sharded MySQL Datastore](https://www.youtube.com/watch?v=aFJm1YlRr5Q)\n  * [Timelines at Scale (Twitter)](https://www.infoq.com/presentations/Twitter-Timeline-Scalability)\n  * [Architecture at Scale at ESPN](https://www.infoq.com/presentations/Architecture-Scale-ESPN)\n  * [Building Highly-resilient Systems at Pinterest](https://www.infoq.com/presentations/pinterest-resilient-systems)\n  * [Scaling Uber](https://www.infoq.com/presentations/uber-scalability-arch)\n  * [How Zoom works](http://highscalability.com/blog/2020/5/14/a-short-on-how-zoom-works.html)\n  * [Scaling Engineering Culture at Twitter](https://www.youtube.com/watch?v=H9wTuT6yQlc)\n  * [Keynote - Systems at Facebook Scale](https://www.youtube.com/watch?v=dlixGkelP9U)\n  * [Scaling YouTube's Backend: The Vitess Trade-offs](https://www.youtube.com/watch?v=5yDO-tmIoXY)\n  * [Hacker Way: Rethinking Web App Development at Facebook](https://www.youtube.com/watch?v=nYkdrAPrdcw)\n  * [Data Platform Architecture, Evolution, and Philosophy at Netflix](https://www.youtube.com/watch?v=YT-ztW5QNUQ)\n  * [Data Platform Architecture, Evolution, and Philosophy at Netflix](https://www.youtube.com/watch?v=uH8T7JMzloM)\n  * [Structure, Personalization, Scale: A Deep Dive into LinkedIn Search](https://www.infoq.com/presentations/linkedin-search)\n  * [Scaling Foursquare: From Check-ins to Recommendations](https://www.infoq.com/presentations/scale-foursquare)\n  * [How Netflix Leverages Multiple Regions to Increase Availability: An Active-Active Case Study](https://www.infoq.com/presentations/netflix-availability-regions)\n  * [That's 'Billion' with a 'B': Scaling to the Next Level at WhatsApp](https://www.infoq.com/presentations/whatsapp-scalability)\n  * [How Facebook Scales Big Data Systems](https://www.infoq.com/presentations/scale-facebook-big-data)\n  * [Scaling Uber's Real-time Market Platform](https://www.infoq.com/presentations/uber-market-platform)\n  * [Software Development \u0026 Architecture @ LinkedIn](https://www.infoq.com/presentations/linkedin-architecture-practices)\n  * [Etsy Search: How We Index and Query 26 Million One-of-a-kind Items](https://www.infoq.com/presentations/etsy-search-ecosystem)\n  * [Scalability Lessons from eBay, Google, and Real-time Games](https://www.infoq.com/presentations/scalability-ebay-google-kixeye)\n  * [How SoundCloud Uses Cassandra](https://www.infoq.com/presentations/soundcloud-cassandra)\n  * [Service Architectures at Scale: Lessons from Google and eBay](https://www.infoq.com/presentations/service-arch-scale-google-ebay)\n  * [Solidifying the Cloud: How Google Backs up the Internet](https://www.infoq.com/presentations/google-cloud-backup-failure)\n  * [Real-Time Systems at Twitter](https://www.infoq.com/presentations/real-time-twitter)\n  * [Serving user intent : Facebook style notifications using HBase and Event streams](https://www.youtube.com/watch?v=hIxXaH0CrVQ)\n  * [Netflix's Distributed Computing Strategies: Optimistic Design for the Eventual Consistency Model](https://www.youtube.com/watch?v=6R1WhWkh6pg)\n\n#### Distributed Computing\n  * [Intro to Hadoop and MapReduce (Udacity)](https://www.youtube.com/playlist?list=PLAwxTw4SYaPkXJ6LAV96gH8yxIfGaN3H-)\n  * [Introduction to Hadoop](https://www.youtube.com/watch?v=Pq3OyQO-l3E)\n  * [MapReduce Flow Chart](https://www.youtube.com/watch?v=6OemZEJdMp8)\n  * [Distributed Computing CS 61A UC Berkeley](https://www.youtube.com/watch?v=zGTZ-wwA9O8\u0026list=PL6BsET-8jgYWkCySVpSbOj-MCk6xC0i4N)\n  * [MapReduce  CS 61A UC Berkeley](https://www.youtube.com/watch?v=6mABLxxtIlg\u0026list=PL6BsET-8jgYVivQ1AS4UsPR2tyr830OHj)\n  * [Cluster Computing and MapReduce Lecture 1](https://www.youtube.com/watch?v=yjPBkvYh-ss)\n  * [Cluster Computing and MapReduce Lecture 2](https://www.youtube.com/watch?v=-vD6PUdf3Js)\n  * [Introducing Apache Hadoop: The Modern Data Operating System](https://www.youtube.com/watch?v=d2xeNpfzsYI)\n\n#### Distributed Database/Large-Scale Storage\n  * [Spanner: Google’s Globally-Distributed Database](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/corbett) ([Youtube link](https://www.youtube.com/watch?v=NthK17nbpYs))\n  * [Spanner - multi-version, globally- distributed, and synchronously-replicated database](https://www.youtube.com/watch?v=Q7pcMn0_tac)\n  * [BigTable: A Distributed Structured Storage System](https://www.youtube.com/watch?v=2cXBNQClehA) ([slides](http://static.googleusercontent.com/media/research.google.com/en//people/jeff/bigtable-uw-2005.pdf)) \n  * [Large-Scale Low-Latency Storage for the Social Network - Data@Scale](https://www.youtube.com/watch?v=5RfFhMwRAic)\n  * [Structured Data at Box: How We're Building for Scale](https://www.youtube.com/watch?v=KHrw0Xo8tYo)\n  * [F4 - Photo Storage at Facebook](https://www.youtube.com/watch?v=34e_g-Ji_30)\n  * [The Storage Technologies Behind Facebook Messages](https://www.youtube.com/watch?v=cSNGGAKJqwk)\n  * [Cold Storage at Facebook](https://www.youtube.com/watch?v=FGZqToLuwU0)\n  * [Taking Storage for a Ride with Uber](https://www.youtube.com/watch?v=Dg76cNaeB4s)\n  * [Zen: Pinterest's Graph Storage Service - @Scale 2014 - Data](https://www.youtube.com/watch?v=yI0vHfgK6oI) ([With Slides](https://www.infoq.com/presentations/zen-pinterest-graph-storage-service))\n  * [Storage Systems at a Rapidly Scaling Startup with a Small Team - Data@Scale](https://www.youtube.com/watch?v=bLyv8zKa5DU)\n  * [f4: Facebook's Warm BLOB Storage System](https://www.youtube.com/watch?v=imlP3lxwGgQ)\n\n#### Distributed graph processing\n  * [Giraph](https://www.youtube.com/watch?v=xyvs7NQdfZk)\n  * [Apache Giraph Large Scale Graph Processing On Hadoop](https://www.youtube.com/watch?v=uXw2TKHsqvM)\n  * [Processing Over a Billion Edges on Apache Giraph](https://www.youtube.com/watch?v=b5Qmz4zPj-M)\n  * [Graph Search: The Power of Connected Data](https://www.infoq.com/presentations/graph-search-connected-data)\n  * [Using Graph Partitioning in Distributed Systems Design](https://www.youtube.com/watch?v=QHkhyY9atkE)\n  * [Let Me Graph That For You: Building a Graph Database Application](https://www.infoq.com/presentations/graph-db-tools)\n  * [GraphChi: Large-Scale Graph Computation on Just a PC](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/kyrola)\n  * [PowerGraph: Distributed Graph-Parallel Computation on Natural Graphs](https://www.usenix.org/conference/osdi12/technical-sessions/presentation/gonzalez)\n  \n#### Stream Processing\n  * [Heron: Real-time Stream Data Processing at Twitter](https://www.youtube.com/watch?v=pUaFOuGgmco)\n  * [Samza in LinkedIn: How LinkedIn Processes Billions of Events Everyday in Real-time](https://www.infoq.com/presentations/samza-linkedin-2014)\n  * [Mantis: Netflix's Event Stream Processing System](https://www.infoq.com/presentations/mantis)\n  * [High Throughput Stream Processing with ACID Guarantees](https://www.infoq.com/presentations/acid-stream-processing)\n  * [Martin Kleppmann — Event Sourcing and Stream Processing at Scale](https://www.youtube.com/watch?v=avi-TZI9t2I)\n  * [ETE 2012 - Nathan Marz on Storm](https://www.youtube.com/watch?v=bdps8tE0gYo)\n  * [Cassandra NYC 2011: Nathan Marz - The Storm and Cassandra Realtime Computation Stack](https://www.youtube.com/watch?v=cF8a_FZwULI)\n  \n#### API Design\n  * [How To Design A Good API and Why it Matters](https://www.youtube.com/watch?v=aAb7hSCtvGw)\n  * [How to Design Great APIs - Parse Developer Day 2013](https://www.youtube.com/watch?v=qCdpTji8nxo)\n  * [Google I/O 2010 - How Google builds APIs](https://www.youtube.com/watch?v=nyu5ZxGUfgs)\n  * [Designing a Beautiful REST+JSON API](https://www.youtube.com/watch?v=5WXYw4J4QOU)\n  \n#### Web Services and SOA\n  * [Introduction to Service Design and Engineering - University of Trento, Italy](https://www.youtube.com/playlist?list=PLBdajHWwi0JCn87QuFT3e58mekU0-6WUT)\n  * [REST+JSON API Design - Best Practices for Developers](https://www.youtube.com/watch?v=hdSrT4yjS1g)\n  * [What is a Service Oriented Architecture?](https://www.youtube.com/watch?v=nRayJZmj2oY)\n  * [Webinar : Practical SOA for the Solution Architect](https://www.youtube.com/watch?v=1KXKppaOgtY)\n\n#### Caching\n  * [Scaling Redis at Twitter](https://www.youtube.com/watch?v=rP9EKvWt0zo)\n  * [Facebook and memcached - Tech Talk](https://www.youtube.com/watch?v=UH7wkvcf0ys)\n  * [Scaling Memcache at Facebook](https://www.youtube.com/watch?v=6phA3IAcEJ8)\n  * [How Netflix and reddit scale to handle massive demand](https://www.youtube.com/watch?v=zVqA7Ogutiw)\n  * [An analysis of Facebook photo caching](https://www.youtube.com/watch?v=ENaQScyvOzY)\n\n#### NoSQL\n  * [Introduction to NoSQL • Martin Fowler](https://www.youtube.com/watch?v=qI_g07C_Q5I)\n  * [NoSQL Distilled to an hour by Martin Fowler](https://www.youtube.com/watch?v=ASiU89Gl0F0)\n  * [NoSQL Distilled • Pramod Sadalage](https://www.youtube.com/watch?v=l68zxgZS_wc)\n  * [Tech Talk: Cassandra Data Modeling](https://www.youtube.com/watch?v=tg6eIht-00M)\n  * [NoSQL Explained](https://www.youtube.com/watch?v=Q6gn2kuOTV0)\n  * [Big Data Architecture Patterns](https://www.youtube.com/watch?v=-N9i-YXoQBE)\n  * [Graph Databases Exposed](https://www.youtube.com/watch?v=A44CQ4P7oWw)\n\n#### Messaging\n  * [Queue It! What Job Queues Can Do for You!](https://www.infoq.com/presentations/job-queue-gearman-beanstalkd)\n  * [Joydeep Sen Sarma - Messaging architecture at Facebook](https://www.youtube.com/watch?v=5hUmdoMOrpo)\n  * [Messaging at Scale at Instagram](https://www.youtube.com/watch?v=E708csv4XgY)\n  * [Building a Distributed Data Ingestion System with RabbitMQ](https://www.infoq.com/presentations/data-replication-rabbitmq)\n  * [scaling web applications with message queues - Lenz Gschwendtner](https://www.youtube.com/watch?v=aOrGq9yb6og)\n  \n#### Object Oriented Analysis and Design\n  * [Software Architecture \u0026 Design | Udacity](https://www.youtube.com/playlist?list=PLAwxTw4SYaPkMTetlG7xKWaI5ZAZFX8fL)\n  * [OOSE: Software Dev Using UML and Java](https://www.youtube.com/playlist?list=PLJ9pm_Rc9HesnkwKlal_buSIHA-jTZMpO)\n  * [Eric Evans — Tackling Complexity in the Heart of Software](https://www.youtube.com/watch?v=dnUFEg68ESM)\n  * [Domain Driven Design](https://www.youtube.com/watch?v=BPAZwFNxgrg)\n  * [Domain-Driven Design](https://www.youtube.com/watch?v=RNUn2R7TptM)\n  * [How You Can Architect and Develop Enterprise Mission-Critical Applications with Domain-Driven Design]( https://www.youtube.com/watch?v=aieoAWXNjl0)\n  * [DDD: putting the model to work](https://www.infoq.com/presentations/model-to-work-evans)\n  * [Eric Evans on DDD: Strategic Design](https://www.infoq.com/presentations/strategic-design-evans)\n  * [Architecting and Implementing Domain-Driven Design Patterns with Microsoft .NET](https://www.youtube.com/watch?v=DaQCGy_GIPE)\n  * [SOLID Design Patterns in C#](https://www.youtube.com/playlist?list=PL8m4NUhTQU48oiGCSgCP1FiJEcg_xJzyQ)\n  * [Object Oriented Design](https://www.youtube.com/playlist?list=PLGLfVvz_LVvS5P7khyR4xDp7T9lCk9PgE)\n  * [Design Patterns Video Tutorial](https://www.youtube.com/playlist?list=PLF206E906175C7E07)\n  * [Object Oriented Design Interview Question: Design a Car Parking Lot.](https://www.youtube.com/watch?v=2vtT6TBnOAM)\n  * [Google's Clean Code Talks](https://www.youtube.com/playlist?list=PL693EFD059797C21E)\n  * [NYC Tech Talk Series: How Google Backs Up the Internet](https://www.youtube.com/watch?v=eNliOm9NtCM)\n  * [Robert C Martin(Uncle Bob) -Clean Architecture and Design-2012](https://www.youtube.com/watch?v=asLUTiJJqdE)\n  * [Robert C Martin - Clean Architecture and Design](https://www.youtube.com/watch?v=Nsjsiz2A9mg)\n  * [Robert C Martin - The Single Responsibility Principle](https://www.youtube.com/watch?v=Gt0M_OHKhQE)\n  * [Robert C Martin - Clean Architecture](https://www.youtube.com/watch?v=Nltqi7ODZTM)\n  * [The S.O.L.I.D. Principles of OO and Agile Design - by Uncle Bob Martin](https://www.youtube.com/watch?v=t86v3N4OshQ)\n  * [Solid Principles by Uncle Bob Martin](https://www.youtube.com/watch?v=oar-T2KovwE)\n  * [The Principles of Clean Architecture by Uncle Bob Martin](https://www.youtube.com/watch?v=o_TH-Y78tt4)\n  * [Unleash Your Domain - Greg Young](https://vimeo.com/19428577)\n  \n#### Misc\n  * [Differential Synchronization](https://www.youtube.com/watch?v=S2Hp_1jqpY8)\n  * [Infrastructure at Scale: Apache Kafka, Twitter Storm \u0026 Elastic Search (ARC303) | AWS re:Invent 2013](https://www.youtube.com/watch?v=LpNbjXFPyZ0)\n  * [Deepak Agarwal: Recommender Systems - The Art and Science of Matching Items to Users](https://www.youtube.com/watch?v=bRzOBGLCRbc)\n  * [Lecture 12 -Analyzing Big Data with Twitter: Recommender Systems by Alpa Jain](https://www.youtube.com/watch?v=NSscbT7JwxY)\n  * [Transactions across Datacenters](https://www.youtube.com/watch?v=srOgpXECblk)  ([Slides](http://snarfed.org/transactions_across_datacenters_io.html))\n  * [Finding the Needle in the Haystack - or - Troubleshooting Distributed Systems](https://www.infoq.com/presentations/mondemand-troubleshoot-distributed-system)\n  * [Finding the Needle in a Big Data Haystack](https://www.infoq.com/presentations/search-hadoop-data-hub)\n  * [Large scale image processing on the fly in 25ms with Google's first Network Engineer](https://www.youtube.com/watch?v=H_9cH1tiigs)\n  * [Bringing Push Notifications to the Mobile Web](https://www.youtube.com/watch?v=HbmcnjWFGbY)\n  \n---------------------------------------------------------------\n\n## Papers\n\n#### General\n  * [papers-we-love](https://github.com/papers-we-love/papers-we-love)\n  * [Google Research](http://research.google.com/pubs/papers.html)\n  * [Facebook Research](https://research.facebook.com/publications/)\n  * [MIT PDOS](https://pdos.csail.mit.edu/papers/)\n  * [Distributed Systems Reading List](https://dancres.github.io/Pages/)\n  * [Hints for Computer System Design](http://research.microsoft.com/en-us/um/people/blampson/33-Hints/Acrobat.pdf)\n  * \u003ca href=\"http://people.mpi-inf.mpg.de/~jblanche/api-design.pdf\" target=\"_blank\"\u003eThe Little Manual of API Design\u003c/a\u003e\n  * [On Designing and Deploying Internet-Scale Services](https://www.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton.pdf)\n  * [Time, Clocks, and the Ordering of Events in a Distributed System](http://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf)\n  * [Above the Clouds: A Berkeley View of Cloud Computing](http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf)\n  * [The Byzantine Generals Problem](http://www.andrew.cmu.edu/course/15-749/READINGS/required/resilience/lamport82.pdf)\n  * [How to Design a Good API and Why it Matters - Google Research](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/32713.pdf)\n  * \u003ca href=\"http://www.umiacs.umd.edu/~jimmylin/publications/Leibert_etal_SoCC2011.pdf\" target=\"_blank\"\u003eTwitter - Automatic Management of Partitioned, Replicated Search Services\u003c/a\u003e\n  * [High-Availability at Massive Scale: Building Google’s Data Infrastructure for Ads](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/44686.pdf)\n  * [Making Reliable Distributed Systems in the Presence of Software Errors](http://www.erlang.org/download/armstrong_thesis_2003.pdf)\n  * [Fallacies of Distributed Computing Explained](http://www.rgoarchitects.com/Files/fallacies.pdf)\n\n#### Search\n  * [The Anatomy of a Large-Scale Hypertextual Web Search Engine (Google Paper)](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/334.pdf)  (\u003ca href=\"http://infolab.stanford.edu/~backrub/google.html\" target=\"_blank\"\u003eWeblink\u003c/a\u003e)\n  * [The PageRank Citation Ranking: Bringing Order to the Web](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf)\n  * [Web Search for a Planet: The Google Cluster Architecture](http://research.google.com/archive/googlecluster-ieee.pdf)\n  * [Unicorn: A System for Searching the Social Graph](http://db.disi.unitn.eu/pages/VLDBProgram/pdf/industry/p871-curtiss.pdf)  ([FB link](https://research.facebook.com/publications/unicorn-a-system-for-searching-the-social-graph/))\n\n#### P2P\n  * [Chord: A scalable peer-to-peer lookup service for Internet applications](https://pdos.csail.mit.edu/papers/chord:sigcomm01/chord_sigcomm.pdf)\n  * [Building peer-to-peer systems with Chord, a distributed lookup service](https://pdos.csail.mit.edu/papers/chord:hotos01/hotos8.pdf)\n  * [Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web](https://www.akamai.com/us/en/multimedia/documents/technical-publication/consistent-hashing-and-random-trees-distributed-caching-protocols-for-relieving-hot-spots-on-the-world-wide-web-technical-publication.pdf)  ([CS 168: Consistent Hashing](http://web.stanford.edu/class/cs168/l/l1.pdf)  | [Algorithmic Nuggets in Content Delivery](http://www.sigcomm.org/sites/default/files/ccr/papers/2015/July/0000000-0000009.pdf))\n  * \u003ca href=\"http://www.cs.columbia.edu/~asherman/papers/cachePaper.pdf\" target=\"_blank\"\u003eWeb Caching with Consistent Hashing\u003c/a\u003e ([Web link](http://www8.org/w8-papers/2a-webserver/caching/paper2.html))\n  * [Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems](https://github.com/papers-we-love/papers-we-love/blob/master/distributed_systems/pastry-scalable-decentralized-object-location-and-routing-for-large-scale-peer-to-peer-systems.pdf)\n  * [Simple Efficient Load Balancing Algorithms for Peer-to-Peer Systems](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/33339.pdf)\n\n#### Distributed Computing\n  * [MapReduce: Simplified Data Processing on Large Clusters](http://research.google.com/archive/mapreduce-osdi04.pdf)\n  * [Resident Distributed Datasets: a Fault-Tolerant Abstraction for In-Memory Cluster Computing (Zahari et al.)](https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf)\n  * [Kafka: a Distributed Messaging System for Log Processing](http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf)\n\n\n#### Distributed Database/Large-Scale Storage\n  * [Dynamo: Amazon's Highly Available Key-value Datastore](http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf)\n  * [The Google File System](http://research.google.com/archive/gfs-sosp2003.pdf)\n  * [Bigtable: A Distributed Storage System for Structured Data](http://research.google.com/archive/bigtable-osdi06.pdf)\n  * [Spanner: Google's Globally-Distributed Database - Google Research](http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf)\n  * \u003ca href=\"https://cs.uwaterloo.ca/~brecht/courses/854-Emerging-2014/readings/data-store/tao-facebook-distributed-datastore-atc-2013.pdf\" target=\"_blank\"\u003eTAO: Facebook’s Distributed Data Store for the Social Graph\u003c/a\u003e\n  * [F1: A Distributed SQL Database That Scales](http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41344.pdf)\n  * [Scuba: Diving into Data at Facebook](http://db.disi.unitn.eu/pages/VLDBProgram/pdf/industry/p767-wiener.pdf)\n  * \u003ca href=\"http://www-bcf.usc.edu/~wyattllo/papers/f4-osdi14.pdf\" target=\"_blank\"\u003ef4: Facebook’s Warm BLOB Storage System\u003c/a\u003e\n  * [Finding a needle in Haystack: Facebook’s photo storage](https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf)\n  * [Cassandra - A Decentralized Structured Storage System](https://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf)\n\n\n#### Consistency\n  * [Consistency Tradeoffs in Modern Distributed Database System Design](http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf)\n  * [Paxos Made Live - An Engineering Perspective](http://www.cs.utexas.edu/users/lorenzo/corsi/cs380d/papers/paper2-1.pdf)\n  * [Paxos Made Simple](https://github.com/papers-we-love/papers-we-love/blob/master/distributed_systems/paxos-made-simple.pdf)\n  * [Existential Consistency: Measuring and Understanding Consistency at Facebook](https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xpt1/t39.2365-6/12057015_1010569855670644_348748003_n/Existential_Consistency_Measuring_and_Understanding_Consistency_at_Facebook.pdf)\n  * [In Search of an Understandable Consensus Algorithm](https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf)\n\n\n#### Distributed Graph processing\n  * [SQLGraph: An Efficient Relational-Based Property Graph Store](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43287.pdf)\n  * [One Trillion Edges: Graph Processing at FacebookScale](http://www.vldb.org/pvldb/vol8/p1804-ching.pdf)\n  * [Pregel: A System for Large-Scale Graph Processing](http://kowshik.github.io/JPregel/pregel_paper.pdf)\n  * [Dremel: Interactive Analysis of Web-Scale Datasets](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36632.pdf)\n\n\n#### Company/Product specific\n  * [Scaling Memcache at Facebook](https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf)\n  * [Realtime Data Processing at Facebook](https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xat1/t39.2365-6/13331599_975087972607457_1796386216_n/Realtime_Data_Processing_at_Facebook.pdf)\n  * [Holistic Configuration Management at Facebook](https://fbcdn-dragon-a.akamaihd.net/hphotos-ak-xta1/t39.2365-6/12057212_1623087361291843_1316956191_n/Holistic_Configuration_Management_at_Facebook.pdf)\n  * [The Unified Logging Infrastructure for Data Analytics at Twitter](http://vldb.org/pvldb/vol5/p1771_georgelee_vldb2012.pdf)\n  * [Scaling Big Data Mining Infrastructure: The Twitter Experience](http://www.kdd.org/sites/default/files/issues/14-2-2012-12/V14-02-02-Lin.pdf)\n  * [Large-scale cluster management at Google with Borg](http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43438.pdf)\n\n#### Misc\n  * [Differential Synchronization](http://research.google.com/pubs/archive/35605.pdf)\n  * \u003ca href=\"https://www.cis.upenn.edu/~bcpierce/courses/dd/papers/lbfs.pdf\" target=\"_blank\"\u003eA Low-bandwidth Network File System\u003c/a\u003e\n  * [Maglev: A Fast and Reliable Software Network Load Balancer](http://research.google.com/pubs/archive/44824.pdf)\n  * [The Chubby Lock Service for Loosely-Coupled Distributed Systems](http://research.google.com/archive/chubby-osdi06.pdf)\n  * [Everything You Always Wanted to Know About Synchronization but Were Afraid to Ask](http://sigops.org/sosp/sosp13/papers/p33-david.pdf)\n  * [Transactional storage for geo-replicated systems](https://github.com/papers-we-love/papers-we-love/blob/master/datastores/transactional-storage-for-geo-replicated-systems.pdf)\n  * [Highly Available Transactions: Virtues and Limitations](http://www.bailis.org/papers/hat-vldb2014.pdf)\n  * [The Log-Structured Merge-Tree](http://www.cs.umb.edu/~poneil/lsmtree.pdf)\n\n\n--------------------------------------------------------\n\n\n## Web Links\n\n#### General\n  * https://www.hiredintech.com/\n  * https://github.com/checkcheckzz/system-design-interview\n  * https://github.com/shashank88/system_design\n  * http://highscalability.com/ ([All Time Favorites](http://highscalability.com/all-time-favorites/))\n  * http://blog.gainlo.co/index.php/category/system-design-interview-questions/\n  * http://www.allthingsdistributed.com/archives.html  (Back-to-Basics series)\n  * [ Scalability for dummies - Part 1](http://www.lecloud.net/post/7295452622/scalability-for-dummies-part-1-clones)  ([Part 2](http://www.lecloud.net/post/7994751381/scalability-for-dummies-part-2-database) | [Part 3](http://www.lecloud.net/post/9246290032/scalability-for-dummies-part-3-cache) | [Part 4](http://www.lecloud.net/post/9699762917/scalability-for-dummies-part-4-asynchronism))\n  * [Scalable Web Architecture and Distributed Systems](http://aosabook.org/en/distsys.html)\n  * [Introduction to Architecting Systems for Scale](http://lethain.com/introduction-to-architecting-systems-for-scale/)\n  * [Software Architect Roadmap](https://roadmap.sh/software-architect)  \n\n#### Examples \n\nHighscalability.com has wide collection of articles on Scalable architecture. Individual web links will be added below if they are not already highlighted in popular sites like highscalability.\n\n  * [Trending at Instagram](http://instagram-engineering.tumblr.com/post/122961624217/trending-at-instagram)\n  * [Search Architecture - Instagram](http://instagram-engineering.tumblr.com/post/124162066737/search-architecture)\n  * [THE UBER ENGINEERING TECH STACK, PART I](https://eng.uber.com/tech-stack-part-one/) ([PART II](https://eng.uber.com/tech-stack-part-two/))\n  * [Processing Payments At Scale](https://engineering.groupon.com/2016/misc/processing-payments-at-scale/)\n  * [Personalized Group Recommendations on Flickr](http://code.flickr.net/2016/09/30/personalized-group-recommendations-on-flickr/)\n  * [Building The LinkedIn Knowledge Graph](https://engineering.linkedin.com/blog/2016/10/building-the-linkedin-knowledge-graph)\n  * [Personal recommendations for the Foursquare homescreen](http://engineering.foursquare.com/2015/08/06/personal-recommendations-for-the-foursquare-homescreen/)\n\n--------------------------------------------------\n  \n## Books\n  * [Microsoft Application Architecture Guide, 2nd Edition (Online)](https://msdn.microsoft.com/en-us/library/ff650706.aspx)\n  * [The Architecture of Open Source Applications](http://aosabook.org/en/index.html)\n  * Design Patterns: Elements of Reusable Object-Oriented Software\n  * Head First Design Patterns\n  * Patterns of Enterprise Application Architecture\n  * Domain Driven Design by Eric Evans\n  * Agile Software Development, Principles, Patterns and Practices by Robert Martin\n\n--------------------------------------------------\n","funding_links":[],"categories":["Others","HarmonyOS","Scalable Software Architecture"],"sub_categories":["Windows Manager","Resources"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeveloper-Y%2FScalable-Software-Architecture","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FDeveloper-Y%2FScalable-Software-Architecture","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FDeveloper-Y%2FScalable-Software-Architecture/lists"}