Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/dttung2905/flink-at-scale

đź“š Tech blogs & talks by companies that run Apache Flink in production
https://github.com/dttung2905/flink-at-scale

Last synced: 9 days ago
JSON representation

đź“š Tech blogs & talks by companies that run Apache Flink in production

Awesome Lists containing this project

README 

        
# Flink-at-scale



![HitCount](http://hits.dwyl.com/dttung2905/flink-at-scale.svg)

![license](https://img.shields.io/github/license/dttung2905/flink-at-scale)

![stars](https://img.shields.io/github/stars/dttung2905/flink-at-scale)



Are you new to the world of stream data processing ? Or are you looking for reference how others are running Flink ?



- **What** problems companies aim to solve with Flink? đź“ť

- **How** Flink use case evolves over the years? :star_struck:

- **How** companies go around with running Flink at scale? :rocket:



## Table of Contents



1. [Adobe](#adobe)

1. [Airbnb](#airbnb)

1. [Alibaba](#alibaba)

1. [Amazon](#amazon)

1. [Apple](#apple)

1. [Bloomberg](#bloomberg)

1. [Booking](#booking)

1. [ContentSquare](#contentsquare)

1. [Criteo](#criteo)

1. [Deliveroo](#deliveroo)

1. [Deloitte](#Deloitte)

1. [Doordash](#doordash)

1. [Ebay](#ebay)

1. [GoTo](#goto)

1. [Grab](#grab)

1. [Instacart](#instacart)

1. [Lyft](#lyft)

1. [Netflix](#netflix)

1. [Options Clearing Corporation](#options-clearing-corporation)

1. [Palo Alto Network](#palo-alto-network)

1. [Pinterest](#pinterest)

1. [Reddit](#reddit)

1. [Salesforce](#salesforce)

1. [Shopify](#shopify)

1. [Uber](#uber)

1. [Yelp](#yelp)

1. [Zalando](#zalando)

1. [Zhongyuan Bank](#zhongyuan-bank)



## Adobe



- [Stream Processing at Scale within Adobe Experience Platform](https://experienceleaguecommunities.adobe.com/t5/adobe-experience-platform-blogs/stream-processing-at-scale-within-adobe-experience-platform/ba-p/432442) - `2021` - :books:



## Airbnb



- [Apache Flink® on Kubernetes](https://medium.com/airbnb-engineering/apache-flink-on-kubernetes-84425d66ee11) - `2024` - :books:

- [Building production Flink jobs with Airstream at Airbnb - Pala Muthiah & Hao Wang](https://www.youtube.com/watch?v=mZwQ_SfrssI&ab_channel=FlinkForward) - `2019` - :books:



## Alibaba



- [Four Billion Records per Second! What is Behind Alibaba Double 11 - Flink Stream-Batch Unification Practice during Double 11 for the Very First Time](https://www.alibabacloud.com/blog/four-billion-records-per-second-what-is-behind-alibaba-double-11---flink-stream-batch-unification-practice-during-double-11-for-the-very-first-time_596962) - `2020` - :books:

- [A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development](https://hackernoon.com/a-brief-history-of-flink-tracing-the-big-data-engines-open-source-development-87464fd19e0f) - `2019` - :studio_microphone:

- [Better to Give and to Receive: Alibaba’s Open-source Contributions to Flink](https://medium.com/hackernoon/better-to-give-and-to-receive-alibabas-open-source-contributions-to-flink-295b3aef1da8) - `2019` - :studio_microphone:

- [In Search of Data Dominance: Spark Versus Flink](https://hackernoon.com/in-search-of-data-dominance-spark-versus-flink-45cefb28f377) - `2018` - :studio_microphone:

- [Flink or Flunk? Why Ele.me Is Developing a Taste for Apache Flink](https://hackernoon.com/flink-or-flunk-why-ele-me-is-developing-a-taste-for-apache-flink-7d2a74e4d6c0) - `2018` - :studio_microphone:

- [From Code Quality to Integration: Optimizing Alibaba’s Blink Testing Framework](https://hackernoon.com/from-code-quality-to-integration-optimizing-alibabas-blink-testing-framework-dc9c357319de) - `2018` - :studio_microphone:

- [Alibaba Blink: Real-Time Computing for Big-Time Gains](https://alibabatech.medium.com/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26) - `2018` - :studio_microphone:



## Amazon



- [Practical learnings from running thousands of Flink jobs | Amazon Web Services](https://www.youtube.com/watch?v=F5yKSznkls8&ab_channel=AmazonWebServices) - `2022` - :studio_microphone:



## Apple



- [MEETUP: Apple employees discuss; Streaming from Iceberg Data Lake & Multi Cluster Kafka Source](https://www.youtube.com/watch?v=H1SYOuLcUTI&ab_channel=Ververica) - `2022` - :studio_microphone:



## Bloomberg



- [3 Lessons We Learned Running Stateful Streaming Pipelines with Apache Flink and Kafka](https://current.confluent.io/2024-sessions/3-lessons-we-learned-running-stateful-streaming-pipelines-with-apache-flink-and-kafka) - `2024` - :studio_microphone:



## Booking



- [Flink Forward SF 2022: Keynote: Apache Flink as a security data processing engine at Booking.com](https://www.youtube.com/watch?v=-wSbb4JSuZU&ab_channel=FlinkForward) - `2022` - :studio_microphone:



## ContentSquare



- [Ten Flink Gotchas we wish we had known](https://engineering.contentsquare.com/2021/ten-flink-gotchas/) - `2021` - :books:



## Criteo



- [Data Streaming with Flink](https://medium.com/criteo-engineering/criteo-streaming-flink-31816c08da50) - `2018` - :books:



## Deliveroo



- [The road to running Apache Flink applications on AWS KDA](https://deliveroo.engineering/2023/03/30/the-road-to-run-apache-flink-applications-on-aws-kda.html) - `2023` - :books:

- [How Apache Flink Delivers for Deliveroo](https://www.youtube.com/watch?v=FWPmGpvemF0&ab_channel=TheNewStack) - `2023` - :studio_microphone:



## Deloitte 



- [Simplify Stream Processing With Serverless Apache Flink® From Confluent and Deloitte](https://engineering.deloitte.com.au/articles/simplify-stream-processing-with-serverless-apache-flink-from-confluent-and-deloitte) - `2024` - :books:



## Doordash



- [Building Scalable Real Time Event Processing with Kafka and Flink](https://doordash.engineering/2022/08/02/building-scalable-real-time-event-processing-with-kafka-and-flink/) - `2022` - :books:



## Ebay



- [eBay monitoring platform preprocessing and alerting on Flink - Garrett Li](https://www.youtube.com/watch?v=KTran5SM9fw&ab_channel=FlinkForward) - `2018` - :studio_microphone:



## GoTo



- [Managing Flink operations at GoJek - Ravi Suhag & Sumanth Nakshatrithaya](https://www.youtube.com/watch?v=9WmM-f3OjeM&ab_channel=FlinkForward) - `2018` - :studio_microphone:



## Grab



- [Safer deployment of streaming applications](https://engineering.grab.com/safer-flink-deployments) - `2023` - :books:



## Instacart



- [Building a Flink Self-Serve Platform on Kubernetes at Scale](https://tech.instacart.com/building-a-flink-self-serve-platform-on-kubernetes-at-scale-c11ef19aef10) - `2023` - :books:



## Lyft



- [Where’s My Data — A Unique Encounter with Flink Streaming’s Kinesis Connector](https://eng.lyft.com/wheres-my-data-a-unique-encounter-with-flink-streaming-s-kinesis-connector-6da3b11b164a) - `2023` - :books:

- [How Lyft built a streaming data platform with Flink on Kubernetes - Micah Wylde](https://www.youtube.com/watch?v=scv_u4lg8qg&ab_channel=FlinkForward) - `2020` - :studio_microphone:



## Netflix



- [Building a Scalable Flink Platform: A Tale of 15,000 Jobs at Netflix](https://current.confluent.io/2024-sessions/building-a-scalable-flink-platform-a-tale-of-15-000-jobs-at-netflix) - `2024` - :studio_microphone:

- [Streaming SQL in Data Mesh](https://netflixtechblog.com/streaming-sql-in-data-mesh-0d83f5a00d08) - `2023` - :books:

- [2021 Apache Flink Meetup - Hosted by Netflix](https://www.youtube.com/watch?v=rtz3p_iijP8&t=2420s&ab_channel=NetflixData) - `2021` - :studio_microphone:

- [Building metric platform using Flink for massive scale at Netflix - Abhay Amin](https://www.youtube.com/watch?v=AUreuwzU4nA&ab_channel=FlinkForward) - `2020` - :studio_microphone:

- [Flink Meetup at Netflix (Los Gatos) - January 28, 2020](https://www.youtube.com/watch?v=lvYgCGrwMK0&ab_channel=NetflixData) - `2020` - :studio_microphone:

- [Autoscaling Flink at Netflix - Timothy Farkas](https://www.youtube.com/watch?v=NV0jvA5ZDNc&ab_channel=FlinkForward) - `2020` - :studio_microphone:

- [High-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale](https://www.youtube.com/watch?v=NE5NzkskEtc&ab_channel=FlinkForward) - `2020` - :studio_microphone:

- [Massive Scale Data Processing at Netflix using Flink - Snehal Nagmote & Pallavi Phadnis](https://www.youtube.com/watch?v=lC0d3gAPXaI&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [Real-time Processing with Flink for Machine Learning at Netflix - Elliot Chow](https://www.youtube.com/watch?v=o4C7TDneH00&pp=ygUNZmxpbmsgbmV0ZmxpeA%3D%3D&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [#FlinkForward SF 2017: Monal Daxini - Stream Processing with Flink at Netflix](https://www.youtube.com/watch?v=sPB8w-YXX1s&pp=ygUNZmxpbmsgbmV0ZmxpeA%3D%3D&ab_channel=FlinkForward) - `2017` - :studio_microphone:



## Options Clearing Corporation



- [How the World's Largest Options Clearinghouse Moved From Batch to Stream Processing to Manage Risk](https://current.confluent.io/2024-sessions/how-the-worlds-largest-options-clearinghouse-moved-from-batch-to-stream-processing-to-manage-risk) - `2024` - :studio_microphone:



## Palo Alto Network



- [How to Build a Scalable Self-Managed Streaming Infrastructure with Flink - Part 1](https://live.paloaltonetworks.com/t5/engineering-blogs/how-to-build-a-scalable-self-managed-streaming-infrastructure/ba-p/563778) - `2023` - :books:



## Pinterest



- [Tuning Flink Clusters for Stability and Efficiency](https://medium.com/pinterest-engineering/tuning-flink-clusters-for-stability-and-efficiency-50d3d50384ed) - `2023` - :books:

- [Pinterest Flink Deployment Framework](https://medium.com/pinterest-engineering/pinterest-flink-deployment-framework-512c6cd4a1b7) - `2021` - :books:

- [Detecting Image Similarity in (Near) Real-time Using Apache Flink](https://medium.com/pinterest-engineering/detecting-image-similarity-in-near-real-time-using-apache-flink-723ce072b7d2) - `2021` - :books:

- [Unified Flink Source at Pinterest: Streaming Data Processing](https://medium.com/pinterest-engineering/unified-flink-source-at-pinterest-streaming-data-processing-c9d4e89f2ed6) - `2021` - :books:

- [Faster Flink adoption with self-service diagnosis tool at Pinterest](https://medium.com/pinterest-engineering/faster-flink-adoption-with-self-service-diagnosis-tool-at-pinterest-50a07143f444) - `2021` - :books:

- [Real-time experiment analytics at Pinterest using Apache Flink](https://medium.com/pinterest-engineering/real-time-experiment-analytics-at-pinterest-using-apache-flink-841c8df98dc2) - `2019` - :books:



## Reddit



- [Protecting Reddit Users in Real Time at Scale](https://www.reddit.com/r/RedditEng/comments/16m3t7m/protecting_reddit_users_in_real_time_at_scale/) - `2023` - :books:

- [Keeping Redditors safe in real-time with Flink Stateful Functions](https://www.youtube.com/watch?v=lWCt4t1Dhvc) - `2021` - :studio_microphone:



## Salesforce



- [Application Log Intelligence & Performance Insights at Salesforce using Flink](https://engineering.salesforce.com/application-log-intelligence-performance-insights-at-salesforce-using-flink-92955f30573f/) - `2023` - :books:



## Shopify



- [7 Tips For Optimizing Apache Flink Applications](https://shopify.engineering/optimizing-apache-flink-applications-tips) - `2022` - :books:

- [3 (More) Tips for Optimizing Apache Flink Applications](https://shopify.engineering/optimizing-apache-flink-tips-part-two) - `2022` - :books:

- [Scaling Shopify's BFCM Live Map: An Apache Flink Redesign](https://shopify.engineering/bfcm-live-map-2021-apache-flink-redesign) - `2021` - :books:



## Uber



- [Protobuf Support in Uber's Real-Time Data Stack](https://current.confluent.io/2024-sessions/protobuf-support-in-ubers-real-time-data-stack) - `2024` - :studio_microphone:

- [Driving Real-Time Data Solutions: Insights from Uber's Na Yang](https://www.ververica.com/blog/driving-real-time-data-solutions-insights-from-ubers-na-yang) - `2024` - :books:

- [Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot](https://www.uber.com/en-GB/blog/real-time-exactly-once-ad-event-processing/) - `2021` - :books:

- [Powering OLAP at Uber using Apache Flink](https://www.youtube.com/watch?app=desktop&v=SbPO7MHdwEc&ab_channel=FlinkForward) - `2020` - :studio_microphone:

- [[Uber Seattle] Introduction to Kappa+ Architecture using Apache Flink](https://www.youtube.com/watch?v=4qSlsYogALo&ab_channel=UberEngineering) - `2019` - :studio_microphone:

- [Scaling Uber’s Realtime Optimization with Apache Flink - Xingzhong Xu](https://www.youtube.com/watch?v=ydFXKrad6lo&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [[Uber Seattle] When Apache Pulsar Meets Apache Flink](https://www.youtube.com/watch?v=_Kjn8oLy-H8&ab_channel=UberEngineering) - `2019` - :studio_microphone:

- [Building Flink As a Service platform at Uber - Shuyi Chen & Rong Rong](https://www.youtube.com/watch?v=VX3S9POGAdU&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [Practical Experience running Flink in Production - Shuyi Chen & Rong Rong](https://www.youtube.com/watch?v=eBrjgU-sjwM&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [#FlinkForward SF 2017: Liu & Mai - AthenaX: Uber’s streaming processing platform on Flink](https://www.youtube.com/watch?v=VURuT8GbExo&ab_channel=FlinkForward) - `2017` - :studio_microphone:



## Yelp 

- [Flink on PaaSTA: Yelp’s new stream processing platform runs on Kubernetes](https://engineeringblog.yelp.com/2020/10/flink-on-paasta.html) - `2020` - :books:

- [Realtime bot detection with Flink](https://www.youtube.com/watch?v=1GefkDe7Ul8&ab_channel=FlinkForward) - `2020` - :studio_microphone:

- [Kubernetes + Operator + PaaSTA = Flink @ Yelp - Antonio Verardi](https://www.youtube.com/watch?v=hL5nNAMx8Bk&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [Stream SQL with Flink @ Yelp - Enrico Canzonieri](https://www.youtube.com/watch?v=QQJnHYXdl5g&ab_channel=FlinkForward) - `2019` - :studio_microphone:

- [How Yelp uses Flink for predicting store visits in real time](https://www.ververica.com/blog/how-yelp-uses-flink-for-predicting-store-visits-in-real-time) - `2019` - :books:

- [Powering Yelp’s Data Pipeline Infrastructure with Apache Flink - Enrico Canzonieri](https://www.youtube.com/watch?v=CmBP5bUd2OU&ab_channel=FlinkForward) - `2018` - :studio_microphone:



## Zalando



- [Running Apache Flink on Kubernetes](https://engineering.zalando.com/posts/2019/03/running-apache-flink-on-kubernetes.html) - `2019` - :books:

- [Complex Event Generation for Business Process Monitoring using Apache Flink](https://engineering.zalando.com/posts/2017/07/complex-event-generation-for-business-process-monitoring-using-apache-flink.html) - `2017` - :books:



## Zhongyuan Bank



- [Application of Apache Flink in Real-time Financial Data Lake](https://www.alibabacloud.com/blog/application-of-apache-flink-in-real-time-financial-data-lake_597529) - `2021` - :books: