rc19822209/flink-at-scale

📚 Tech blogs & talks by companies that run Apache Flink in production

Flink-at-scale

Are you new to the world of stream data processing ? Or are you looking for reference how others are running Flink ?

• What problems companies aim to solve with Flink? 📝
• How Flink use case evolves over the years? 🤩
• How companies go around with running Flink at scale? 🚀

Table of Contents

1. Adobe
2. Airbnb
3. Alibaba
4. Amazon
5. Apple
6. Booking
7. ContentSquare
8. Criteo
9. Deliveroo
10. Doordash
11. Ebay
12. GoTo
13. Grab
14. Instacart
15. Lyft
16. Netflix
17. Palo Alto Network
18. Pinterest
19. Salesforce
20. Shopify
21. Uber
22. Yelp
23. Zalando
24. Zhongyuan Bank

Adobe

• Stream Processing at Scale within Adobe Experience Platform - 2021 - 📚

Airbnb

• Building production Flink jobs with Airstream at Airbnb - Pala Muthiah & Hao Wang - 2019 - 📚

Alibaba

• Four Billion Records per Second! What is Behind Alibaba Double 11 - Flink Stream-Batch Unification Practice during Double 11 for the Very First Time - 2020 - 📚
• A Brief History of Flink: Tracing the Big Data Engine’s Open-source Development - 2019 - 🎙️
• Better to Give and to Receive: Alibaba’s Open-source Contributions to Flink - 2019 - 🎙️
• In Search of Data Dominance: Spark Versus Flink - 2018 - 🎙️
• Flink or Flunk? Why Ele.me Is Developing a Taste for Apache Flink - 2018 - 🎙️
• From Code Quality to Integration: Optimizing Alibaba’s Blink Testing Framework - 2018 - 🎙️
• Alibaba Blink: Real-Time Computing for Big-Time Gains - 2018 - 🎙️

Amazon

• Practical learnings from running thousands of Flink jobs | Amazon Web Services - 2022 - 🎙️

Apple

• MEETUP: Apple employees discuss; Streaming from Iceberg Data Lake & Multi Cluster Kafka Source - 2022 - 🎙️

Booking

• Flink Forward SF 2022: Keynote: Apache Flink as a security data processing engine at Booking.com - 2022 - 🎙️

ContentSquare

• Ten Flink Gotchas we wish we had known - 2021 - 📚

Criteo

• Data Streaming with Flink - 2018 - 📚

Deliveroo

• The road to running Apache Flink applications on AWS KDA - 2023 - 📚
• How Apache Flink Delivers for Deliveroo - 2023 - 🎙️

Doordash

• Building Scalable Real Time Event Processing with Kafka and Flink - 2022 - 📚

Ebay

• eBay monitoring platform preprocessing and alerting on Flink - Garrett Li - 2018 - 🎙️

GoTo

• Managing Flink operations at GoJek - Ravi Suhag & Sumanth Nakshatrithaya - 2018 - 🎙️

Grab

• Safer deployment of streaming applications - 2023 - 📚

Instacart

• Building a Flink Self-Serve Platform on Kubernetes at Scale - 2023 - 📚

Lyft

• Where’s My Data — A Unique Encounter with Flink Streaming’s Kinesis Connector - 2023 - 📚
• How Lyft built a streaming data platform with Flink on Kubernetes - Micah Wylde - 2020 - 🎙️

Netflix

• Streaming SQL in Data Mesh - 2023 - 📚
• 2021 Apache Flink Meetup - Hosted by Netflix - 2021 - 🎙️
• Building metric platform using Flink for massive scale at Netflix - Abhay Amin - 2020 - 🎙️
• Flink Meetup at Netflix (Los Gatos) - January 28, 2020 - 2020 - 🎙️
• Autoscaling Flink at Netflix - Timothy Farkas - 2020 - 🎙️
• High-Quality Performant and Cost Efficient Schema-Aware Data Streams on Flink at Netflix Scale - 2020 - 🎙️
• Massive Scale Data Processing at Netflix using Flink - Snehal Nagmote & Pallavi Phadnis - 2019 - 🎙️
• Real-time Processing with Flink for Machine Learning at Netflix - Elliot Chow - 2019 - 🎙️
• #FlinkForward SF 2017: Monal Daxini - Stream Processing with Flink at Netflix - 2017 - 🎙️

Palo Alto Network

• How to Build a Scalable Self-Managed Streaming Infrastructure with Flink - Part 1 - 2023 - 📚

Pinterest

• Tuning Flink Clusters for Stability and Efficiency - 2023 - 📚
• Pinterest Flink Deployment Framework - 2021 - 📚
• Detecting Image Similarity in (Near) Real-time Using Apache Flink - 2021 - 📚
• Unified Flink Source at Pinterest: Streaming Data Processing - 2021 - 📚
• Faster Flink adoption with self-service diagnosis tool at Pinterest - 2021 - 📚
• Real-time experiment analytics at Pinterest using Apache Flink - 2019 - 📚

Salesforce

• Application Log Intelligence & Performance Insights at Salesforce using Flink - 2023 - 📚

Shopify

• 7 Tips For Optimizing Apache Flink Applications - 2022 - 📚
• 3 (More) Tips for Optimizing Apache Flink Applications - 2022 - 📚
• Scaling Shopify's BFCM Live Map: An Apache Flink Redesign - 2021 - 📚

Uber

• Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot - 2021 - 📚
• Powering OLAP at Uber using Apache Flink - 2020 - 🎙️
• [Uber Seattle] Introduction to Kappa+ Architecture using Apache Flink - 2019 - 🎙️
• Scaling Uber’s Realtime Optimization with Apache Flink - Xingzhong Xu - 2019 - 🎙️
• [Uber Seattle] When Apache Pulsar Meets Apache Flink - 2019 - 🎙️
• Building Flink As a Service platform at Uber - Shuyi Chen & Rong Rong - 2019 - 🎙️
• Practical Experience running Flink in Production - Shuyi Chen & Rong Rong - 2019 - 🎙️
• #FlinkForward SF 2017: Liu & Mai - AthenaX: Uber’s streaming processing platform on Flink - 2017 - 🎙️

Yelp

• Flink on PaaSTA: Yelp’s new stream processing platform runs on Kubernetes - 2020 - 📚
• Realtime bot detection with Flink - 2020 - 🎙️
• Kubernetes + Operator + PaaSTA = Flink @ Yelp - Antonio Verardi - 2019 - 🎙️
• Stream SQL with Flink @ Yelp - Enrico Canzonieri - 2019 - 🎙️
• How Yelp uses Flink for predicting store visits in real time - 2019 - 📚
• Powering Yelp’s Data Pipeline Infrastructure with Apache Flink - Enrico Canzonieri - 2018 - 🎙️

Zalando

• Running Apache Flink on Kubernetes - 2019 - 📚
• Complex Event Generation for Business Process Monitoring using Apache Flink - 2017 - 📚

Zhongyuan Bank

• Application of Apache Flink in Real-time Financial Data Lake - 2021 - 📚