What can Apache Kafka Developers learn from Online Gaming?

Streaming Audio: Apache Kafka® & Real-Time Data

What can online gaming teach us about making large-scale event management more collaborative in real-time? Ben Gamble (Developer Relations Manager, Aiven) has come to the world of real-time event streaming from an usual source: the video games industry. And if you stop to think about it, modern online games are complex, distributed real-time data systems with decades of innovative techniques to teach us.
In this episode, Ben talks with Kris about integrating gaming concepts with Apache Kafka®. Using Kafka’s state management stream processing, Ben has built systems that can handle real-time event processing at a massive scale, including interesting approaches to conflict resolution and collaboration.
Building latency into a system is one way to mask data processing time. Ben says that you can efficiently hide latency issues and prioritize performance improvements by setting an initial target and then optimizing from there. If you measure before optimizing, you can add an extra layer to manage user expectations better. Tricks like adding a visual progress bar give the appearance of progress but actually hide latency and improve the overall user experience.
To effectively handle challenging activities, like resolving conflicts and atomic edits, Ben suggests “slicing” (or nano batching) to break down tasks into small, related chunks. Slicing allows each task to be evaluated separately, thus producing timely outcomes that resolve potential background conflicts without the user knowing.
Ben also explains how he uses pooling to make collaboration seamless. Pooling is a process that links open requests with potential matches. Similar to booking seats on an airplane, seats are assigned when requests are made. As these types of connections are handled through a Kafka event stream, the initial open requests are eventually fulfilled when seats become available.
According to Ben, real-world tools that facilitate collaboration (such as Google Docs and Slack) work similarly. Just like multi-player gaming systems, multiple users can comment or chat in real-time and users perceive instant responses because of the techniques ported over from the gaming world.
As Ben sees it, the proliferation of these types of concepts across disciplines will also benefit a more significant number of collaborative systems. Despite being long established for gamers, these patterns can be implemented in more business applications to improve the user experience significantly.
EPISODE LINKS

• Going Multiplayer With Kafka—Current 2022
• Building a Dependable Real-Time Betting App with Confluent Cloud and Ably
• Event Streaming Patterns
• Watch the video version of this podcast
• Kris Jenkins’ Twitter
• Streaming Audio Playlist
• Join the Confluent Community
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Intro to Event-Driven Microservices with Confluent
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

Chapter

1. Intro (00:00:00)

2. Building real-time inspection machines (00:05:29)

3. Going Multiplayer with Kafka (00:07:14)

4. What does a real-time experience look like in gaming? (00:14:35)

5. Broken message offsets vs. dead letter queues (00:17:05)

6. The distributed transaction problem (00:19:15)

7. Solving for conflicting events with nano-batching (00:25:02)

8. Pooling in real-time systems (00:31:56)

9. Optimizing for latency (00:38:29)

10. Using column modeling to improve performance (00:44:08)

11. Principles for stream-processing (00:50:49)

12. It's a wrap! (00:52:46)