Reinventing Kafka on object storage (Interview)

The Changelog: Software Development, Open Source

Ryan Worl, Co-founder and CTO at WarpStream, joins us to talk about the world of Kafka and data streaming and how WarpStream redesigned the idea of Kafka to run in modern cloud environments directly on top of object storage. Last year they posted a blog titled, “Kafka is dead, long live Kafka” that hit the top of Hacker News to put WarpStream on the map. We get the backstory on Kafka and why it’s so widely used, who created it and for what purpose, and the behind the scenes on all things WarpStream.

Join the discussion

Changelog++ members save 14 minutes on this episode because they made the ads disappear. Join today!

Sponsors:

• Speakeasy – Production-ready, enterprise-resilient, best-in-class SDKs crafted in minutes. Speakeasy takes care of the entire SDK workflow to save you significant time, delivering SDKs to your customers in minutes with just a few clicks! Create your first SDK for free!
• Supabase – Supabase just finished their 12th launch week! Check it out. Or get a month of Supabase Pro (FREE) by going to supabase.com/changelogpod
• Paragon – Ship native integrations to production in days with more than 130 pre-built connectors, or configure your own custom integrations. Built for product and engineering. Learn more at useparagon.com/changelog
• Unblocked – Other developer tools can’t tell you how your codebase works and why. Unblocked can. We augment your code with context from Slack, Confluence, Jira, and more, so you get accurate answers without having to search for them. Sign up for free at getunblocked.com

Featuring:

• Ryan Worl – Twitter, GitHub, LinkedIn, Website
• Adam Stacoviak – Mastodon, Twitter, GitHub, LinkedIn, Website
• Jerod Santo – Mastodon, Twitter, GitHub, LinkedIn

Show Notes:

• Kafka is dead, long live Kafka
• Apache Kafka - open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
• Current 2024
• Amazon Managed Streaming for Apache Kafka
• Bobby Brown - My Prerogative

Something missing or broken? PRs welcome!

Chapter

1. This week on The Changelog (00:00:00)

2. Sponsor: Speakeasy (00:01:37)

https://speakeasy.com/

3. Start the show! (00:05:09)

4. What is Kafka? (00:06:28)

5. Kafka use cases (00:08:11)

6. Long live Kafka! (00:10:39)

7. Kafka on prem? (00:12:45)

8. What makes it hard to run? (00:15:01)

9. A margin of haters (00:17:45)

10. We met at Percona Live (00:23:18)

11. Sponsor: Supabase (00:28:14)

https://supabase.com/changelogpod

12. Object Storage sercrets (00:31:39)

13. Kafka brokers (00:36:20)

14. Downsides besides latency (00:46:13)

15. Could you R2? (00:49:38)

16. Very good demo (00:50:21)

17. $ warpstream playground (00:53:08)

18. The path for greenfield (01:00:10)

19. Why not open source? (01:02:37)

20. Sponsor: Paragon (01:16:54)

https://www.useparagon.com/changelog

21. Sponsor: Unblocked (01:19:50)

https://getunblocked.com

22. Why not bootstrap? (01:21:52)

23. Let's talk pricing (01:32:40)

24. Deep into pricing (01:35:57)

25. A good next step? (01:40:21)

26. Closing thoughts and stuff (01:42:04)